Pages:
Author

Topic: [BETA] MTGox websocket API, testers wanted - page 4. (Read 77653 times)

full member
Activity: 217
Merit: 100
Since mtgox trading halt socket.io api does not work, or works sparadically.

I hade to upgrade our client to be able to work in hybrid mode basically request full orderbook via http without socket.io connect.

and additional timeout logic not to get blocked by mtgox blocker.

So you try socket.io and if it fails you fall back to http get with extended connection timeout. Kudos for working around it!

What kind of delay do you have?... Approx how old is the latest book entry that you receive? Do you get depth and trades?
sr. member
Activity: 379
Merit: 250
Since mtgox trading halt socket.io api does not work, or works sparadically.

I hade to upgrade our client to be able to work in hybrid mode basically request full orderbook via http without socket.io connect.

and additional timeout logic not to get blocked by mtgox blocker.
full member
Activity: 217
Merit: 100
Is it me or the websocket has been down for days? Even my most basic implementation is silent.

My socket.io implementation gets as far as upgrade and then I just receive zero-byte returns, I had previously coded zero-byte return to trigger connection reset, I took that trigger out and the receive function just finishes instantly with zero-bytes received.

It's been like this since MtGox went down, and has not recovered even after MtGox opened for trading via the user-account web interface again.

So I'm not sure what the problem is. Maybe we just wait? Sad


13:34:43
=============================================================
THR_MtGox::SocketCreate: CONNECTED TO: socketio.mtgox.com:80
=============================================================

13:34:43 TRANSMIT>>>>>>>>
GET /socket.io/1/?Currency=EUR HTTP/1.1
Host: socketio.mtgox.com/mtgox?Currency=EUR
Connection: keep-alive



13:34:44 RECEIVE>>>>>>>>>
HTTP/1.1 200 OK
Content-Type: text/plain
Date: Sat, 13 Apr 2013 11:34:43 GMT
Connection: keep-alive
Transfer-Encoding: chunked

30
b-LJ3EKgBF4kkuvdLP1d:60:60:websocket,flashsocket
0


13:34:44 TRANSMIT>>>>>>>>
GET /socket.io/1/websocket/b-LJ3EKgBF4kkuvdLP1d?Currency=EUR HTTP/1.1
Upgrade: websocket
Connection: Upgrade
Host: socketio.mtgox.com/mtgox?Currency=EUR
Origin: socketio.mtgox.com/mtgox?Currency=EUR
Sec-WebSocket-Key: UWaphFPiSqq3f2gOmaD5Sg==
Sec-WebSocket-Version: 13


13:34:45 RECEIVE>>>>>>>>>
HTTP/1.1 101 Switching Protocols
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Accept: Bzz8qKJPEMNNOgC4hOZd3iZNb5o=

�1::
13:34:45 TRANSMIT>>>>>>>>
TX_STR_FRAMED -> "1::/mtgox"


...ZERO-BYTE RX FROM NOW ON...
newbie
Activity: 17
Merit: 0
Is it me or the websocket has been down for days? Even my most basic implementation is silent.
hero member
Activity: 548
Merit: 502
So much code.
sr. member
Activity: 379
Merit: 250
Well Done !!! Clark Smiley
hero member
Activity: 548
Merit: 502
So much code.
This is the list of channel ids I descovered till the moment

Well since we're playing that game  Smiley

Code:
            {
                'AUD': {'eb6aaa11-99d0-4f64-9e8c-1140872a423d': 'ticker',
                        '296ee352-dd5d-46f3-9bea-5e39dede2005': 'depth',
                        'dbf1dee9-4f2e-4a08-8cb7-748919a71b21': 'trades'},
                'CAD': {'10720792-084d-45ba-92e3-cf44d9477775': 'ticker',
                        '5b234cc3-a7c1-47ce-854f-27aee4cdbda5': 'depth',
                        'dbf1dee9-4f2e-4a08-8cb7-748919a71b21': 'trades'},
                'CHF': {'2644c164-3db7-4475-8b45-c7042efe3413': 'ticker',
                        '113fec5f-294d-4929-86eb-8ca4c3fd1bed': 'depth',
                        'dbf1dee9-4f2e-4a08-8cb7-748919a71b21': 'trades'},
                'CNY': {'c251ec35-56f9-40ab-a4f6-13325c349de4': 'ticker',
                        '0d1ecad8-e20f-459e-8bed-0bdcf927820f': 'depth',
                        'dbf1dee9-4f2e-4a08-8cb7-748919a71b21': 'trades'},
                'DKK': {'e5ce0604-574a-4059-9493-80af46c776b3': 'ticker',
                        '9219abb0-b50c-4007-b4d2-51d1711ab19c': 'depth',
                        'dbf1dee9-4f2e-4a08-8cb7-748919a71b21': 'trades'},
                'EUR': {'0bb6da8b-f6c6-4ecf-8f0d-a544ad948c15': 'ticker',
                        '057bdc6b-9f9c-44e4-bc1a-363e4443ce87': 'depth',
                        'dbf1dee9-4f2e-4a08-8cb7-748919a71b21': 'trades'},
                'GBP': {'7b842b7d-d1f9-46fa-a49c-c12f1ad5a533': 'ticker',
                        '60c3af1b-5d40-4d0e-b9fc-ccab433d2e9c': 'depth',
                        'dbf1dee9-4f2e-4a08-8cb7-748919a71b21': 'trades'},
                'HKD': {'d3ae78dd-01dd-4074-88a7-b8aa03cd28dd': 'ticker',
                        '049f65dc-3af3-4ffd-85a5-aac102b2a579': 'depth',
                        'dbf1dee9-4f2e-4a08-8cb7-748919a71b21': 'trades'},
                'JPY': {'a39ae532-6a3c-4835-af8c-dda54cb4874e': 'ticker',
                        '94483e07-d797-4dd4-bc72-dc98f1fd39e3': 'depth',
                        'dbf1dee9-4f2e-4a08-8cb7-748919a71b21': 'trades'},
                'NOK': {'7532e866-3a03-4514-a4b1-6f86e3a8dc11': 'ticker',
                        '66da7fb4-6b0c-4a10-9cb7-e2944e046eb5': 'depth',
                        'dbf1dee9-4f2e-4a08-8cb7-748919a71b21': 'trades'},
                'NZD': {'5ddd27ca-2466-4d1a-8961-615dedb68bf1': 'ticker',
                        'cedf8730-bce6-4278-b6fe-9bee42930e95': 'depth',
                        'dbf1dee9-4f2e-4a08-8cb7-748919a71b21': 'trades'},
                'PLN': {'b4a02cb3-2e2d-4a88-aeea-3c66cb604d01': 'ticker',
                        'e4ff055a-f8bf-407e-af76-676cad319a21': 'depth',
                        'dbf1dee9-4f2e-4a08-8cb7-748919a71b21': 'trades'},
                'RUB': {'bd04f720-3c70-4dce-ae71-2422ab862c65': 'ticker',
                        'd6412ca0-b686-464c-891a-d1ba3943f3c6': 'depth',
                        'dbf1dee9-4f2e-4a08-8cb7-748919a71b21': 'trades'},
                'SEK': {'6caf1244-655b-460f-beaf-5c56d1f4bea7': 'ticker',
                        '8f1fefaa-7c55-4420-ada0-4de15c1c38f3': 'depth',
                        'dbf1dee9-4f2e-4a08-8cb7-748919a71b21': 'trades'},
                'SGD': {'2cb73ed1-07f4-45e0-8918-bcbfda658912': 'ticker',
                        '41e5c243-3d44-4fad-b690-f39e1dbb86a8': 'depth',
                        'dbf1dee9-4f2e-4a08-8cb7-748919a71b21': 'trades'},
                'THB': {'d58e3b69-9560-4b9e-8c58-b5c0f3fda5e1': 'ticker',
                        '67879668-532f-41f9-8eb0-55e7593a5ab8': 'depth',
                        'dbf1dee9-4f2e-4a08-8cb7-748919a71b21': 'trades'},
                'USD': {'d5f06780-30a8-4a48-a2f8-7ed181b4a13f': 'ticker',
                        '24e67e0d-1cad-4cc0-9e7a-f8523ef460fe': 'depth',
                        'dbf1dee9-4f2e-4a08-8cb7-748919a71b21': 'trades'}
            }
sr. member
Activity: 379
Merit: 250
This is the list of channel ids I descovered till the moment

Quote
{ 'USD' : {
                              'trades' : 'dbf1dee9-4f2e-4a08-8cb7-748919a71b21',
                              'ticker' : 'd5f06780-30a8-4a48-a2f8-7ed181b4a13f',
                              'depth' : '24e67e0d-1cad-4cc0-9e7a-f8523ef460fe'
                           },
                           'EUR' : {
                              'trades' : 'dbf1dee9-4f2e-4a08-8cb7-748919a71b21',
                              'depth' : '057bdc6b-9f9c-44e4-bc1a-363e4443ce87',
                              'ticker' : '0bb6da8b-f6c6-4ecf-8f0d-a544ad948c15'
                           },
                           'GBP' : {
                              'trades' : 'dbf1dee9-4f2e-4a08-8cb7-748919a71b21',
                              'depth' : '60c3af1b-5d40-4d0e-b9fc-ccab433d2e9c',
                              'ticker': '7b842b7d-d1f9-46fa-a49c-c12f1ad5a533'
                           },
                           'CNY' : {
                              'trades' : 'dbf1dee9-4f2e-4a08-8cb7-748919a71b21',
                              'depth' : '0d1ecad8-e20f-459e-8bed-0bdcf927820f',
                              'ticker' : 'c251ec35-56f9-40ab-a4f6-13325c349de4'
                           },
                           'RUB' : {
                              'trades' : 'dbf1dee9-4f2e-4a08-8cb7-748919a71b21',
                              'depth' : 'bd04f720-3c70-4dce-ae71-2422ab862c65',
                              'ticker' : 'd6412ca0-b686-464c-891a-d1ba3943f3c6'
                           },
                           'CAD' : {
                              'trades' : 'dbf1dee9-4f2e-4a08-8cb7-748919a71b21',
                              'depth' : '10720792-084d-45ba-92e3-cf44d9477775',
                              'ticker' : '5b234cc3-a7c1-47ce-854f-27aee4cdbda5'
                           },
                           'PLN' : {
                              'trades' : 'dbf1dee9-4f2e-4a08-8cb7-748919a71b21',
                              'depth' : 'b4a02cb3-2e2d-4a88-aeea-3c66cb604d01',
                              'ticker' : 'e4ff055a-f8bf-407e-af76-676cad319a21'
                           },
                           'AUD' : {
                              'trades' : 'dbf1dee9-4f2e-4a08-8cb7-748919a71b21',
                              'depth' : 'eb6aaa11-99d0-4f64-9e8c-1140872a423d',
                              'ticker' : '296ee352-dd5d-46f3-9bea-5e39dede2005'
                           }
full member
Activity: 217
Merit: 100
No, it cannot exceed the holdings of the owner, otherwise it would not show up in the order book. It would stay in "invalid" state until funds become available and only be visible to the owner himself but not for anyone else.
Wow well there are some large players then!

Well I now have stable multicurrency stream.io coming into my app. I still notice the odd hang during heavy times (maybe the ddos is not helping?).

I look for consecutive zero receive bytes or receive timout, if I see this I close the socket, destroy it, recreate and reconnect after a short delay.

One note to others (has been mentioned before), the channel UUIDs are not worth much from the client end. if you want multicurrency stream then you should filter on the human readable channel type and currency type. All UUIDs are not published in MtGox API docs, I suppose if you want to use them you would have to discover them yourself in the RX log.

Now how long this will stay good until I have to revisit this part of my code I dont know, hopefully for a long long time. Thankyou to all who helped me stumble to this point  Grin
hero member
Activity: 938
Merit: 500
https://youengine.io/
This was clearly exceeding the holdings of its owner. The stream window below is 5 minutes.

No, it cannot exceed the holdings of the owner, otherwise it would not show up in the order book. It would stay in "invalid" state until funds become available and only be visible to the owner himself but not for anyone else.
full member
Activity: 217
Merit: 100
I think I remember reading in this thread that MagicalTux was going to change the order API so that people cant place more asks than BTC they have on account.

Watching the live depth stream I dont think that this is implemented yet.

On the GBP stream below we can see a bot went crazy and started placing many asks of the 2 same values resulting in 2 long horizontal lines.

This was clearly exceeding the holdings of its owner. The stream window below is 5 minutes.

hero member
Activity: 938
Merit: 500
https://youengine.io/
are you plotting volume also? if so do you find the live numbers sum closely to the history?

Not yet. I'm only plotting a quick & dirty price chart of the last 24 hours so I can quickly see from 5 meters distance if it has gone up or down during the night. I was too lazy until now to also write code to plot volume bars (I also don't have enough room in the terminal window to plot too many details in ascii-art with ncurses)
full member
Activity: 217
Merit: 100
...trade history from https://data.mtgox.com/api/2/BTCUSD/money/trades gives me correct timestamps from today and I'm using them to plot an M15 candle chart and then use the date from the streaming trade messages to continue this chart (update the last candle until time_trade > time_candle_open + 15 minutes) and apend new candles to the chart....

cool, similar to what i'm doing.

It would be nice to have a mtgox call similar to bitcoincharts, so one could specify to get all trades since X unixtime.

or even better binned trades in choice of bin size, M15, M30 etc. this would really save some bandwidth for mtgox!

P.S.
are you plotting volume also? if so do you find the live numbers sum closely to the history?
full member
Activity: 217
Merit: 100
you are correct prof7bit, my parsing state machine is buggy causing 4s to get dropped Embarrassed.

I'm doing it with inc pointer on the raw incoming data buffer for speed and screwed up. I will rework it now. thanks man!
hero member
Activity: 938
Merit: 500
https://youengine.io/
thanks for checking it out!

your trade UUID (top) vs the one I get (bottom)

dbf1dee9-4f2e-4a08-8cb7-748919a71b21
dbf1dee9-f2e-a08-8cb7-78919a71b21


the stream I am parsing is the default stream after first connecting, this is probably the problem. I would like to unsubscribe from them but have not been able to yet. It seems odd that the default stream contains junk, why bother including it?

No, I am using them too.

But all your "4" are missing! Are you trying to parse the "4::/mtgox:" by any chance and have an error there in your parser? In your previous post the "4::/mtgox:" is appearing between the json objects and is the only thing that has a "4" in it, all other "4" seem to have gone missing!
full member
Activity: 217
Merit: 100
thanks for checking it out!

your trade UUID (top) vs the one I get (bottom)

dbf1dee9-4f2e-4a08-8cb7-748919a71b21
dbf1dee9-f2e-a08-8cb7-78919a71b21


the stream I am parsing is the default stream after first connecting, this is probably the problem. I would like to unsubscribe from them but have not been able to yet. It seems odd that the default stream contains junk, why bother including it?

Then I will subscribe to the new streams, so i will try subscribing to new streams anyway and see how it goes, i guess i just send a masked/framed subscription request as per the mtgox stream api wiki.

So perhaps i will just have to ignore the junk stream, it will still have to flow thru my parser though, i dislike wasted clock cycles, plus it loads the server.
hero member
Activity: 938
Merit: 500
https://youengine.io/
This is strange.

My timestamps as received here are correct. Below are some messages I have received here from socketio.mtgox.com a few minutes not long ago, the timestamps look normal.

Also the trade history from https://data.mtgox.com/api/2/BTCUSD/money/trades gives me correct timestamps from today and I'm using them to plot an M15 candle chart and then use the date from the streaming trade messages to continue this chart (update the last candle until time_trade > time_candle_open + 15 minutes) and apend new candles to the chart. I could not see any mismatch between the timestamps of "money/trades" and new incoming trade messages.

The first thing that catches the eye is that in your trade message the date field is only 7 digits long while mine is a complete timestamp (in seconds, 10 digits) and also your microseconds timestamps seem too short (12 digits instead of 16), how can this happen? Something eating characters and always the *same* sequence of characters? Btw I cannot see the character "4" appearing anywhere in your dump (except for the 4::/mtgox (!)) and also some of the GUID strings are too short!

Below are the mesages as received by me (and pretty-printed/formatted by json.dumps())

trade
Code:
{
     "origin": "broadcast",
     "trade": {
          "price_currency": "USD",
          "trade_type": "bid",
          "price_int": "9120000",
          "item": "BTC",
          "price": 91.2,
          "primary": "Y",
          "tid": "1364662801201914",
          "amount": 1.41528958,
          "amount_int": "141528958",
          "date": 1364662801,
          "type": "trade",
          "properties": "limit"
     },
     "private": "trade",
     "channel": "dbf1dee9-4f2e-4a08-8cb7-748919a71b21",
     "op": "private"
}

Depth
Code:
{
     "origin": "broadcast",
     "depth": {
          "volume_int": "0",
          "price_int": "9120000",
          "currency": "USD",
          "price": "91.2",
          "volume": "0",
          "item": "BTC",
          "type_str": "ask",
          "total_volume_int": "9828724331",
          "now": "1364662801229415",
          "type": 1
     },
     "private": "depth",
     "channel": "24e67e0d-1cad-4cc0-9e7a-f8523ef460fe",
     "op": "private"
}



Ticker
Code:
{
     "origin": "broadcast",
     "ticker": {
          "sell": {
               "value_int": "9120000",
               "currency": "USD",
               "display_short": "$91.20",
               "display": "$91.20000",
               "value": "91.20000"
          },
          "last_orig": {
               "value_int": "9120000",
               "currency": "USD",
               "display_short": "$91.20",
               "display": "$91.20000",
               "value": "91.20000"
          },
          "now": "1364662937327085",
          "buy": {
               "value_int": "9119999",
               "currency": "USD",
               "display_short": "$91.20",
               "display": "$91.19999",
               "value": "91.19999"
          },
          "last": {
               "value_int": "9120000",
               "currency": "USD",
               "display_short": "$91.20",
               "display": "$91.20000",
               "value": "91.20000"
          },
          "vol": {
               "value_int": "5095833923444",
               "currency": "BTC",
               "display_short": "50,958.34\u00a0BTC",
               "display": "50,958.33923444\u00a0BTC",
               "value": "50958.33923444"
          },
          "last_local": {
               "value_int": "9120000",
               "currency": "USD",
               "display_short": "$91.20",
               "display": "$91.20000",
               "value": "91.20000"
          },
          "last_all": {
               "value_int": "9120000",
               "currency": "USD",
               "display_short": "$91.20",
               "display": "$91.20000",
               "value": "91.20000"
          },
          "vwap": {
               "value_int": "9064567",
               "currency": "USD",
               "display_short": "$90.65",
               "display": "$90.64567",
               "value": "90.64567"
          },
          "high": {
               "value_int": "9498000",
               "currency": "USD",
               "display_short": "$94.98",
               "display": "$94.98000",
               "value": "94.98000"
          },
          "low": {
               "value_int": "8550000",
               "currency": "USD",
               "display_short": "$85.50",
               "display": "$85.50000",
               "value": "85.50000"
          },
          "avg": {
               "value_int": "9037305",
               "currency": "USD",
               "display_short": "$90.37",
               "display": "$90.37305",
               "value": "90.37305"
          }
     },
     "private": "ticker",
     "channel": "d5f06780-30a8-4a48-a2f8-7ed181b4a13f",
     "op": "private"
}

Lag
Code:
{
     "origin": "broadcast",
     "lag": {
          "stamp": "1364663115576119",
          "qid": "74655212-1bc0-4b69-95da-79e40f4f771f",
          "age": 329545
     },
     "private": "lag",
     "channel": "85174711-be64-4de1-b783-0628995d7914",
     "op": "private"
}
full member
Activity: 217
Merit: 100
LOL, i've routed into a temporal rift, i will keep this a secret, future prices are good!!! Grin

But seriously guys the JSON timestamps are wrong, or i'm just not understanding what that time is for???

Nothing wrong at my RX end as server sends non masked frames, just plain text blocks between delimiters.

TRADE and DEPTH json stream:


{"channel":"dbf1dee9-f2e-a08-8cb7-78919a71b21","op":"private","origin":"broadcast","private":
"trade","trade":{"type":"trade","date":1366631,"amount":2,"price":91.00021,"tid":"136663155159",
"amount_int":"200000000","price_int":"9100021","item":"BTC","price_currency":"USD","trade_type":"bid",
"primary":"Y","properties":"market"}}�~4::/mtgox:{"channel":"2e67e0d-1cad-cc0-9e7a-f8523ef60fe",
"op":"private","origin":"broadcast","private":"depth",
"depth":{"price":"91.00021","type":1,"type_str":"ask","volume":"0","price_int":"9100021","volume_int":"0",
"item":"BTC","currency":"USD","now":"136663158281","total_volume_int":"3016823508"}}�

{"channel":"2e67e0d-1cad-cc0-9e7a-f8523ef60fe","op":"private","origin":"broadcast","private":
"depth","depth":{"price":"87.5001","type":2,"type_str":"bid","volume":"0","price_int":"8750010",
"volume_int":"0","item":"BTC","currency":"USD","now":"136663161999","total_volume_int":"136753855"}}



trade has "now":"136663158281"  === Mon, 22 Apr 2013 11:53:02 GMT

depth has "now":"136663161999" === Mon, 22 Apr 2013 11:53:39 GMT

if you dont trust my unixtime to datetime conversion then plug them into an online converter (using the 10 leftmost bolded characters):
http://www.onlineconversion.com/unix_time.htm

so what gives, does nobody else use this data, do I have to discard timestamps and stamp them myself using local clock? that's not so good!
Pages:
Jump to: