Topic: [BETA] MTGox websocket API, testers wanted - page 8. (Read 77681 times)

You seem to be looking at a different version of websocket.py, there is no _debug variable at line 43 (I'm using version 0.9.0 (this is the latest version, implementing websocket version 13). I have enabled tracing, this is the log until after the channel_subscribe()

Note the messages starting with 0x81, this is the framing to delimit the messages, this is why it is absolutely no fun implementing this by hand if you can get a working implementation from somewhere. Now after looking at this dump myself I remember, my first attempt writing the webocket client myself the first problem I stumbled upon were the Sec-Websocket-Key and the Sec-Websocket-Version headers, they look slightly different for every of the dozen ws versions that exist and MtGox does not like all variants of them. The funny thing is my first implementation worked just fine with their plain websocket server and refused to connect to the socketio. Then i threw it away and used websocket.py.

Im actually using php server side, and there is very little to be found on the web. Only code to create websockets, not to connect to them.

Anyway, I dont mind coding it if I know what to send, but python is mostly like chinese to me, so can you do me one more favor and modify websocket.py
change line 43 to:
_debug = True

Then run your app.
It will spit out the handshakes. If you copy/paste the output  I can try to implement it in php.

Looks good but remember there exist multiple different versions of websocket and their server might refuse to connect if you use the wrong version. The python example uses the websocket.py module https://pypi.python.org/pypi/websocket-client/ to do the actual websocket connect and so does my program to.

I remember before that I experimented with different implementations of websocket and not all of them worked, one of them was disconnected when sending the headers, i guess because it tried to use the wrong websocket version or something else was wrong with its headers (iirc by investigating the reasons it seemed to have to do with the sec-websocket-* headers and the random generated key, but then I just threw the code away and started from scratch with websocket.py). What I now use is urllib2 to do the initial http request, then construct the URL like above (the url you posted looks good) and then I let the websocket.py do the websocket connect and don't care about low level details like request headers

I don't know how this is in javascript/node.js (assuming this is what you are using?) but I guess there should exist some library too that will let you establish a webocket connection that takes care about all the low-level websocket request header stuff.

this is my working python code if this is of any help for you (its basically doing exactly the same as the other code snippets in this thread): (I guess to solve your problem you need to find the equivalent of "websocket.py" for your programming language)

Ah, you make it sound easy! And it probably is when you know what you are doing, but I dont
I found the url to get the session thingy. Im not sure what to do next; this is the closest I got so far:

edit: I guess the handshake should be this instead?
GET /socket.io/1/websocket/454534534534 HTTP/1.1
Upgrade: WebSocket
Connection: Upgrade
Host: socketio.mtgox.com
Origin: null

Nothing is happening, no response.

edit2.
I debugged the python app linked above, and that made me use this string:

Still no response, but now I wonder, do i need to set the authentication thing?

If you have a working version for plain websocket already then it should be trivial to add the little socketio overhead on top of it without needing a complete socketio library.

basically instead of just connecting to websocket.mtgox.com like you do it now you do the following:

send a http request to socketio.mtgox.com (look for the example somewhere in this thread for the exact URL), parse the result to get the session ID which is part of your personalized websocket url and then you use *that* to do the websocket connect. The rest is adding (or removing) the "1::" and "4::/mtgox:" headers before/after send/receive and the rest stays the same, its effectively only a handful of code you have to add, you don't have to implement everything of socketio, just these few things, its actually simpler than it might seem.

There was a python example near the beginning (or somewhere in the middle) of this thread that helped me a lot, and of course also the official socketio specification.

Btw, if you're doing node.js+socket.io you can use my example: https://github.com/icbit/trader.nodejs
It's not very well tested and debugged yet, but it supports both Mt.Gox and ICBIT.

This sucks. I spent a day learning about websockets and making goxgame work pretty decently, now I can start over again and it seems got a choice between redoing most of the work while learning (a lot more) javascript and node.js to implement some existing nodejs socket.io implementation and make it talk with my db, or  redoing most of the work trying to reverse engineering the socketio server to use websockets. Bah!

Thanks for the long explanation though.

With "I'm not totally sure" I was referring to websocket framing (0xff and friends), not to socketio. For websocket there exists an RFC. Websocket is actually not the problem, client implementations exist for many languages.

I actually just wanted to explain that these two MtGox servers are not the same and after writing down the bytes I realized that I might have forgotten some tiny details and therefore wanted to make clear that this is NOT a protocol specification, I just wrote it down as I remember it and so I wrote "I'm not totally sure" as a warning that it might be not 100% correct and nobody tries to implement something based on this posting.

There is a specification describing socket.io protocol.

its a separate server. And on the socketio websocket all messages look like this:

0x00 "4::/mtgox:{a json message}" 0xff
0x00 "4::/mtgox:{another json msg}" 0xff

and occasionally:

0x00 "2::" 0xff     <-- this is a ping, must be answered with "2::"

(I'm not totally sure about the 0xff 0x00 thing, there exist 13 different (no joke!) versions of websocket and there exist also different ways to delimit the messages, crazy stuff like: if bit7=1 then bit 0..6 = message length and when the next byte also has bit7=1 then its even more bits for the message length until a byte comes along that has bit7=0 but if all bits = 0 then its variable length ascii, 0xff terminated.

Fortunately the binary stuff is abstracted away by libraries like websocket.py in Python and other client implementations for other languages, so usually you just have a send() and a recv() method and don't need to care about 0x00 and 0xff etc., only the socketio "1::", "2::" and "4::" must be done manually.

The "/mtgox" part is some kind of "channel" name on the socketio layer but MtGox actually does not make any use of this feature, there is only one channel, the channels you subscribe *through* the websocket actually are an MtGox-internal thing, not part of the socketio layer. So there is some socketio protocol related stuff going on and wrapped around it that is not actually used for anything but its needed to satisfy the protocol rules.


On the plain websocket server you just connect, send the request header and it immediately begins talking to you with no additional handshake or other overhead:

0x00 "{a json message}" 0xff
0x00 "{another json msg}" 0xff

So its not the *same* websocket server (also it has a different hostname and IP address). Socketio is using websocket transport but a totally different handshake and protocol and framing.



depth (fulldepth) definitely has a limit and I ran into this once while testing (and it tried to ban me from calling this API for 24 hours [getting a new IP address on my DSL line fixed it]). I don't know about the other APIs.

Thanks for confirming its not just me. I thought socketio was implemented on top of websockets though, so its weird that still works?
Either way, Ill make a HTTP fallback option. Do you know how often you can poll it? The documentation says no more than once per 10 seconds, but Im not sure if that is per request or for all requests (I need trades, depth, and ticker) ?

Yes, its down again (for almost 3 hours now [this is nothing, sometimes its down for a few days]). And in the 24/7 IRC support channel #mtgox on freenode (btw. why does a commercial company use freenode?) there is currently nobody answering any questions.

Socketio works (clarkmoody is using socketio) but trading via the socketio server is no fun at all because of the 10 seconds lag (on top of the usual goxlag) even during quiet times and becomes totally unusable when there is some more action.

is websocket service down? I can connected but I cant get any response (nor via telnet).
 Clarkmoody and the other sites still seem to work though?

thanks a lot. That explains the weirdness I was getting.

On trade messages you need to update the orderbook yourself, there will be no depth message. [Edit: this is wrong, there is a depth message following every trade message but it can't do any damage if you always only use the total_volume and never use the volume difference]

If the trade is type:"bid" then it has filled an ask order, update your asks, if trade is type:"ask" then it has filled a bid order, update your bids.

On *own* trades there will be two trade messages: One public that everybody receives (like above), use that to update your orderbook like above and one private only to notify you about your trade that happened. Also on every trade that affects your *own* orders there will be user_order messages, at least one that sets the volume to the remaining volume (or zero and another one to finally remove it).

[Edit: the code below is flawed (maybe even outright wrong: You should not use the volume difference in the depth message, you should use the total volume]

WRONG WRONG WRONG
/WRONG /WRONG /WRONG


Edit: Here is a corrected version:
you would have two methods updating the orderbook, one that accepts a delta and one that accepts the new total volume.


And here is a simplified version that just waits for the inevitable depth message and completely ignores the trade message:

One thing isnt clear to me. when a trade occurs. does the API also send a depth message that reflects this trade? Or do I need to subscribe to the trade channel as well and then  match the trade with the orderbook to keep the orderbook up to date?

STOP

forget everything I just wrote about the nonce, I had a bug in my program (I used struct.unpack() which creates a tuple and put that tuple with the random number into my json instead of only the nonce). And the mtgox-server also has a bug so that it *accepts* this without complaining!

It will *not* accept it if the nonce is a plain random number and not incrementing. And it will never forget, you need a new API key after the change.

I have just spent another hour with my code and now my code look like this, I also requested a brand new virgin API key and now it works:


you find it at github: prof7bit/goxtool

Thanks, I'll try this out.

The documentation and PHP example code led me to believe that the nonce needs to increment for each message. Currently I'm setting the nonce of the first request to the Unix time in microseconds and then incrementing by 1 in each additional request.

It's weird that this would cause problems if a random value works fine. I'll see if I can get some clarification from MtGox.


Sounds heinous.

I have had invalid call before i changed the way my nonce was created. Now I'm just using 8 bytes from urandom to make an unsigned 64 bit integer and not one invalid calls anymore since the day I made that change, before i used something like time.time()*1E6 and saw a lot of "invalid call". One day I'm going to investigate what exactly it was that was wrong with my microseconds and why it works now.

The socketio server only accepts the socketio protocol (http request, then get a session ID, then use that ID to have a new URL to connect to a websocket service but this is not the same websocket as the plain old websocket, all messages are wrapped into some socketio protocol ("1::/mtgox" headers and the like). This is a whole new complicated layer on top of websocket.

According to the person I just talked to the plain websocket server is NOT (I repeat: NOT) deprecated, we can use it! Its also 10 seconds faster (no annoying lag, no sluggishness), reacts instantly to all commands.

fixed within 5 minutes after quick chat in #mtgox on freenode ;-)
Now I know where to go when there are problems