Oh my... thanks muchly writhe! Slowly but surely, I'll get this stream established.
Here's an ammended table for little endian:
/*-------------------------------------------------------------*\
| LITTLE ENDIAN FORMAT WHERE LSB == 0x00000001 |
| 3 2 1 0|
|1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0|
+-------------------------------+-+-------------+-+-+-+-+-------+
| Extended payload length |M| Payload len |F|R|R|R| opcode|
| (16/64) |A| (7) |I|S|S|S| (4) |
| (if payload len==126/127) |S| |N|V|V|V| |
| |K| | |1|2|3| |
+-------------------------------+-+-------------+-+-+-+-+-------+
| Extended payload length continued, if payload len == 127 |
+-------------------------------+-------------------------------+
|Masking-key, if MASK set to 1 | |
+-------------------------------+-------------------------------+
| Payload Data | Masking-key (continued) |
+-------------------------------+-------------------------------+
: Payload Data continued ... :
+---------------------------------------------------------------+
| Payload Data continued ... |
\*-------------------------------------------------------------*/
Topic: [BETA] MTGox websocket API, testers wanted - page 6. (Read 77681 times)
For anyone else getting down and dirty at bit level, heres a redrawn table for RFC 6455 header.
In this table the least significant bit is on the right, makes it easier to mask and shift.
I cannot say this has no error, I have not tested, so verify by eye against original table first:
In this table the least significant bit is on the right, makes it easier to mask and shift.
I cannot say this has no error, I have not tested, so verify by eye against original table first:
Your table is incorrect. The original version in RFC 6455 is drawn with the most significant on the left. It's a bizarre numbering scheme used in most RFCs called MSB-0.
It's easier to read the RFC tables a byte at a time. The protocols assume the granularity of data transfer is 8-bit bytes so they don't really have the concept of a multi-bit word. When they do need multi-bit word they usually encode it as multiple 8-bit bytes using big-endian ordering.
For example, the lower 4 bits of the first byte are the opcode and the most significant bit is the FIN flag. Similarly, for the second byte the payload length is the lower 7 bits and the mask flag is the most significant bit.
Assuming little-endian with least significant bit (0) on the right, the first 2 bytes should look like this:
Code:
1
5 14 - 8 7 6 5 4 3 - 0
+-+-------------+-+-+-+-+--------+
|M| Payload len |F|R|R|R| opcode |
|A| (7) |I|S|S|S| (4) |
|S| |N|V|V|V| |
|K| | |1|2|3| |
+-+-------------+-+-+-+-+--------+
|-------------| |-------------|
second byte first byte
5 14 - 8 7 6 5 4 3 - 0
+-+-------------+-+-+-+-+--------+
|M| Payload len |F|R|R|R| opcode |
|A| (7) |I|S|S|S| (4) |
|S| |N|V|V|V| |
|K| | |1|2|3| |
+-+-------------+-+-+-+-+--------+
|-------------| |-------------|
second byte first byte
For anyone else getting down and dirty at bit level, heres a redrawn table for RFC 6455 header.
In this table the least significant bit is on the right, makes it easier to mask and shift.
I cannot say this has no error, I have not tested, so verify by eye against original table first:
EDIT:BELOW TABLE IS WRONG! SCROLL DOWN PAGE FOR FIXED VERSION
/*-------------------------------------------------------------*\
| LSB ON RIGHT IN THIS TABLE |
| 3 2 1 |
|1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0|
+-------------------------------+-------------+-+-------+-+-+-+-+
| Extended payload length | Payload len |M| opcode|R|R|R|F|
| (16/64) | (7) |A| (4) |S|S|S|I|
| (if payload len==126/127) | |S| |V|V|V|N|
| | |K| |3|2|1| |
+-------------------------------+-------------+-+-------+-+-+-+-+
| Extended payload length continued, if payload len == 127 |
+-------------------------------+-------------------------------+
|Masking-key, if MASK set to 1 | |
+-------------------------------+-------------------------------+
| Payload Data | Masking-key (continued) |
+-------------------------------+-------------------------------+
: Payload Data continued ... :
+---------------------------------------------------------------+
| Payload Data continued ... |
+--------------------------------------------------------------*/
ORIGINAL TABLE: LSB ON LEFT.
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-------+-+-------------+-------------------------------+
|F|R|R|R| opcode|M| Payload len | Extended payload length |
|I|S|S|S| (4) |A| (7) | (16/64) |
|N|V|V|V| |S| | (if payload len==126/127) |
| |1|2|3| |K| | |
+-+-+-+-+-------+-+-------------+ - - - - - - - - - - - - - - - +
| Extended payload length continued, if payload len == 127 |
+ - - - - - - - - - - - - - - - +-------------------------------+
| |Masking-key, if MASK set to 1 |
+-------------------------------+-------------------------------+
| Masking-key (continued) | Payload Data |
+-------------------------------- - - - - - - - - - - - - - - - +
: Payload Data continued ... :
+ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - +
| Payload Data continued ... |
+---------------------------------------------------------------+
In this table the least significant bit is on the right, makes it easier to mask and shift.
I cannot say this has no error, I have not tested, so verify by eye against original table first:
EDIT:BELOW TABLE IS WRONG! SCROLL DOWN PAGE FOR FIXED VERSION
/*-------------------------------------------------------------*\
| LSB ON RIGHT IN THIS TABLE |
| 3 2 1 |
|1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0|
+-------------------------------+-------------+-+-------+-+-+-+-+
| Extended payload length | Payload len |M| opcode|R|R|R|F|
| (16/64) | (7) |A| (4) |S|S|S|I|
| (if payload len==126/127) | |S| |V|V|V|N|
| | |K| |3|2|1| |
+-------------------------------+-------------+-+-------+-+-+-+-+
| Extended payload length continued, if payload len == 127 |
+-------------------------------+-------------------------------+
|Masking-key, if MASK set to 1 | |
+-------------------------------+-------------------------------+
| Payload Data | Masking-key (continued) |
+-------------------------------+-------------------------------+
: Payload Data continued ... :
+---------------------------------------------------------------+
| Payload Data continued ... |
+--------------------------------------------------------------*/
ORIGINAL TABLE: LSB ON LEFT.
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-------+-+-------------+-------------------------------+
|F|R|R|R| opcode|M| Payload len | Extended payload length |
|I|S|S|S| (4) |A| (7) | (16/64) |
|N|V|V|V| |S| | (if payload len==126/127) |
| |1|2|3| |K| | |
+-+-+-+-+-------+-+-------------+ - - - - - - - - - - - - - - - +
| Extended payload length continued, if payload len == 127 |
+ - - - - - - - - - - - - - - - +-------------------------------+
| |Masking-key, if MASK set to 1 |
+-------------------------------+-------------------------------+
| Masking-key (continued) | Payload Data |
+-------------------------------- - - - - - - - - - - - - - - - +
: Payload Data continued ... :
+ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - +
| Payload Data continued ... |
+---------------------------------------------------------------+
I noticed that my server socket.io-client is connected for more 10 hours already without any reconnects whille my local dev server reconnects often but reconnects stably.
Can it be gox fixed the new socket.io server you were taling about to be able to connect without any patches?
Can it be gox fixed the new socket.io server you were taling about to be able to connect without any patches?
Yes, this socketio-beta from yesterday is now reachable via the main hostname socketio (apperently he fixed the bug so it works with all clients) and the hostname socketio-beta does no longer exist. And the old server is still reachable with socketio-old (and its lightening fast now because it has only few users now) but I don't know when he will switch it off.
thank you for clarification
I noticed that my server socket.io-client is connected for more 10 hours already without any reconnects whille my local dev server reconnects often but reconnects stably.
Can it be gox fixed the new socket.io server you were taling about to be able to connect without any patches?
Can it be gox fixed the new socket.io server you were taling about to be able to connect without any patches?
Yes, this socketio-beta from yesterday is now reachable via the main hostname socketio (apperently he fixed the bug so it works with all clients) and the hostname socketio-beta does no longer exist. And the old server is still reachable with socketio-old (and its lightening fast now because it has only few users now) but I don't know when he will switch it off.
It works with md5?
No, I just use large numbers calculated from system time.
I noticed that my server socket.io-client is connected for more 10 hours already without any reconnects whille my local dev server reconnects often but reconnects stably.
Can it be gox fixed the new socket.io server you were taling about to be able to connect without any patches?
Can it be gox fixed the new socket.io server you were taling about to be able to connect without any patches?
We can stop worrying about socketio-beta, MT has taken it live. socketio is now the new server and socketio-beta is obsolete and the old server is now socketio-old.mtgox.com
Yes, now it works again. Big Thanks!
It works with md5? Then the bug is not fixed! It must NOT work with md5! Please show a json dump of one of your api calls, let me see how it looks like.
There we have the reason why it is not working. It *MUST* be a numeric value and it MUST be strictly incrementing. You MUST NEVER send the same or lower number ever again or it will answer with "Invalid call". It will remember the value of the last nonce you sent indefinitely on a per-key basis (to reset this to 0 again you need a new key)
Last week I found by accident that it accepted non-numeric values without complaint, I told MagicalTux about this security bug and I assume he has fixed it now.
Last week I found by accident that it accepted non-numeric values without complaint, I told MagicalTux about this security bug and I assume he has fixed it now.
Yes, now it works again. Big Thanks!
But what is the difference& between sockeio and socketio-beta?
Why do I need to use GET request and all those things you describe?
Why do I need to use GET request and all those things you describe?
There seems a bug in the server, MT has promised to fix it. Therefore its only "beta" and he hasn't replaced the old server yet. I have no idea when this will happen. Some client implementations can handle this bug (the bug is that both requests MUST come over the same socket), some other clients cant (until he fixes this bug). The ones who happen to have a working client (like me) are lucky because this server seems much more reliable and has had 0% downtime so far.
The GET requests and basically the entire last few pages of this thread with all their technical details about socket.io and GET requests and frames are only of interest if one must write his own socket.io/websocket client from scratch because there exist no socket.io libraries for many (most) programming languages except JavaScript (and this is because socket.io is not an internet protocol and not a standard of any kind, socket.io is just a private hobby project from some guys who hacked some code together as a non-solution for a non-problem and uploaded it to github and unfortunately MagicalTux somehow accidentally found this hobby-protocol and thought it was a good idea to use it for MtGox-API and now we all have to suffer from the consequences).
Hey, guys, I am sorry but can you advice?
is it possible or will it be possible to use just plain nodjs socket.io client to connect to socketio-beta?
I use socket.io-client on server to fetch mtgox socketio stream. I do not have any http requests - client is connecting on its own.
But what is the difference& between sockeio and socketio-beta?
Why do I need to use GET request and all those things you describe?
thanks
is it possible or will it be possible to use just plain nodjs socket.io client to connect to socketio-beta?
I use socket.io-client on server to fetch mtgox socketio stream. I do not have any http requests - client is connecting on its own.
But what is the difference& between sockeio and socketio-beta?
Why do I need to use GET request and all those things you describe?
thanks
prof7bit
Thanks! I have not changed anything in the message signing function, my nonce is MD5 of timestamp.
Thanks! I have not changed anything in the message signing function, my nonce is MD5 of timestamp.
There we have the reason why it is not working. It *MUST* be a numeric value and it MUST be strictly incrementing. You MUST NEVER send the same or lower number ever again or it will answer with "Invalid call". It will remember the value of the last nonce you sent indefinitely on a per-key basis (to reset this to 0 again you need a new key)
Last week I found by accident that it accepted non-numeric values without complaint, I told MagicalTux about this security bug and I assume he has fixed it now.
I've got a little further initiating the beta socketio stream, below is my TRACE.
############SOCKET OPENED BY CLIENT
############TX:
GET /socket.io/1 HTTP/1.1
Host: socketio-beta.mtgox.com:80
Connection: keep-alive
############RX:
HTTP/1.1 200 OK
Content-Type: text/plain
Date: Mon, 25 Mar 2013 03:48:51 GMT
Connection: keep-alive
Transfer-Encoding: chunked
############TX:
GET /socket.io/1/websocket/ HTTP/1.1
Upgrade: websocket
Connection: Upgrade
Host: socketio-beta.mtgox.com:80
Origin: socketio-beta.mtgox.com:80
############RX:
53
GEyZU7Bw-ivuvI6CZon3:60:60:websocket,flashsocket,htmlfile,xhr-polling,jsonp-polling
0
############TX:
GET /socket.io/1/websocket/GEyZU7Bw-ivuvI6CZon3 HTTP/1.1
Upgrade: websocket
Connection: Upgrade
Host: socketio-beta.mtgox.com:80
Origin: socketio-beta.mtgox.com:80
Sec-WebSocket-Key: UWaphFPiSqq3f2gOmaD5Sg==
Sec-WebSocket-Version: 13
############SOCKET CLOSED BY HOST
The server closes the socket when I dont give it correct Sec-WebSocket-Key.
So I'm guessing in my last TX above I need to derive Sec-WebSocket-Key somehow.
Should i do it like mentioned in websocket wiki?:
############SOCKET OPENED BY CLIENT
############TX:
GET /socket.io/1 HTTP/1.1
Host: socketio-beta.mtgox.com:80
Connection: keep-alive
############RX:
HTTP/1.1 200 OK
Content-Type: text/plain
Date: Mon, 25 Mar 2013 03:48:51 GMT
Connection: keep-alive
Transfer-Encoding: chunked
############TX:
GET /socket.io/1/websocket/ HTTP/1.1
Upgrade: websocket
Connection: Upgrade
Host: socketio-beta.mtgox.com:80
Origin: socketio-beta.mtgox.com:80
############RX:
53
GEyZU7Bw-ivuvI6CZon3:60:60:websocket,flashsocket,htmlfile,xhr-polling,jsonp-polling
0
############TX:
GET /socket.io/1/websocket/GEyZU7Bw-ivuvI6CZon3 HTTP/1.1
Upgrade: websocket
Connection: Upgrade
Host: socketio-beta.mtgox.com:80
Origin: socketio-beta.mtgox.com:80
Sec-WebSocket-Key: UWaphFPiSqq3f2gOmaD5Sg==
Sec-WebSocket-Version: 13
############SOCKET CLOSED BY HOST
The server closes the socket when I dont give it correct Sec-WebSocket-Key.
So I'm guessing in my last TX above I need to derive Sec-WebSocket-Key somehow.
Should i do it like mentioned in websocket wiki?:
Quote
The client sends a Sec-WebSocket-Key which is base64 encoded. To form a response, the magic string 258EAFA5-E914-47DA-95CA-C5AB0DC85B11 is appended to this (undecoded) key. The resulting string is then hashed with SHA-1, then base64 encoded. Finally, the resulting reply occurs in the header Sec-WebSocket-Accept.
There is something wrong. you are sending 3 GET requests instead of 2, the first one looks ok, the second one is wrong.
The sec-websocket-key is a random number (i believe 16 bytes without looking into the RFC) base64 encoded. the server does some simple transformations with it and returns it as sec-websocket-accept. You can basically throw this away, you could use it to detect that there is no caching proxy in between because for every different sec-websocket-key you will get a different sec-websocket-accept. So you could check if the server response really belongs to your request and is not something cached but you could just as well simply ignore it because no sane webcache would ever attempt to cache a websocket connection anyways.
Once you are past this initial handshake the real fun (or no fun at all) will begin: You will have to implement the framing and masking as defined in RFC 6455. This initial handshake will look like childs play compared to what is awaiting you in this RFC.
I've got a little further initiating the beta socketio stream, below is my TRACE.
############SOCKET OPENED BY CLIENT
############TX:
GET /socket.io/1 HTTP/1.1
Host: socketio-beta.mtgox.com:80
Connection: keep-alive
############RX:
HTTP/1.1 200 OK
Content-Type: text/plain
Date: Mon, 25 Mar 2013 03:48:51 GMT
Connection: keep-alive
Transfer-Encoding: chunked
############TX:
GET /socket.io/1/websocket/ HTTP/1.1
Upgrade: websocket
Connection: Upgrade
Host: socketio-beta.mtgox.com:80
Origin: socketio-beta.mtgox.com:80
############RX:
53
GEyZU7Bw-ivuvI6CZon3:60:60:websocket,flashsocket,htmlfile,xhr-polling,jsonp-polling
0
############TX:
GET /socket.io/1/websocket/GEyZU7Bw-ivuvI6CZon3 HTTP/1.1
Upgrade: websocket
Connection: Upgrade
Host: socketio-beta.mtgox.com:80
Origin: socketio-beta.mtgox.com:80
Sec-WebSocket-Key: UWaphFPiSqq3f2gOmaD5Sg==
Sec-WebSocket-Version: 13
############SOCKET CLOSED BY HOST
The server closes the socket when I dont give it correct Sec-WebSocket-Key.
So I'm guessing in my last TX above I need to derive Sec-WebSocket-Key somehow.
Should i do it like mentioned in websocket wiki?:
############SOCKET OPENED BY CLIENT
############TX:
GET /socket.io/1 HTTP/1.1
Host: socketio-beta.mtgox.com:80
Connection: keep-alive
############RX:
HTTP/1.1 200 OK
Content-Type: text/plain
Date: Mon, 25 Mar 2013 03:48:51 GMT
Connection: keep-alive
Transfer-Encoding: chunked
############TX:
GET /socket.io/1/websocket/ HTTP/1.1
Upgrade: websocket
Connection: Upgrade
Host: socketio-beta.mtgox.com:80
Origin: socketio-beta.mtgox.com:80
############RX:
53
GEyZU7Bw-ivuvI6CZon3:60:60:websocket,flashsocket,htmlfile,xhr-polling,jsonp-polling
0
############TX:
GET /socket.io/1/websocket/GEyZU7Bw-ivuvI6CZon3 HTTP/1.1
Upgrade: websocket
Connection: Upgrade
Host: socketio-beta.mtgox.com:80
Origin: socketio-beta.mtgox.com:80
Sec-WebSocket-Key: UWaphFPiSqq3f2gOmaD5Sg==
Sec-WebSocket-Version: 13
############SOCKET CLOSED BY HOST
The server closes the socket when I dont give it correct Sec-WebSocket-Key.
So I'm guessing in my last TX above I need to derive Sec-WebSocket-Key somehow.
Should i do it like mentioned in websocket wiki?:
Quote
The client sends a Sec-WebSocket-Key which is base64 encoded. To form a response, the magic string 258EAFA5-E914-47DA-95CA-C5AB0DC85B11 is appended to this (undecoded) key. The resulting string is then hashed with SHA-1, then base64 encoded. Finally, the resulting reply occurs in the header Sec-WebSocket-Accept.
prof7bit
Thanks! I have not changed anything in the message signing function, my nonce is MD5 of timestamp. API key and secret are correct. Multiple attempts give same result "Invalid call". So bad that the server does not provide a detailed description of the error.
Thanks! I have not changed anything in the message signing function, my nonce is MD5 of timestamp. API key and secret are correct. Multiple attempts give same result "Invalid call". So bad that the server does not provide a detailed description of the error.
My bot received trades with timestamps from future (~ 5 seconds). But my system clock it is precisely synchronized with atomic time (~0.5 ms) through several NTP servers.
Also now all calls (order/lag or private/info) always rejected with message "Invalid call". Can I retrieve more info for this error or API have limited verbosity?
Also now all calls (order/lag or private/info) always rejected with message "Invalid call". Can I retrieve more info for this error or API have limited verbosity?
Did you change your algorithm that creates your nonce? If you did then you might need a fresh API key. Nonce problems were the only problems I have witnessed causing sudden mysterious "Invalid call" problems.
Also if you send a lot of commands immediately after each other they will be processed in parallel on the server and this can mess up the chronological order (basically a race condition on their server) that will cause sporadic "Invalid call" (because the nonce not strictly incrementing anymore if the sequence is messed up). The only way around it is to answer each "Invalid call" by just sending the same command (with a new nonce of course) again... until it succeeds.
My bot received trades with timestamps from future (~ 5 seconds). But my system clock it is precisely synchronized with atomic time (~0.5 ms) through several NTP servers.
Also now all calls (order/lag or private/info) always rejected with message "Invalid call". Can I retrieve more info for this error or API have limited verbosity?
Also now all calls (order/lag or private/info) always rejected with message "Invalid call". Can I retrieve more info for this error or API have limited verbosity?
@prof7bit,
Thanks so much for your great reply!
I explored thru telnet first just to get my toes wet.
I am working ontop of WindowsMFC CAsyncSocket class and I got the websocket going OK (when server is up).
I'll set about trying to get CAsyncSocket to act as socket.io now and manage the binary frames as they arrive.
I'm not very experienced when it comes to streaming stuff, you just helped me bigtime!
Thanks so much for your great reply!
I explored thru telnet first just to get my toes wet.
I am working ontop of WindowsMFC CAsyncSocket class and I got the websocket going OK (when server is up).
I'll set about trying to get CAsyncSocket to act as socket.io now and manage the binary frames as they arrive.
I'm not very experienced when it comes to streaming stuff, you just helped me bigtime!
So this is a problem of certificates and not of socket.io-client version?
Or which version of socket.io-client should I use?
Or which version of socket.io-client should I use?
Maybe not use SSL, use unencrypted on port 80
Tried that already - the same result - errors
Jump to: