[BETA] MTGox websocket API, testers wanted - page 11.

sirk390

full member

Activity: 153

Merit: 100

Quote from: Clark on April 01, 2012, 03:53:18 PM

Quote from: sirk390 on April 01, 2012, 03:41:54 PM

Thanks but what subscribe URL do you mean ?

Sorry, it should have been ?Currency=

Like this:

https://socketio.mtgox.com/mtgox?Currency=EUR

Thanks you! It works now although i'm not using exactly the same URL.
I'm using https://socketio.mtgox.com/socket.io/1/?Currency=EUR .

Clark

hero member

Activity: 548

Merit: 502

So much code.

Quote from: sirk390 on April 01, 2012, 03:41:54 PM

Thanks but what subscribe URL do you mean ?

Sorry, it should have been ?Currency=

Like this:

https://socketio.mtgox.com/mtgox?Currency=EUR

sirk390

full member

Activity: 153

Merit: 100

Quote from: Clark on April 01, 2012, 11:33:18 AM

Quote from: sirk390 on April 01, 2012, 10:55:33 AM

Does anyone know if it is possible to receive market depth updates for other currencies (EUR) in the socket.io/websocket API?
I found this list of channels: https://en.bitcoin.it/wiki/Talk:MtGox/API but couldn't subscribe using the API.
The API only takes a "type" argument (Possible values: "trades, ticker, depth") e.g. {"op":"mtgox.subscribe","type":"trades"}
but there is no documented value for "depth.EUR".

You can subscribe with ?Channel=EUR added to the subscribe URL. I have a working EUR order book, chart, and time & sales on my website. Simply choose EUR from the drop down menu at the top: bitcoin.clarkmoody.com

Thanks but what subscribe URL do you mean ?
I'm connection with python using something like this https://github.com/osmosis79/gox_socketio_py .
It sends a request to "https://socketio.mtgox.com/socket.io/1" and then opens a websocket to "wss://socketio.mtgox.com/socket.io/1/1/websocket/{id}"

Clark

hero member

Activity: 548

Merit: 502

So much code.

Quote from: sirk390 on April 01, 2012, 10:55:33 AM

Does anyone know if it is possible to receive market depth updates for other currencies (EUR) in the socket.io/websocket API?
I found this list of channels: https://en.bitcoin.it/wiki/Talk:MtGox/API but couldn't subscribe using the API.
The API only takes a "type" argument (Possible values: "trades, ticker, depth") e.g. {"op":"mtgox.subscribe","type":"trades"}
but there is no documented value for "depth.EUR".

You can subscribe with ?Channel=EUR added to the subscribe URL. I have a working EUR order book, chart, and time & sales on my website. Simply choose EUR from the drop down menu at the top: bitcoin.clarkmoody.com

sirk390

full member

Activity: 153

Merit: 100

Hi,

Does anyone know if it is possible to receive market depth updates for other currencies (EUR) in the socket.io/websocket API?
I found this list of channels: https://en.bitcoin.it/wiki/Talk:MtGox/API but couldn't subscribe using the API.
The API only takes a "type" argument (Possible values: "trades, ticker, depth") e.g. {"op":"mtgox.subscribe","type":"trades"}
but there is no documented value for "depth.EUR".

molecular

donator

Activity: 2772

Merit: 1019

Quote from: runeks on March 27, 2012, 10:18:58 AM

Here's my version. Just yesterday I found out that I need to handle exceptions in "response = urllib2.urlopen(req)" line in connect(). I couldn't understand why the script had stopped, but found out it was because this line had thrown a ConnectionRefused exception. So it's continually improving as I find out the various ways it can fail. Still haven't gotten rid of the SSL errors though.

https://github.com/runeksvendsen/traidor/blob/master/socketio.py

thanks, so cool.

it bails in line 94 for me, though, with:

Code:

Exception in thread Thread-4:
Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 530, in __bootstrap_inner
self.run()
File "/usr/lib/python2.7/threading.py", line 483, in run
self.__target(*self.__args, **self.__kwargs)
File "/home/nick/bitcoin/traidor/socketio.py", line 94, in thread_func
S.callback(S, msg[10:])
TypeError: onMessage() takes exactly 2 arguments (3 given)

this is because you added a parameter to the callback function and I still pass a callback function that takes one parameter less.

note that the "test_callback" function is in global scope, not in the class.

suggestion: If you need the instance of the socket.io in testing mode you can just use the global variable "sio" instead of passing S as follows (changes marked in bold):

Quote

def test_callback(msg):
import ast
print 'msg: ', msg
#convert received message string into dictionary
msg_dict = ast.literal_eval(msg)
#print msg_dict
if msg_dict['op'] == 'subscribe':
if msg_dict['channel'] == 'd5f06780-30a8-4a48-a2f8-7ed181b4a13f' or msg_dict['channel'] == '24e67e0d-1cad-4cc0-9e7a-f8523ef460fe':
sio.unsubscribe(msg_dict['channel'])

that way everything remains compatible and I can use your code

runeks

legendary

Activity: 980

Merit: 1008

Here's my version. Just yesterday I found out that I need to handle exceptions in "response = urllib2.urlopen(req)" line in connect(). I couldn't understand why the script had stopped, but found out it was because this line had thrown a ConnectionRefused exception. So it's continually improving as I find out the various ways it can fail. Still haven't gotten rid of the SSL errors though.

https://github.com/runeksvendsen/traidor/blob/master/socketio.py

molecular

donator

Activity: 2772

Merit: 1019

Quote from: runeks on March 25, 2012, 03:38:58 PM

Quote from: molecular on March 20, 2012, 09:37:59 AM

Quote from: runeks on March 20, 2012, 06:09:30 AM

The above code doesn't work. Don't bother trying it out. break'ing isn't enough, we need to raise an exception to exit the thread. I've written something else that I have yet to test. I will report back how it went when I have tested it.

Thanks so much for your analysis and help so far, runeks.

I think you are correct in saying we need to "decommission" those 2 threads on disconnect.

I'm currently trying

Code:

raise SystemExit

EDIT: looks good so far, seeing some "error: [Errno 32] Broken pipe" and it reconnects

EDIT2: great, more "broken pipes", but still running (exeptionally long). So I figure with the help of runeks, I got the threading issues fixed.

EDIT3: here's socketio.py snapshot of the current version: http://pastebin.com/b5CmTy8a, git is here (might change in the future): https://github.com/molecular/traidor/blob/master/socketio.py

Hi!

Totally forgot about this. Thanks for your token. I've modified your code a bit to see if I can get it completely robust. I'm basically doing the controlling from thread_func instead, because exceptions are almost always detected here first (because "msg = S.ws.recv()" is always hanging there just waiting for a message to come). This means we won't experience a 0-15 second delay before reconnecting (by having the keepalive thread do it). It adds some more complexity though. I still experience the occasional SSL error "EOF occurred in violation of protocol" (besides the ConnectionClosedException) nonetheless. But it seems to be running very robustly so far. No running amok like the previous script. The occasional error comes, but never multiple errors in a row. I will release the code when it has proved to be stable. I'm not experiencing any "Broken pipe" messages though. Only the two aforementioned errors.

I remember while coding it it didn't feel right to detect for reconnect (only) in the keepalive thread.

Thanks for fixing that, please pastebin your version if you consider it robust.

My one is running stably for days now. I might just not care about the 15 seconds.

runeks

legendary

Activity: 980

Merit: 1008

Quote from: molecular on March 20, 2012, 09:37:59 AM

Quote from: runeks on March 20, 2012, 06:09:30 AM

The above code doesn't work. Don't bother trying it out. break'ing isn't enough, we need to raise an exception to exit the thread. I've written something else that I have yet to test. I will report back how it went when I have tested it.

Thanks so much for your analysis and help so far, runeks.

I think you are correct in saying we need to "decommission" those 2 threads on disconnect.

I'm currently trying

Code:

raise SystemExit

EDIT: looks good so far, seeing some "error: [Errno 32] Broken pipe" and it reconnects

EDIT2: great, more "broken pipes", but still running (exeptionally long). So I figure with the help of runeks, I got the threading issues fixed.

EDIT3: here's socketio.py snapshot of the current version: http://pastebin.com/b5CmTy8a, git is here (might change in the future): https://github.com/molecular/traidor/blob/master/socketio.py

Hi!

Totally forgot about this. Thanks for your token. I've modified your code a bit to see if I can get it completely robust. I'm basically doing the controlling from thread_func instead, because exceptions are almost always detected here first (because "msg = S.ws.recv()" is always hanging there just waiting for a message to come). This means we won't experience a 0-15 second delay before reconnecting (by having the keepalive thread do it). It adds some more complexity though. I still experience the occasional SSL error "EOF occurred in violation of protocol" (besides the ConnectionClosedException) nonetheless. But it seems to be running very robustly so far. No running amok like the previous script. The occasional error comes, but never multiple errors in a row. I will release the code when it has proved to be stable. I'm not experiencing any "Broken pipe" messages though. Only the two aforementioned errors.

Ente

legendary

Activity: 2126

Merit: 1001

Quote from: Clark on March 24, 2012, 10:14:01 AM

According to the wiki API reference, you should be able to do authentication stuff over the WebSocket interface.

Yep:

Quote

Authenticated commands
These commands require an API key and secret pair to sign requests. Any of the HTTP API version 1 methods can be called.

https://en.bitcoin.it/wiki/MtGox/API/Streaming#Authenticated_commands

..with more details and a php example too

Ente

Clark

hero member

Activity: 548

Merit: 502

So much code.

According to the wiki API reference, you should be able to do authentication stuff over the WebSocket interface.

molecular

donator

Activity: 2772

Merit: 1019

Quote from: Ente on March 24, 2012, 09:35:44 AM

Quote from: molecular on March 24, 2012, 09:17:24 AM

Quote from: Ente on March 24, 2012, 08:47:40 AM

I tried several hours to create signed commands to send over the socket instead of the old api.

I used this: https://bitcointalksearch.org/topic/m.592388, it works.

Yes, I used that too, for my old http api. I try to update to socket-only, which supposedly is almost the same.. Almost, but for me not close enough *sigh*

Thanks for replying!

Ente

"socket-only"? I know of the "new" and "old" (1, 2) http apis. And I know of socket.io interface for depth/trades (no auth necessary). Care to explain?

Ente

legendary

Activity: 2126

Merit: 1001

Quote from: molecular on March 24, 2012, 09:17:24 AM

Quote from: Ente on March 24, 2012, 08:47:40 AM

I tried several hours to create signed commands to send over the socket instead of the old api.

I used this: https://bitcointalksearch.org/topic/m.592388, it works.

Yes, I used that too, for my old http api. I try to update to socket-only, which supposedly is almost the same.. Almost, but for me not close enough *sigh*

Thanks for replying!

Ente

molecular

donator

Activity: 2772

Merit: 1019

Quote from: Ente on March 24, 2012, 08:47:40 AM

I tried several hours to create signed commands to send over the socket instead of the old api.

I used this: https://bitcointalksearch.org/topic/m.592388, it works.

Ente

legendary

Activity: 2126

Merit: 1001

I tried several hours to create signed commands to send over the socket instead of the old api.
No luck at all. Lastly, I tried to "translate" the php example to python from https://en.bitcoin.it/wiki/MtGox/API/Streaming#Authenticated_commands which doesnt look too complicated..

Code:

def get_nonce():
return int(time.time()*100000)

def encoder(raw):
dec = json.dumps(raw)
return dec

def sign_data(secret, data):
return base64.b64encode(str(HMAC(secret, data, sha512).digest()))

call = "1/info.php"
params = "testparams"
item = "BTC"
currency = "BTC"
id = "1234567890"

Code:

query1 = {'call':call, 'params':params, 'item':item, 'currency':currency, 'id':id, 'nonce':get_nonce()}

Quote

{'nonce': 133259594451109, 'item': 'BTC', 'currency': 'BTC', 'params': 'testparams', 'call': '1/info.php', 'id': '1234567890'}

Code:

query2 = encoder(query1)

Quote

{"nonce": 133259594451109, "item": "BTC", "currency": "BTC", "params": "testparams", "call": "1/info.php", "id": "1234567890"}

Code:

sign = sign_data(secret, query2)

Quote

xjAc1rbh0T4xFVJT/eZo/nxI4vXxiVmaZxbyM6a5RQtHqld2RW1y9HYdE9z/TlM5e9++P+MMWEd9YFdf6Nenqg==

Code:

query3 = sign, query2

Quote

('xjAc1rbh0T4xFVJT/eZo/nxI4vXxiVmaZxbyM6a5RQtHqld2RW1y9HYdE9z/TlM5e9++P+MMWEd9YFdf6Nenqg==', '{"nonce": 133259594451109, "item": "BTC", "currency": "BTC", "params": "testparams", "call": "1/info.php", "id": "1234567890"}')

Code:

call = {'op':'call', 'call':base64.b64encode(str(query3)), 'id':id, 'context':'mtgox.com'}

Quote

{'call': 'KCd4akFjMXJiaDBUNHhGVkpUL2Vaby9ueEk0dlh4aVZtYVp4YnlNNmE1UlF0SHFsZDJSVzF5OUhZZEU 5ei9UbE01ZTkrK1ArTU1XRWQ5WUZkZjZOZW5xZz09JywgJ3sibm9uY2UiOiAxMzMyNTk1OTQ0NTExMD ksICJpdGVtIjogIkJUQyIsICJjdXJyZW5jeSI6ICJCVEMiLCAicGFyYW1zIjogInRlc3RwYXJhbXMiL CAiY2FsbCI6ICIxL2luZm8ucGhwIiwgImlkIjogIjEyMzQ1Njc4OTAifScp', 'id': '1234567890', 'context': 'mtgox.com', 'op': 'call'}

Finally I take this last message and paste it in my socket.send:

Code:

socket.send("4:::{'call': 'KCd4akFjMXJiaDBUNHhGVkpUL2Vaby9ueEk0dlh4aVZtYVp4YnlNNmE1UlF0SHFsZDJSVzF5OUhZZEU5ei9UbE01ZTkrK1ArTU1XRWQ5WUZkZjZOZW5xZz09JywgJ3sibm9uY2UiOiAxMzMyNTk1OTQ0NTExMDksICJpdGVtIjogIkJUQyIsICJjdXJyZW5jeSI6ICJCVEMiLCAicGFyYW1zIjogInRlc3RwYXJhbXMiLCAiY2FsbCI6ICIxL2luZm8ucGhwIiwgImlkIjogIjEyMzQ1Njc4OTAifScp', 'id': '1234567890', 'context': 'mtgox.com', 'op': 'call'}")

Non-descriptive error-reply:

Quote

{u'debug': {u'data': False, u'uuid': u'249bcb53-ccb0-43f1-bac6-e5ae0748e2e1', u'op': u'client'}, u'message': u'Unknown command', u'success': False, u'op': u'remark'}

If someone has enough mercy to have a closer look I would be really happy!
Since this probably takes more than just a few moments, dont forget to quote your bitcoin-address too ;-)

Ente

/who now goes doing some woodwork to clear his head from this

molecular

donator

Activity: 2772

Merit: 1019

Quote from: runeks on March 20, 2012, 06:09:30 AM

The above code doesn't work. Don't bother trying it out. break'ing isn't enough, we need to raise an exception to exit the thread. I've written something else that I have yet to test. I will report back how it went when I have tested it.

Thanks so much for your analysis and help so far, runeks.

I think you are correct in saying we need to "decommission" those 2 threads on disconnect.

I'm currently trying

Code:

raise SystemExit

EDIT: looks good so far, seeing some "error: [Errno 32] Broken pipe" and it reconnects

EDIT2: great, more "broken pipes", but still running (exeptionally long). So I figure with the help of runeks, I got the threading issues fixed.

EDIT3: here's socketio.py snapshot of the current version: http://pastebin.com/b5CmTy8a, git is here (might change in the future): https://github.com/molecular/traidor/blob/master/socketio.py

runeks

legendary

Activity: 980

Merit: 1008

The above code doesn't work. Don't bother trying it out. break'ing isn't enough, we need to raise an exception to exit the thread. I've written something else that I have yet to test. I will report back how it went when I have tested it.

runeks

legendary

Activity: 980

Merit: 1008

Quote from: molecular on March 20, 2012, 03:09:32 AM

I'm experiencing this, too Sad

I also see problems with threads. I don't know if my thread creation/burial works corretly. When the thread_func() exits that thread should be dead and recycled or whatever, right? Or do I need to still join it?

I tried to make reconnect more robust (at some point I get some "cannot create thread" exception and program exit), but so far failed.

I don't know wether the ConnectionClosed and "SLL" errors are client side fault.

Any insight welcome!

One thing that springs to mind is that upon error, we need to make sure that the existing threads exit, before we start new ones (through calling S.connect() in keepalive_func).
As it is now, keepalive_func will keep running after it has tried re-connecting, and as far as I can see, thread_func will keep running as well. So for each reconnect, we have one extra thread of both keepalive and thread_func running, sending and receiving data on the same socket:

Code:

def keepalive_func(S):
while S.run:
try:
S.ws.send('2::');
except:
if S.run:
print 'error sending keepalive socket.io, trying reconnect'
S.connect() #this thread will keep running after this. connect creates a new thread_func, which creates a new keepalive_func
else:
print 'exiting socket.io keepalive thread'
time.sleep(S.heartbeat_interval)

Code:

def thread_func(S):
print 'SocketIO: websocket thread started'

my_url = 'wss://' + S.url + "/1/websocket/" + S.id

S.ws = create_connection(my_url)

S.run = True
S.ws.send('1::/mtgox')

# start keepalive thread
S.keepalive_thread = Thread(target = S.keepalive_func)
S.keepalive_thread.setDaemon(True)
S.keepalive_thread.start()

msg = S.ws.recv()
while msg is not None and S.run:
if msg[:10] == "4::/mtgox:":
S.callback(msg[10:])
msg = S.ws.recv()
S.ws.close()

As far as I can tell, this can be fixed by breaking out of the while loop in both threads in case of error, replacing the relevant parts like so:

Code:

def keepalive_func(S):
while S.run:
try:
S.ws.send('2::');
except:
if S.run:
print 'error sending keepalive socket.io, closing existing connection...'
S.ws.close()
print '\ttrying reconnect...'
S.connect()
raise
break
else:
print 'exiting socket.io keepalive thread'
time.sleep(S.heartbeat_interval)

and in thread_func:

Code:

try:
msg = S.ws.recv()
except:
raise
break
S.ws.close()

I will see how this runs and report back.

Quote from: molecular on March 20, 2012, 03:09:32 AM

Quote from: runeks on March 20, 2012, 02:37:38 AM

It also looks like I'm receiving corrupted/incorrect data on the socket sometimes:

Code:

08:54:02 4.79 10.0 (*10.0)
08:54:08 4.79 9.9243215 (*19.9)
08:54:11 4.78991 96.0 (4.79)
08:54:12 4.78941 27.0 (4.79)
08:54:12 3.09251 17.0 (4.50)
08:54:12 4.78902 17.0 (4.50)
08:55:16 3.09251 16.65379914 (4.22)
08:55:16 4.78902 16.65379914 (4.22)
08:55:16 4.78272 1.086e-05 (4.22)

That's two trades at $3.09251/BTC happening in between trades at $4.78902/BTC... That shouldn't be possible.

So, not sure what to do about this... seems something isn't right somewhere (duh!).

You might be looking at differenct currencies? 3.09 could be EUR or GBP or something?

Indeed I was. I had just accidentally switched off the ignore-non-USD-trades option. Fixed now.

bzzard

sr. member

Activity: 398

Merit: 250

Quote from: molecular on March 20, 2012, 03:09:32 AM

I havent's seen weird stuff in the trade channel for a long time myself (filtering for USD). My depth stuff is still not syncing correctly and my depth table just gets cluttered with obviously old entries that are not removed after a while. I'm not sure wether this is due to websocket implementation or maybe I'm doing something wrong.

Again, any insight? Please post.

I've tested this topic in details and all I can say that lately (~2 weeks) depth data from socketio is useless. Don't trust it.

molecular

donator

Activity: 2772

Merit: 1019

Topic: [BETA] MTGox websocket API, testers wanted - page 11. (Read 77671 times)