[BETA] MTGox websocket API, testers wanted - page 10.

runeks

legendary

Activity: 980

Merit: 1008

I'm seeing:

Code:

01:21:39 > Got 3864 asks and 5085 bids.

01:21:38 > Synchronizing the order book.

01:21:16 > Subscribed to depth

01:21:16 > Subscribed to ticker

01:21:16 > Subscribed to trades

01:21:16 > MtGox: Connected.

01:21:16 > Attempting connection over websocket

Looks to me like it's working. What am I missing?

Has Mt. Gox gone public about why it's closed down, as you say?

EDIT: It's working for your site because it seems to be up now: works for me too! Yay Cheesy

Oh live feed, how I've missed you.

Clark

hero member

Activity: 548

Merit: 502

So much code.

Quote from: runeks on August 20, 2012, 05:24:23 PM

Is it not working for anyone else, or is it just my corner of the internet that's broken?

Take a glance at the console on the bottom of bitcoin.clarkmoody.com, and you will see the same thing. Streaming data has been disabled (by Gox) for most of the afternoon on my website.

So you're not alone.

runeks

legendary

Activity: 980

Merit: 1008

Is the websocket API still working for everyone else? It worked for me fine until today. Now this is the first message I receive after connecting:

Code:

{'suggest': 'retry', 'error': 'command failed', 'op': 'error'}

This is when I connect to https://socketio.mtgox.com/socket.io as I've always been doing. When I connect to https://socketio.mtgox.com/mtgox the mtgox_io script fails because it can't read the heartbeat interval from the first piece of data received when connecting to the server.
The data received in mtgox_io's connect() function at the line "r = response.read().split(':')" from https://socketio.mtgox.com/socket.io looks like this:

Code:

['9861138011216185358', '60', '60', 'websocket,htmlfile,xhr-polling,jsonp-polling']

While, when connecting to https://socketio.mtgox.com/mtgox, it's:

Code:

['Welcome to Tibanne Websocket']

A welcoming the script doesn't seem to appreciate.

Is it not working for anyone else, or is it just my corner of the internet that's broken?

molecular

donator

Activity: 2772

Merit: 1019

Quote from: daybyter on May 24, 2012, 02:48:06 PM

Scary....maybe you are my long lost twin brother, or so...

I already send a PM to Seal and waiting for some reply with an email, so I can send him some code...

Part of my code is a TradeSite interface (just started), that I already implement for MtGox and BtcE. So you can loop over the sites, combine all trades, compare fees etc.

be sure to tell this thread (or me) when you have some shareable code. I'm interested Wink

molecular

donator

Activity: 2772

Merit: 1019

Quote from: Seal on May 24, 2012, 08:00:40 PM

I dont think my current host offers terminal sessions so ideally it'd run in a standard LAMP build environment.

... either that or I'll have to purchase some sort of virtual box.

oh, ok. hmm. I'm very content with one of these which I pay in BTC: http://www.cinfu.com/vps/

Seal

donator

Activity: 848

Merit: 1078

I'm here! Yes, I'll reply to your PM now daybyter.

Thanks for the advice molecular. I've been developing trading scripts for a while, mostly in PHP, however I've got to admit, socket.io just confuses me.

It most likely won't be serving client pages. However the script I'd like to write will be based on a server and receiving the mtgox stream, whereby it could then decide on trades itself.

Quote from: molecular on May 24, 2012, 06:48:09 AM

If not, I'd probably write an init-script that just runs "java -jar myjar.jar" and do scheduling within the app. To be honest, I might just start it in a screen (gnu screen) session if I'm the only one depending on it Wink

I dont think my current host offers terminal sessions so ideally it'd run in a standard LAMP build environment.

... either that or I'll have to purchase some sort of virtual box.

daybyter

legendary

Activity: 965

Merit: 1000

Scary....maybe you are my long lost twin brother, or so...

I already send a PM to Seal and waiting for some reply with an email, so I can send him some code...

Part of my code is a TradeSite interface (just started), that I already implement for MtGox and BtcE. So you can loop over the sites, combine all trades, compare fees etc.

molecular

donator

Activity: 2772

Merit: 1019

Quote from: daybyter on May 24, 2012, 06:51:45 AM

Perfect. We are thinking exactly in the same lines.

I started to write the Java app in a way, that I could run in client or server mode. In client mode, a GUI comes up. In server mode, there's no GUI, so it could be started in an init.d script.

Drop me a PM, if you want to discuss more details.

In case you're talking to me: maybe better talk to Seal, I was merely giving ideas to Seal and have currently nothing of the sort planned (except maybe (but unlikely (unless someone coughs up coins)) at some point trading/bot infrastructure and exchange abstraction layer in java)

daybyter

legendary

Activity: 965

Merit: 1000

Perfect. We are thinking exactly in the same lines.

I started to write the Java app in a way, that I could run in client or server mode. In client mode, a GUI comes up. In server mode, there's no GUI, so it could be started in an init.d script.

Drop me a PM, if you want to discuss more details.

molecular

donator

Activity: 2772

Merit: 1019

Quote from: Seal on May 23, 2012, 07:39:31 PM

Quote from: molecular on May 23, 2012, 12:05:29 PM

java, node.js, write timer loop yourself in php or use some php scheduling stuff? I don't know. For the kind of stuff we're looking at here, there's good APIs in every language, so it really boils down to personal language preference, I guess. I'd probably use java because I'm most comfortable with it. Unfortunately I cannot recommend my "traidor" code, because the orderbook still gets out of sync.

One remark: I don't know about the stability of the socket.io feed nowadays, but maybe you should implement a fallback to http api?

molecular - I currently use a scheduler to execute a PHP call every minute using the http api, however I'm interested in having a live feed for the best prices as they are added to the orderbook. To my knowledge, PHP isnt brilliant at handling streaming feeds and I'm yet to find a way of doing this server-side.

I could stand corrected though, I'm just waiting for a script-kiddy to tell me otherwise.

Whats java like at handling streaming data?

Are you talking about streaming from your server to some clients or about receiving mtgox socket.io/websocket stream?

If the latter: http://stackoverflow.com/questions/5783086/java-socket-io-client
If the former: http://code.google.com/p/socketio-java/

I never tried any of these, so I don't know how well they handle things.

Quote from: Seal on May 23, 2012, 07:39:31 PM

How would you run a script like that on a server?

If it's also serving web-pages I'd probably run it in the context of a servlet container like tomcat or jetty.

If not, I'd probably write an init-script that just runs "java -jar myjar.jar" and do scheduling within the app. To be honest, I might just start it in a screen (gnu screen) session if I'm the only one depending on it Wink

daybyter

legendary

Activity: 965

Merit: 1000

I would simply run it as a daemon on the server. It would poll every x seconds/minutes . I'll drop you a mail with more details and some code (cannot attach anything here).

Seal

donator

Activity: 848

Merit: 1078

Quote from: molecular on May 23, 2012, 12:05:29 PM

java, node.js, write timer loop yourself in php or use some php scheduling stuff? I don't know. For the kind of stuff we're looking at here, there's good APIs in every language, so it really boils down to personal language preference, I guess. I'd probably use java because I'm most comfortable with it. Unfortunately I cannot recommend my "traidor" code, because the orderbook still gets out of sync.

One remark: I don't know about the stability of the socket.io feed nowadays, but maybe you should implement a fallback to http api?

molecular - I currently use a scheduler to execute a PHP call every minute using the http api, however I'm interested in having a live feed for the best prices as they are added to the orderbook. To my knowledge, PHP isnt brilliant at handling streaming feeds and I'm yet to find a way of doing this server-side.

I could stand corrected though, I'm just waiting for a script-kiddy to tell me otherwise.

Whats java like at handling streaming data? How would you run a script like that on a server?

molecular

donator

Activity: 2772

Merit: 1019

Quote from: Seal on May 23, 2012, 02:55:21 AM

Quote from: molecular on May 14, 2012, 02:04:04 PM

Quote from: knite on April 07, 2012, 10:20:53 AM

- Consider making your SocketIO classes more generic and creating an MTGox sub-class which inherits from it, passing in event handlers (on_connect, on_msg, etc). This will make things easier if and when other exchanges add streaming APIs.

Consider this considered. However, when thinking about integrating other exchanges, there is a lot more work on other parts of my code to be done. I've been wanting to do this for a while now but decided if I would do it, I'd reimplement in another language that better suits my experience.

What languages do you recommend for best integration with SocketIO and streaming data?

I've been looking into scripting pages with PHP but it seems a little tough to integrate as its a language which generates pages on call.

Also I'd like to write a server-side script to run at home which will accept a socketio feed and place trades automatically. I currently do this again with the json calls API however because its written in PHP, it requires a cron call every x minutes. I'm looking for a better solution.

java, node.js, write timer loop yourself in php or use some php scheduling stuff? I don't know. For the kind of stuff we're looking at here, there's good APIs in every language, so it really boils down to personal language preference, I guess. I'd probably use java because I'm most comfortable with it. Unfortunately I cannot recommend my "traidor" code, because the orderbook still gets out of sync.

One remark: I don't know about the stability of the socket.io feed nowadays, but maybe you should implement a fallback to http api?

daybyter

legendary

Activity: 965

Merit: 1000

Java? I wrote some code for the HTTP API. Maybe we could collaborate for the socket stuff?

Seal

donator

Activity: 848

Merit: 1078

Quote from: molecular on May 14, 2012, 02:04:04 PM

Quote from: knite on April 07, 2012, 10:20:53 AM

- Consider making your SocketIO classes more generic and creating an MTGox sub-class which inherits from it, passing in event handlers (on_connect, on_msg, etc). This will make things easier if and when other exchanges add streaming APIs.

Consider this considered. However, when thinking about integrating other exchanges, there is a lot more work on other parts of my code to be done. I've been wanting to do this for a while now but decided if I would do it, I'd reimplement in another language that better suits my experience.

What languages do you recommend for best integration with SocketIO and streaming data?

I've been looking into scripting pages with PHP but it seems a little tough to integrate as its a language which generates pages on call.

Also I'd like to write a server-side script to run at home which will accept a socketio feed and place trades automatically. I currently do this again with the json calls API however because its written in PHP, it requires a cron call every x minutes. I'm looking for a better solution.

molecular

donator

Activity: 2772

Merit: 1019

Hi knite, nice to see the code being used / looked at and thanks for your comments. I'll comment on some of them...

Quote from: knite on April 07, 2012, 10:20:53 AM

- Is there a reason for using the urllib2 module? The requests module is much nicer.

No, not from my side. This (coding traidor) was the first time I did any requesting in python. Tried urllib (didn't work for some reason), the tried urllib2 (worked) so I just stuck with that.

Quote from: knite on April 07, 2012, 10:20:53 AM

- Why aren't you using the logging module instead of print statements?

Forgive my ignorance of the logging module, will take a look next time I need to dump some info.

On a more general note: there are never reasons for not doing something, there can only be reasons for doing something Wink

. Why do you not believe in the flying spaghetti monster?

Quote from: knite on April 07, 2012, 10:20:53 AM

- Consider making your SocketIO classes more generic and creating an MTGox sub-class which inherits from it, passing in event handlers (on_connect, on_msg, etc). This will make things easier if and when other exchanges add streaming APIs.

Consider this considered. However, when thinking about integrating other exchanges, there is a lot more work on other parts of my code to be done. I've been wanting to do this for a while now but decided if I would do it, I'd reimplement in another language that better suits my experience. Let me point you to Goncalo Pinheira (goncalopp, don't know if on forum, he mailed me, will give you his email upon request in pm), who has been asking me about this and if he could use my code (been asking for lgpl) and who seems to want to do exactly that (exchange abstraction)

Quote from: knite on April 07, 2012, 10:20:53 AM

- I strongly recommend reducing your client heartbeat interval. I've set mine to 85% of the value given by the server, rounded down.

done

knite

newbie

Activity: 33

Merit: 0

Threads are still melting my brain. Can I get the best of both worlds by doing something like this?

Code:

def start_thread(function):
t = Thread(target=function)
t.setDaemon(True)
t.start()
return t

def connect(S):
#setup
start_thread(thread_func)

def reconnect(S):
S.ws.close()
S.connect()

def thread_func(S):
S.ws = create_connection(S.ws_url)
S.run = True
start_thread(keepalive_func)

msg = S.ws.recv()
while msg is not None and S.run:
S.process(msg)
try:
msg = S.ws.recv()
except:
if S.run:
S.reconnect()
break
else:
if run:
S.reconnect()

def keepalive_func():
while S.run:
try:
S.ws.send('2::')
except:
S.ws.close()
break
sleep(S.heartbeat)

The keepalive thread does nothing except send heartbeats and close the connection in the event of an exception. Thread_func reconnects if it hits an exception or exits the while loop, if run is True. Thoughts?

knite

newbie

Activity: 33

Merit: 0

runeks and molecular:

I've reviewed and begun using a combination of your codebases. I have a few questions and comments.

- Why the different approaches to your keepalive threads? Molecular's version is more concise. Is it an issue with the ws.recv() delay on disconnect?
- Is there a reason for using the urllib2 module? The requests module is much nicer.
- Why aren't you using the logging module instead of print statements?
- Consider making your SocketIO classes more generic and creating an MTGox sub-class which inherits from it, passing in event handlers (on_connect, on_msg, etc). This will make things easier if and when other exchanges add streaming APIs.
- I strongly recommend reducing your client heartbeat interval. I've set mine to 85% of the value given by the server, rounded down.

runeks

legendary

Activity: 980

Merit: 1008

Quote from: molecular on March 27, 2012, 06:12:20 PM

Quote from: runeks on March 27, 2012, 11:18:58 AM

Here's my version. Just yesterday I found out that I need to handle exceptions in "response = urllib2.urlopen(req)" line in connect(). I couldn't understand why the script had stopped, but found out it was because this line had thrown a ConnectionRefused exception. So it's continually improving as I find out the various ways it can fail. Still haven't gotten rid of the SSL errors though.

https://github.com/runeksvendsen/traidor/blob/master/socketio.py

thanks, so cool.

it bails in line 94 for me, though, with:

Code:

Exception in thread Thread-4:
Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 530, in __bootstrap_inner
self.run()
File "/usr/lib/python2.7/threading.py", line 483, in run
self.__target(*self.__args, **self.__kwargs)
File "/home/nick/bitcoin/traidor/socketio.py", line 94, in thread_func
S.callback(S, msg[10:])
TypeError: onMessage() takes exactly 2 arguments (3 given)

this is because you added a parameter to the callback function and I still pass a callback function that takes one parameter less.

note that the "test_callback" function is in global scope, not in the class.

suggestion: If you need the instance of the socket.io in testing mode you can just use the global variable "sio" instead of passing S as follows (changes marked in bold):

Quote

def test_callback(msg):
import ast
print 'msg: ', msg
#convert received message string into dictionary
msg_dict = ast.literal_eval(msg)
#print msg_dict
if msg_dict['op'] == 'subscribe':
if msg_dict['channel'] == 'd5f06780-30a8-4a48-a2f8-7ed181b4a13f' or msg_dict['channel'] == '24e67e0d-1cad-4cc0-9e7a-f8523ef460fe':
sio.unsubscribe(msg_dict['channel'])

that way everything remains compatible and I can use your code

Right you are. I think I added it to be able to unsubscribe when a message arrives. But as you say, a global variable works better for this. Fixed in the newest revision.

molecular

donator

Activity: 2772

Merit: 1019

Quote from: sirk390 on April 01, 2012, 04:41:54 PM

I'm connection with python using something like this https://github.com/osmosis79/gox_socketio_py .

a newer version of which (with improved thread handling, much more stable concerning reconnects) can be found here https://github.com/molecular/traidor/blob/master/socketio.py

osmosis79 seems to have copied it about a month ago (no objections)

Topic: [BETA] MTGox websocket API, testers wanted - page 10. (Read 77653 times)