[BETA] MTGox websocket API, testers wanted - page 12.

runeks

legendary

Activity: 980

Merit: 1008

I'm still experiencing these errors with websocket_client unfortunately Undecided

Code:

08:06:59 4.81192 1.0 (4.81)
08:07:15 4.81174 0.55301041 (4.81)
08:07:25 4.81925 0.0616 (4.81)
08:07:26 4.81925 0.48723349 (4.81)
08:08:58 4.81174 16.57171689 (4.81)
trade['price_currency'] != "USD"
08:08:58 4.81146 16.0 (4.81)
08:10:26 4.8 10.0 (4.81)
08:11:01 4.7999 3.0 (4.81)
08:11:25 4.79992 0.54994211 (4.81)
trade['price_currency'] != "USD"
08:11:25 4.79992 5.0 (4.81)
08:11:30 4.78 0.27 (4.81)
08:11:30 4.78 2.0 (4.81)
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 552, in __bootstrap_inner
   self.run()
  File "/usr/lib/python2.7/threading.py", line 505, in run
   self.__target(*self.__args, **self.__kwargs)
  File "/home/rune/Programming/scripts/bitcoin/mtgox_io.py", line 78, in thread_func
   msg = S.ws.recv()
  File "/home/rune/Programming/scripts/bitcoin/websocket_client.py", line 349, in recv
   b = self._recv(1)
  File "/home/rune/Programming/scripts/bitcoin/websocket_client.py", line 413, in _recv
   raise ConnectionClosedException()
ConnectionClosedException

error sending keepalive socket.io, trying reconnect
08:13:00 4.7998 3.0 (4.81)
Exception in thread Thread-3:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 552, in __bootstrap_inner
   self.run()
  File "/usr/lib/python2.7/threading.py", line 505, in run
   self.__target(*self.__args, **self.__kwargs)
  File "/home/rune/Programming/scripts/bitcoin/mtgox_io.py", line 78, in thread_func
   msg = S.ws.recv()
  File "/home/rune/Programming/scripts/bitcoin/websocket_client.py", line 349, in recv
   b = self._recv(1)
  File "/home/rune/Programming/scripts/bitcoin/websocket_client.py", line 413, in _recv
   raise ConnectionClosedException()
ConnectionClosedException

error sending keepalive socket.io, trying reconnect
trade['price_currency'] != "USD"
08:14:27 4.7999 2.0 (4.81)
08:15:52 4.7998 3.0 (4.81)
08:16:06 4.8 30.0 (4.80)
08:17:13 4.7998 4.0 (4.80)
08:19:12 4.79906 0.03358625 (4.80)
trade['price_currency'] != "USD"
08:20:01 4.79001 1.48066947 (4.80)
08:20:37 4.79899 0.03190743 (4.80)
08:21:07 4.79894 1.22928907 (4.80)
08:21:38 4.79895 3.0 (4.80)
08:22:37 4.79895 2.80261836 (4.80)
Exception in thread Thread-5:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 552, in __bootstrap_inner
   self.run()
  File "/usr/lib/python2.7/threading.py", line 505, in run
   self.__target(*self.__args, **self.__kwargs)
  File "/home/rune/Programming/scripts/bitcoin/mtgox_io.py", line 78, in thread_func
   msg = S.ws.recv()
  File "/home/rune/Programming/scripts/bitcoin/websocket_client.py", line 349, in recv
   b = self._recv(1)
  File "/home/rune/Programming/scripts/bitcoin/websocket_client.py", line 413, in _recv
   raise ConnectionClosedException()
ConnectionClosedException

error sending keepalive socket.io, trying reconnect
error sending keepalive socket.io, trying reconnect
Exception in thread Thread-8:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 552, in __bootstrap_inner
   self.run()
  File "/usr/lib/python2.7/threading.py", line 505, in run
   self.__target(*self.__args, **self.__kwargs)
  File "/home/rune/Programming/scripts/bitcoin/mtgox_io.py", line 78, in thread_func
   msg = S.ws.recv()
  File "/home/rune/Programming/scripts/bitcoin/websocket_client.py", line 349, in recv
   b = self._recv(1)
  File "/home/rune/Programming/scripts/bitcoin/websocket_client.py", line 411, in _recv
   bytes = self.io_sock.recv(bufsize)
  File "/home/rune/Programming/scripts/bitcoin/websocket_client.py", line 161, in recv
   return self.ssl.read(bufsize)
SSLError: [Errno 1] _ssl.c:1350: error:1408F096:SSL routines:SSL3_GET_RECORD:encrypted length too long

Exception in thread Thread-7:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 552, in __bootstrap_inner
   self.run()
  File "/usr/lib/python2.7/threading.py", line 505, in run
   self.__target(*self.__args, **self.__kwargs)
  File "/home/rune/Programming/scripts/bitcoin/mtgox_io.py", line 78, in thread_func
   msg = S.ws.recv()
  File "/home/rune/Programming/scripts/bitcoin/websocket_client.py", line 349, in recv
   b = self._recv(1)
  File "/home/rune/Programming/scripts/bitcoin/websocket_client.py", line 411, in _recv
   bytes = self.io_sock.recv(bufsize)
  File "/home/rune/Programming/scripts/bitcoin/websocket_client.py", line 161, in recv
   return self.ssl.read(bufsize)
SSLError: [Errno 1] _ssl.c:1350: error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number

error sending keepalive socket.io, trying reconnect
08:24:08 4.79001 3.8303 (4.80)

First, a ConnectionClosedException exception is caught at the line "msg = S.ws.recv()" in thread_func. I assume this is because this call just hangs until either an error occurs or a message is received.
Then the keepalive_func thread will detect it when trying to send a message, and it will try to reconnect. This succeeds (seemingly), but lasts little time before the connection is lost again, and the process starts over (as you can see from the log). At some point, the errors go from the ConnectionClosedException exception to SSL exceptions like "error:1408F096:SSL routines:SSL3_GET_RECORD:encrypted length too long", but these SSL errors seem to vary a lot. I would guess that the SSL Python module expects the wrong data, but Mt. Gox' server sends the right data, though I'm not sure about that.

So, not sure what to do about this... seems something isn't right somewhere (duh!).

runeks

legendary

Activity: 980

Merit: 1008

Cool. Thanks. I'm trying it out now as we speak. Will report back how it works.

molecular

donator

Activity: 2772

Merit: 1019

Quote from: runeks on February 27, 2012, 12:27:52 PM

You da man molecular! Works like a charm. I'm not sure if Mt. Gox had problems or if the script you posted earlier in this thread missed something, but the script above works for me.

Thanks a lot!

BTW, had to change websocket_client to just websocket to get the script to run.

Also, I added a simple function to the SocketIO class that allows one to unsubscribe from a given channel:

Code:

def unsubscribe(S, channel_id):
S.ws.send('4::/mtgox:{"op":"unsubscribe","channel":"%s"}' % channel_id)

Thanks, I put that in just in case I need it some day Wink

Quote from: runeks on March 04, 2012, 12:26:24 PM

Anyone else getting errors like these with molecular's script?

Code:

SSLError: [Errno 1] _ssl.c:1350: error:1408F096:SSL routines:SSL3_GET_RECORD:encrypted length too long
SSLError: [Errno 1] _ssl.c:1350: error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number
SSLError: [Errno 1] _ssl.c:1350: error:1408F119:SSL routines:SSL3_GET_RECORD:decryption failed or bad record mac
SSLError: [Errno 1] _ssl.c:1350: error:14095044:SSL routines:SSL3_READ_N:internal error
SSLError: [Errno 1] _ssl.c:1350: error:140D2081:SSL routines:TLS1_ENC:block cipher pad is wrong
SSLError: [Errno 8] _ssl.c:1350: EOF occurred in violation of protocol
Segmentation fault
WebSocketException: Invalid frame type

Full log: http://pastebin.com/m4LF3tYs

Also, when these errors start occuring, I notice that bandwidth usage of my ticker script goes up. To like 20 KB/s IIRC. That seems like a lot for a trade every second or so. I figure it might be SSL certificates and the like going back and forth between mtgox.com and my computer. I find it hard to believe that this way of delivering trade information uses less resources than the original websocket API if everyone experiences these errors.

I have no idea, never seen this. But maybe you could try with the websocket_client.py I use (change that import back from "websocket" to "websocket_client" and the get file websocket_client.py from pastebin: http://pastebin.com/CTGj735u). I used to have a lot of troubles and tried at least 3 different websocket implemenations until I had a properly working one.

I _have_ seen "EOF occurred in violation of protocol", but I guess that's just a nasty disconnect...

runeks

legendary

Activity: 980

Merit: 1008

Anyone else getting errors like these with molecular's script?

Code:

SSLError: [Errno 1] _ssl.c:1350: error:1408F096:SSL routines:SSL3_GET_RECORD:encrypted length too long
SSLError: [Errno 1] _ssl.c:1350: error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number
SSLError: [Errno 1] _ssl.c:1350: error:1408F119:SSL routines:SSL3_GET_RECORD:decryption failed or bad record mac
SSLError: [Errno 1] _ssl.c:1350: error:14095044:SSL routines:SSL3_READ_N:internal error
SSLError: [Errno 1] _ssl.c:1350: error:140D2081:SSL routines:TLS1_ENC:block cipher pad is wrong
SSLError: [Errno 8] _ssl.c:1350: EOF occurred in violation of protocol
Segmentation fault
WebSocketException: Invalid frame type

Full log: http://pastebin.com/m4LF3tYs

Also, when these errors start occuring, I notice that bandwidth usage of my ticker script goes up. To like 20 KB/s IIRC. That seems like a lot for a trade every second or so. I figure it might be SSL certificates and the like going back and forth between mtgox.com and my computer. I find it hard to believe that this way of delivering trade information uses less resources than the original websocket API if everyone experiences these errors.

runeks

legendary

Activity: 980

Merit: 1008

You da man molecular! Works like a charm. I'm not sure if Mt. Gox had problems or if the script you posted earlier in this thread missed something, but the script above works for me.

Thanks a lot!

BTW, had to change websocket_client to just websocket to get the script to run.

Also, I added a simple function to the SocketIO class that allows one to unsubscribe from a given channel:

Code:

def unsubscribe(S, channel_id):
S.ws.send('4::/mtgox:{"op":"unsubscribe","channel":"%s"}' % channel_id)

molecular

donator

Activity: 2772

Merit: 1019

Quote from: runeks on February 22, 2012, 10:39:23 AM

I have a hard time following all the tips and tricks in this thread on getting the socket.io interface to work with Python.

Has anyone gotten it to work properly? As far as I can tell there are no socket.io client implementations for Python right, right?

maybe this helps, my socketio.py:

Quote

from threading import *
import urllib2, urllib
import simplejson as json
import ssl, socket
import time
from websocket_client import create_connection
import traceback

class SocketIO:
  def __init__(S, url, callback):
   S.url = url
   S.callback = callback

  def connect(S):
   try:
   data = urllib.urlencode({})
   req = urllib2.Request('https://' + S.url + "/1", data)
   print 'https://' + S.url + "/1"
   response = urllib2.urlopen(req)
   r = response.read().split(':')
   S.heartbeat_interval = int(r[1])
   #print 'heartbeat: ', S.heartbeat_interval
   if 'websocket' in r[3].split(','):
   print "good: transport 'websocket' supported by socket.io server ", S.url
   S.id = r[0]
   print "id: ", S.id
   else:
   print "error: transport 'websocket' NOT supported by socket.io server ", S.url

   S.thread = Thread(target = S.thread_func)
   S.thread.setDaemon(True)
   S.thread.start()
   except:
   traceback.print_exc()

  def stop(S):
   S.run = False
   S.thread.join(timeout=1)
   S.keepalive_thread.join(timeout=1)

  def thread_func(S):
   print 'SocketIO: websocket thread started'

   my_url = 'wss://' + S.url + "/1/websocket/" + S.id

   S.ws = create_connection(my_url)

   #S.ws = WebSocket(my_url, version=0)
   S.run = True
   S.ws.send('1::/mtgox')

   # start keepalive thread
   S.keepalive_thread = Thread(target = S.keepalive_func)
   S.keepalive_thread.setDaemon(True)
   S.keepalive_thread.start()

   msg = S.ws.recv()
   while msg is not None and S.run:
   #print 'SocketIO msg: ', msg
   if msg[:10] == "4::/mtgox:":
   S.callback(msg[10:])
   #elif msg[:3] == "2::":
   # True
   #else:
   # print "SocketIO: dont know how to handle msg: ", msg
   msg = S.ws.recv()
   S.ws.close()

  def keepalive_func(S):
   while S.run:
   try:
   S.ws.send('2::');
   except:
   if S.run:
   print 'error sending keepalive socket.io, trying reconnect'
   S.connect()
   else:
   print 'exiting socket.io keepalive thread'
   time.sleep(S.heartbeat_interval)

def test_callback(msg):
  print 'msg: ', msg

# testcase
if False:
  sio = SocketIO('socketio.mtgox.com/socket.io', test_callback)
  sio.connect()
  time.sleep(100)

EDIT: the websocket_client imported I got from here: http://pypi.python.org/pypi/websocket-client/0.4

EDIT2: I've been using this implementation for months and it works well for me. If you spot any problems with this implementation of fix errors or otherwise improve it, please let me know.

Ente

legendary

Activity: 2126

Merit: 1001

Quote from: zelyony on February 15, 2012, 05:39:41 AM

UPD2:
it's working if wait connected sometimes 20 minutes and sometimes 0 minutes and sometimes "never". very strange and no good.
can someone's debug session stops sending messages to other users?.. who knows implementation of socket.io server?

Same here. Sometimes it takes ages to receive any data, sometimes less than a minute. Totally random, and totally annoying. The 1:: and 2:: pings work straight from the beginning.
As soon as I get a response to my channel-connect, data from this channel comes in too.

Ente

runeks

legendary

Activity: 980

Merit: 1008

I have a hard time following all the tips and tricks in this thread on getting the socket.io interface to work with Python.

Has anyone gotten it to work properly? As far as I can tell there are no socket.io client implementations for Python right, right?

zelyony

newbie

Activity: 23

Merit: 0

and right now I dont recv any useful datas Huh

UPD:
it's working right now (13:44 UTC)

UPD2:
it's working if wait connected sometimes 20 minutes and sometimes 0 minutes and sometimes "never". very strange and no good.
can someone's debug session stops sending messages to other users?.. who knows implementation of socket.io server?

To MtGox:
maybe it's sabotage.. Slow Read DoS attack.. bad guys overflow send pool with slow connections and.. voila!
will be better for all clients drop connections that cannot read string, i.e. send to client not finished for.. hmm.. 0.5sec! +ban by IP 10min (epoll, select or other asynchronous mechanism). I took digits from air, I dont know % loading for servers machines and number of connections.
p.s. ping ("2::") works always

Ente

legendary

Activity: 2126

Merit: 1001

Thank you for checking, zelyony!
You are right, thats a typo in the snippet. I sent '1::' and '2::' accordingly in my full script, typo corrected.

I sent '4:::{"op":"mtgox.subscribe","type":"trades"}' over the websocket, no change. No response whatsoever, no trade-messages coming in.

Ente

zelyony

newbie

Activity: 23

Merit: 0

right now datas arrived (my format after string -> json -> object -> string):
JSON: Packet={ Depth={ price=4,67000 USD, vol=0,05300000, dir=bid, when=2012:02:15_10:03:56.735844 } }
JSON: Packet={ Ticker={ bid=$4.68100, ask=$4.70075, lo=$4.21211, hi=$4.97000, last=$4.70075, vol=256,165.42727403??BTC } }

I dont know Python, but:
- on connected (or after recv "1::" - that's mean u connected now) u must send strings like:
4:::{"op":"mtgox.subscribe","type":"trades"}
- on "2::" u must response "2::" (not "1::" as in ur sample)

Ente

legendary

Activity: 2126

Merit: 1001

Quote from: Ente on February 14, 2012, 05:29:19 PM

1 BTC bounty for the first person who helps me receive regular data again over the websocket!

I made a new thread for this error-solving bounty:
https://bitcointalksearch.org/topic/m.749747

Ente

Ente

legendary

Activity: 2126

Merit: 1001

*bangs head on desk*
I cant get it to work.

1 BTC bounty for the first person who helps me receive regular data again over the websocket!

Here is my (totally stripped) python code:

Code:

#!/usr/bin/env python
# -*- coding: utf-8 -*-

from websocket import create_connection
import urllib
import re
import time

try:
   import json
except ImportError:
   import simplejson as json

wsurl = "wss://socketio.mtgox.com/socket.io/"
iourl = "https://socketio.mtgox.com/socket.io/"

############

print "Connecting to socket.io.."
url = iourl + "1"
f = urllib.urlopen(url)
output = f.read()
f.close()
ausgabe = re.search('[0-9]+', output, 0)
sessionid = ausgabe.group()

print "New websocket url:"
wssurl = wsurl + "1" + "/websocket/" + sessionid
print wssurl

############

print "Connecting to websocket..\n"

socket = create_connection(wssurl)

while 1:
  echo = socket.recv()
  if "1::" in echo:
   print "Heartbeat 1"
   socket.send("1::")
  elif "2::" in echo:
   print "Heartbeat 2"
   socket.send("2::")
  else:
   print "Message: ",echo
  time.sleep(2)

The websocket client I use here is from http://pypi.python.org/pypi/websocket-client/0.4.
I have a rudimentary heartbeat added here.
The whole script dies eventually, from having stripped all error-handling.
It worked some weeks ago. If you like to see the whole script, let me know.

Bedtime here, will check back in, like, 9 hours..

*frustrated and tired*

Ente

Ente

legendary

Activity: 2126

Merit: 1001

OH! Thank you a lot for your help, zelyony!

I was already wondering about the heartbeat. I now answer 1:: and 2:: accordingly, now they keep flowing in constantly.
Well, the socket *must* be working, since the websites using it still work. There would be more outcry in the forum if it was dead altogether..

I am glad I'm not the only one here!

Ente

zelyony

newbie

Activity: 23

Merit: 0

Quote from: Ente on February 13, 2012, 03:32:28 AM

So I got the handshake coded, getting me my connection-ID. I then can connect to the websocket just fine.
However, I dont receive any data, only two messages: '1::' and '2::'. Both come once in the first few seconds of being connected, then total silence. I am not being disconnected neither, I think.
Do I have to join a channel? Or am I connected to the default channels just like before?
I dont have "sending messages" finished yet to test..
Any hints?

in the past (3+ weeks ago):
by default u dont subscribed to channels
u may send simple strings over internal (socket.io's) websocket
for "2::" u must send "2::" (or send "2::" every 15secs)
for subscribe u must send post #162
datas stopped in socket after 3-15min.. u can open & sync 2 socket.io..
for now:
post #162 looks no more working - no datas except "1::" & "2::"
Huh

Ente

legendary

Activity: 2126

Merit: 1001

Quote from: Ente on January 07, 2012, 08:03:58 AM

TL;DR:
Will ws://websocket.mtgox.com/mtgox continue to work?

Some days ago I connected to the old, deprecated websocket. Connecting works, but no data comes in any more.

Now thats an incentive to figure out socket.io :-)

edit:

Quote from: rwcp97 on January 21, 2012, 03:57:16 AM

As far as I know there is no possibility to do orders over the websocket...

Seems like now you can indeed do orders over socket.io:

Quote from: Bitcoin Wiki link=https://en.bitcoin.it/wiki/MtGox/API#Streaming_API

Authenticated commands
These commands require an API key and secret pair to sign requests. Any of the "version 1" requests from the HTTP API can be called.

So I got the handshake coded, getting me my connection-ID. I then can connect to the websocket just fine.
However, I dont receive any data, only two messages: '1::' and '2::'. Both come once in the first few seconds of being connected, then total silence. I am not being disconnected neither, I think.
Do I have to join a channel? Or am I connected to the default channels just like before?
I dont have "sending messages" finished yet to test..

Any hints?

Ente

osmosis

sr. member

Activity: 300

Merit: 250

Quote from: osmosis on January 24, 2012, 09:14:38 PM

Quote from: molecular on November 24, 2011, 09:58:37 AM

#> python socketio.py
it should dump the messages to stdout

Thanks molecular.

For the websocket in python, anyone find a way to deal with the disconnect errors?

File "/home/osmosis/proj_venv/proj_proj/websocket_client.py", line 161, in recv
SSLError: [Errno 1] _ssl.c:1325: error:14095044:SSL routines:SSL3_READ_N:internal error
SSLError: [Errno 1] _ssl.c:1325: error:1408F119:SSL routines:SSL3_GET_RECORD:decryption failed or bad record mac
SSLError: [Errno 1] _ssl.c:1325: error:1408F096:SSL routines:SSL3_GET_RECORD:encrypted length too long
SSLError: [Errno 1] _ssl.c:1325: error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number
SSLError: [Errno 8] _ssl.c:1325: EOF occurred in violation of protocol
ConnectionClosedException

piotr_n

legendary

Activity: 2053

Merit: 1356

aka tonikt

Quote from: zelyony on January 26, 2012, 10:11:16 AM

Quote from: piotr_n on January 25, 2012, 08:14:54 AM

I get my ID nicely from https://socketio.mtgox.com/socket.io/1/ but then after I connect my websocket lib to ws://socketio.mtgox.com:80/socket.io/1/websocket/XXXXX - it just sends me "7:::1+0" and disconnects immediately after that.

If u got ID from https:// than u must connect to wss:// with this ID (SSL/TLS - secure connection)
and if http:// than ws://

shit man.
thank you!!!
that was indeed the problem.

zelyony

newbie

Activity: 23

Merit: 0

Quote from: piotr_n on January 25, 2012, 08:14:54 AM

I get my ID nicely from https://socketio.mtgox.com/socket.io/1/ but then after I connect my websocket lib to ws://socketio.mtgox.com:80/socket.io/1/websocket/XXXXX - it just sends me "7:::1+0" and disconnects immediately after that.

If u got ID from https:// than u must connect to wss:// with this ID (SSL/TLS - secure connection)
and if http:// than ws://

piotr_n

legendary

Activity: 2053

Merit: 1356

aka tonikt

Or just try this page: http://websocket.org/echo.html
... using i.e. ws://socketio.mtgox.com:80/socket.io/1/websocket/14001291971574386363
and tell me that I'm crazy

Topic: [BETA] MTGox websocket API, testers wanted - page 12. (Read 77681 times)