[BETA] MTGox websocket API, testers wanted - page 19.

0x6763

newbie

Activity: 24

Merit: 0

Quote from: martin on May 18, 2011, 05:56:47 PM

I've been playing around a bit with this webcosket feed in C#, and I'm not sure if I'm doing things wrong or the feed is slightly broken.

Every time I receive a depth update I check the volume, if it's positive I add it to my local depth table, if it's negative I search the table for an exact matching depth entry, and remove it. After a while, this ends up with sell prices being less than buy prices - which is clearly wrong. Should this work, or am I missing something?

w.r.t all the people saying that sending the complete depth table on connection would be useful, I agree. It seems vital for the client to be able to request depth data, I would implement it as the client sending a request message for depth data, something like:

{ "op":"request-depth" }

And then the socket sends back the *complete* depth data in a *single message* in the same form that the web API currently returns it. In this way, once you receive the complete table you know you have a reliable dataset to perform updates to as they arrive.

The obvious caveat of this is that once you send the complete depth data, it must act as a "barrier", you can't send the complete state, and then an update to a stale state. (does this make sense?)

Mtgox currently has multiple bugs in the websocket's depth channel...your tables are behaving exactly the same as mine are. Mtgox is moving to a new system in june which will hopefully have all of these problems taken care of.

martin

full member

Activity: 150

Merit: 100

I've been playing around a bit with this webcosket feed in C#, and I'm not sure if I'm doing things wrong or the feed is slightly broken.

Every time I receive a depth update I check the volume, if it's positive I add it to my local depth table, if it's negative I search the table for an exact matching depth entry, and remove it. After a while, this ends up with sell prices being less than buy prices - which is clearly wrong. Should this work, or am I missing something?

w.r.t all the people saying that sending the complete depth table on connection would be useful, I agree. It seems vital for the client to be able to request depth data, I would implement it as the client sending a request message for depth data, something like:

{ "op":"request-depth" }

And then the socket sends back the *complete* depth data in a *single message* in the same form that the web API currently returns it. In this way, once you receive the complete table you know you have a reliable dataset to perform updates to as they arrive.

The obvious caveat of this is that once you send the complete depth data, it must act as a "barrier", you can't send the complete state, and then an update to a stale state. (does this make sense?)

0x6763

newbie

Activity: 24

Merit: 0

Quote from: toffoo on May 17, 2011, 12:40:02 AM

Hi guys,
Does anybody have some example python code that just connects to the Mt.Gox websocket feed and maybe just spits out trades or order changes?
I'd like to start playing around with this, but I hate to try to re-create the wheel from scratch when I'm such a rusty python programmer.
Many thanks for any help!

Here's a naive Mt. Gox websocket client I wrote in python: http://pastebin.com/FH6eWhFm
By default it just prints out trades as they happen, but you can modify it to do whatever you want. If someone really wants an event-based version or something to make it a little easier to work with, you might be able to convince me (with a few bitcoins Wink

) to put something together with pywsc or something. I may eventually release an executable jar (clojure/java) file for those willing to buy a copy that will give you a GUI to view live trades and depth changes (though mtgox's websocket is still quite buggy in respect to depth changes...there would be a button to let you do manual refreshes as well).

mindtheft

newbie

Activity: 11

Merit: 0

A quick js chrome extension hack for WebSocket -

https://github.com/mindtheft/mtgox-Chrome-Extension

You can use the crx file to install the extension, or better yet check the source code and pack your own crx file to make sure there's no foul smells from mine.

toffoo

sr. member

Activity: 408

Merit: 261

Hi guys,
Does anybody have some example python code that just connects to the Mt.Gox websocket feed and maybe just spits out trades or order changes?
I'd like to start playing around with this, but I hate to try to re-create the wheel from scratch when I'm such a rusty python programmer.
Many thanks for any help!

molecular

donator

Activity: 2772

Merit: 1019

Quote from: Keefe on May 12, 2011, 11:48:53 PM

I was going to make a nice interface for the feed as well, but I'm not going to bother until the problems are resolved:
Rounding issues
No perfect way to start out in-sync
Alleged missing data and dark pool inconsistencies

I'm seeing a slight problem for MagilTux to solve the "keep client depth tables in sync" problem, I hadn't thought about:

(Firstly, wether a client gets the initial depth data through the "classic API" or through an initial burst of depth-change messages doesn't make a difference for my thinking here.)

Problem is: the depth data a client receives initially is chopped off at the ends at a certain distance to the current last trade value.

Now whenever a trade happens, this window moves and some trades exit the window on the one end while others become "visible" on the other end.

There's 2 ways (at least) to solve this:

Screw that window completely, give the complete depth data to clients
On trade, determine entering/leaving orders and send them as new/cancelled over the channel

I'd go for the first option in combination with initially bursting change-messages instead of using the "classic API" to initially fill client depth table.

Ycros

newbie

Activity: 1

Merit: 0

Herp, disregard what I posted before, I found my client library was sending an extraneous \r\n\r\n on the end there.

jaybny

sr. member

Activity: 410

Merit: 250

Proof-of-Skill - protoblock.com

my program is listening to the WebSocket and publishing to twitter once an hour, on new highs or lows.

im using c# and am not using JSON schema, just parsing directly... so if anything changes - I will break..

http://twitter.com/newberg_feed

molecular

donator

Activity: 2772

Merit: 1019

Quote from: Keefe on May 12, 2011, 11:48:53 PM

I was going to make a nice interface for the feed as well, but I'm not going to bother until the problems are resolved:
Rounding issues
No perfect way to start out in-sync
Alleged missing data and dark pool inconsistencies

MagicalTux, can you give us an estimate when you'll get around to implementing a solution to these problems or talk to us about a solution?

These problems are really not acceptable: we can't have the client side depth data out of sync or even worse stuff happening (missing data?)

I'm here to help with testing.

Keefe

hero member

Activity: 681

Merit: 500

I was going to make a nice interface for the feed as well, but I'm not going to bother until the problems are resolved:
Rounding issues
No perfect way to start out in-sync
Alleged missing data and dark pool inconsistencies

0x6763

newbie

Activity: 24

Merit: 0

Quote from: molecular on May 12, 2011, 05:47:32 PM

Quote from: Shrapnel on May 12, 2011, 06:19:27 AM

Quote from: molecular on May 12, 2011, 06:07:11 AM

It seems, though, that I'm doing something wrong: I'm getting negative volumes in my depth data after a while.

Actually the negative values are cancelled orders.

Of course. So to keep my copy of the "order book" synced, I add that negative value at the given price and should get 0, or (if some other order at the same price still exitsts) some positive value, but never a negative value, which happens to me.

Mtgox has some bugs. The websocket often entirely skips sending some depth messages, sends some at a slightly different price than what they should be, and the APIs (or site in general) has rounding inconsistencies. I've only partially worked around these issues by rounding all depth message prices to the 5th decimal place and volumes to the 3rd decimal place, and creating a button in my program that I can click to manually refresh the entire depth tables when I start noticing odd stuff.

An example is where a bot will download it's own open orders, see that one of it's orders is something like 6.253, but it wants it to be 6.252999, so it removes the order for 6.253 and creates a new one at 6.252999, and both of those come over the websocket just like that. And the bot does this over and over again because the API for getting your own open orders rounds to the 4th decimal place, but the websocket will do at least 6 decimal places for prices and more than that for volume, so your table ends up removing the order at 6.253 over and over again, resulting in a negative number, while it keeps adding orders to 6.252999 over and over again, making some huge number that's not accurate. The bugs and inconsistencies are pretty annoying.

jed

full member

Activity: 182

Merit: 107

Jed McCaleb

I think it doesn't send every update to the order book. It seems like some random ones get dropped.

molecular

donator

Activity: 2772

Merit: 1019

Quote from: Shrapnel on May 12, 2011, 06:19:27 AM

Quote from: molecular on May 12, 2011, 06:07:11 AM

It seems, though, that I'm doing something wrong: I'm getting negative volumes in my depth data after a while.

Actually the negative values are cancelled orders.

Of course. So to keep my copy of the "order book" synced, I add that negative value at the given price and should get 0, or (if some other order at the same price still exitsts) some positive value, but never a negative value, which happens to me.

Shrapnel

newbie

Activity: 9

Merit: 0

Quote from: molecular on May 12, 2011, 06:07:11 AM

It seems, though, that I'm doing something wrong: I'm getting negative volumes in my depth data after a while.

Actually the negative values are cancelled orders.

Quote

Another point I'm unclear on: How can I possibly ensure that the timepoint of my request of the initial depth table and the starting timepoint of the websocket depth change messages match?

I'm also wondering the same thing. I think you can't unless the API send the complete data first.
I'm trying to do a realtime order book but it's currently very hard due to dark pool and sync issues.

molecular

donator

Activity: 2772

Merit: 1019

ok, I'm receiving messages, nice Wink

Now I want to update the market depth info, which I'm initially getting through http://mtgox.com/code/data/getDepth.php.

so I'm listening on channel "24e67e0d-1cad-4cc0-9e7a-f8523ef460fe" for messages that have op="private", and am getting depth updates like this:

Code:

{u'volume': -10, u'price': 5.4000000000000004, u'type': 2}

It seems, though, that I'm doing something wrong: I'm getting negative volumes in my depth data after a while.

Above message is an example of a message that caused this... I didn't have an entry for price "5.4000" (I'm using 4 decimal places for the index in my depth "dictionary", but that should do no harm, right), so the result is a new entry with negative volume.

Another point I'm unclear on: How can I possibly ensure that the timepoint of my request of the initial depth table and the starting timepoint of the websocket depth change messages match?

EDIT:

Quote from: 0x6763 on April 29, 2011, 01:35:27 AM

Also, I agree with jed that being able to get the current depth table (as well as trade table, and my own full order list) through the websocket instead of the old API would be an eventual nice-to-have.

If my problem actually stems from the two timepoints (initial depth request, start of depth updates) not being equal, I would consider this to be not a nice-to-have but a must-have.

You could maybe simply (from client point of view) send the complete depth table as a burst of updates after websocket connect?

toffoo

sr. member

Activity: 408

Merit: 261

Quote from: Clark on May 11, 2011, 01:03:16 PM

Awesome!

I'm working at the moment to integrate the websockets feed into a nice GUI, akin to what you'd expect from standard day trading software in the finance world.

Fantastic news. Please let us know when you have something ready to test out.

Clark

hero member

Activity: 548

Merit: 502

So much code.

Awesome!

I'm working at the moment to integrate the websockets feed into a nice GUI, akin to what you'd expect from standard day trading software in the finance world.

molecular

donator

Activity: 2772

Merit: 1019

Quote from: Clark on May 11, 2011, 02:19:31 AM

I connect to the exact same address, so I really can't tell what the problem. Maybe the handshake is not executing properly before it enters the main receive loop?

SOLVED! Thanks for your help.

Stupid me forgot about that transparent squid proxy.

Am receiving messages, now to the fun stuff!

Clark

hero member

Activity: 548

Merit: 502

So much code.

Well, I had to change my WebSocket._send function to do this:

Code:

self._sendRaw('%s%s%s' % ('\x00', data, '\xff'))

because it was breaking the socket trying to send a non-string.

My receive code is identical except that I added a Threading.Event for that loop instead of just while True.

I connect to the exact same address, so I really can't tell what the problem. Maybe the handshake is not executing properly before it enters the main receive loop?

molecular

donator

Activity: 2772

Merit: 1019

Quote from: Clark on April 17, 2011, 07:39:23 PM

I grabbed a copy of the pywsc Websockets library for Python and am receiving trades, orders, and ticker events.

Hmm, I also tried this. Can't seem to succeed.

What port do I have to use? I tried like this:

Code:

from pywsc.websocket import WebSocket
ws = WebSocket('ws://websocket.mtgox.com:80/mtgox')

This bails with "error receiving" in receiver.py. I inserted debugging output to see what it's receiving. It seems to receive the page you get when you request http://websocket.mtgox.com/.

I'm confused and obviously doing something wrong. Can someone help?

Topic: [BETA] MTGox websocket API, testers wanted - page 19. (Read 77653 times)