Topic: [BETA] MTGox websocket API, testers wanted - page 3. (Read 77681 times)

I've been having massive problems keeping order book synced in my client, there just seem to be lost messages. Never got it right.

Last resort: just pull full-depth 5 times per hour (magicaltux allows that frequency) and sync your orderbook

that is the question,

there was no any corresponding trading at that time.

Well there was a $1M bid wall that got removed right before the selling started yesterday. I don't believe that large drop was an error.

Hi, all!

We at bitcoin-analytics.com support local state of the orderbook and update it listening depth messages and fix obvious invalid state of orderbook ( when ask price is less than bid price) using ticker messages.

While fixing with ticker messages helps, it seems that it is not enough. Invalid orderbook state occures because some depth mesages does not get to the client from server. It is not so obvious if only you do not maintain history of the orderbook volume which we do on bitcoin-analytics.com.

What we noticed a repeting pattern like on the screenhot.



It looks like the volume increased steadily during several hours and then abruptly reduced. This happend exact at the moment when orderbook was just reinitialized.

The problem as we see it is the following: during the long period of time some depth messages does not come to client but do not invoke fixing procedure cause those lost messages does not break the validity of the orderbook state. But those lost messages did not remove the orders which they had to remove or change. During the period of time new orders were added to the orderbook. Thus the volume was increased while errors of lost messages were accumulated.

Did anyone noticed something similar. And does anyone has any ideas about this problem? How is it possible to detect those lost messages orders and fix them?

thanks

I'm trying to subscribe to "lag" channel on socketIO:

Tried sending this:

{
    "channel":"85174711-be64-4de1-b783-0628995d7914",
    "op":"subscribe"
}

and without quotes around channel ID like this:

{
    "channel":"85174711-be64-4de1-b783-0628995d7914",
    "op":"subscribe"
}


But whatever I try there is no response, and certainly no subscription success response.

My Socket.IO TX framing is correct, if I break the framing the mtgox server drops connection.

I'm not able to subscribe/unsubscribe any channels dynamically.

To add/remove currencies I have to reset connection with new list of currency codes in the initial HTTP exchange.

I wonder what am I doing wrong?

I Agree, MtGox API need strongly to improve. Current realisation can be declared only as "not real-time". Extreme event lags! Some time required tens seconds, from send order to retrive all trades, what make API not usable for fast trailing stops and many others. Since my trading program cannot normal work with MtGox, I just switch to Icbit.

You can also trigger your actions in the user_order message for order removal, this message will be fired when the order is completely filled or the last remaining part of it is filled (and therefore the order is removed). Problem here is: it will also fire when you cancel the order. So you need some additional code that knows what was going on before. If your bot is the only thing that ever cancels orders then will know the reason for the remove message but if you also cancel orders manually or through other bots running on the same account then you need somethng more sophisticated, then you need to keep track of the private trade messages too and if a trade message at the same price preceded the the order remove than it is a filled order, otherwise its a cancel or something like that.

The API clearly was not designed by someone who has ever written a trading bot before, its missing needed information in a lot of places and forces you to do cumbersome workarounds to detect certain events or keep track of the information you need, and at the same time all the json messages are bloated with totally useless redundant data that nobody ever uses.

It is anonymous data stream, without orders association. So bad, protocol provoke for many additional requests

There will be a trade message on the private channel (see streaming API docs) but I'm not sure if it has the order ID.

How can make a subscription to my bot trades? Requesting for every order private/order/result is slow bad solution. I need recieve trades with tid, oid, price and volume fields, as soon as possible after order matched partally or full.

Thanks you both for your answers!

I did some further experimenting and debugging and it seems this old posting of mine is wrong. There *will* be a depth message immediately following the trade message, so theoretically it would be enough to use only depth messages to update the orderbook.

But in goxtool/goxapi I still update the orderbook already in the trade message, so a hypothetical goxtool bot that would rely on the *new* orderbook state *after* the trade has been executed would already find the orderbook in the new updated state from inside the slot_trade() slot. Since I only ever use the total_vol of the depth messages and ignore the volume difference in my implementation the depth messages that immediately follow a trade message will simply result in no change to the orderbook at all.

strange I currently use only depth and ticker messages. and did not have problems with orderbook state.

on my opinion there is not need to use trade messages because some trades can be from orders which are not in orderbook but executed before getting into orderbook.

but probably I missed something. It would be good to hear others

Are you talking about trades in other currencies or about all trades? Because I'm pretty sure that you have to update your depth tables when a trade message comes in as is explained here https://bitcointalksearch.org/topic/m.1636817. I do that and it seems to be working fine. I only don't know what to do with trades in other currencies.

I'm already using the trick to use ticker messages to fix the tables when they appear to be broken, but that doesn't happen a lot.

You should not update orderbooks using trades channel. You need to listen to depth channel to update orderbooks. And also it would be convinient to fix ordebooks with ticker channel messages if neccesary (in case not all depth messages were arrived and ordebook state is broken)

I've downloaded the full bid / ask tables in USD with 'depth/full' and try to update them as depth and trade messages come in. From https://support.mtgox.com/entries/20800336-Multi-Currency-Trading I understand that the tables in USD also include the bids / asks in other currencies, that are converted to USD at the daily rate of the European Central Bank minus / plus a fee of 2.5 %.

How do I update my tables when a primary trade comes in with a price_currency other than USD?

To answer my own question above. It seems to be working now. Now I also get trade and lag messages. Apparently it was something at MtGox's side.

I'm experimenting with the MtGox websocket API. I'm using Python with the websocket module from https://pypi.python.org/pypi/websocket-client. See example code below


When I run this code I receive ticker and depth messages, but no lag or trade messages. Also I do not get any replies to my mtgox.subscribe commands, that the documentation seems to promise. However when I send a mtgox.subscribe command with a wrong type parameter, I get an error message "Unknown mtgox message type", so it seems my subscribe commands are received and accepted.

Could somebody please tell me why I'm not receiving trade and lag messages?

Thanks for sharing your insight, sounds like your going the extra mile to mantain as much order as possible on such a shaky foundation.

I'm not sure I have the drive to start wrtiting/debugging such a workaround, I would end up getting kicked off for >5 tries an hour, I'd reset my gateway and try again and probably end up being block banned.

At the moment I have no users reliant on my software so do not have the stressful job of trying to get a fix/workaround out. I will sit tight for a few days and work on some GUI stuff.

Perhaps when MtGox stream re-establishes properly it will run like a dream, let us hope so. ( until it breaks again: ).

Best of luck with your commendable efforts.

Actually when socket.io works I start with http request full orderbook then when it is initalized socket sends update messages.

when socket.io reconnects the whole initializaton starts again with requesting full order book via http.

the problem is that you if you request full orderbook more then 5 times per hour as stated in mtgox stream api documentation.

So I had to add 5 times limitating logic.

and the whole system worked well until recently.

so now every 15 minutes I trigger connection check and if it is not connected request system to reinitialize orderbook.

this additional logic is implemented as upper layer over original client system.

currently I listen to depth and ticker for  quick fixes of broken orderbook without unnecessary reinitialization.

for trades we use bitcoincharts stream it contains not only mtgox trades.

We plan to advance the system to be able to get trades from mtgox channel too but we want to integrate it with bitcoincharts stream for robustness.