Topic: [BETA] MTGox websocket API, testers wanted - page 9. (Read 77681 times)

Yeah, I'm seeing this to. Also seeing "invalid call".

By the way, is it possible to connect to socketio.mtgox.com via WebSocket? That is, a regular RFC 6455 WebSocket and not via the Socket.IO library (which I believe is meant to be able to fall back to WebSocket). I've never had any success with this, I've only ever been able to make WebSocket connections to websocket.mtgox.com on port 80 / 443.

"Method not found"

is the answer I get from the server for every signed API request (such as private/idkey or private/info or private/orders or order/add, order/cancel etc).

It started happening exactly at 2013-03-14 23:53:48 (central european time), it did not close the connection, it just suddenly stopped accepting any API commands.

Both servers (socketio and websocket) behave this way. I still get ticker, trade and depth but cannot send any trading commands or other commands anymore.

Javascript, among other things. Don't get me wrong I love node but it just can't handle memory copying / allocating data structures as fast as a compiled language, which is understandable as it is using a JIT compiler with GC. In any case the difference with native C implementations isn't that big but it's there. I do realise that I'm comparing apples with oranges here but it's a valid comparison since the only stable socket.io server implementation is in javascript.

I forgot to add that the big advantage of socket.io is its (almost) seamless horizontal scalability as long as your redis box is beefy.



https://github.com/LearnBoost/socket.io/issues?page=1&sort=comments&state=open

Its creator is busy working on 1.0 which is a complete rewrite (uses engine.io as the abstraction for the underlying transports).



They are doing it wrong.

Every layer on top of another layer adds overhead. The overhead in this case (it might also be the way MtGox is using it or how they hooked it up to their other software) must be enormous, otherwise it would not lag behind the plain websocket server by at least 10 seconds during idle times (up to a few minutes at busy times), silently drop API-commands (or not ack them for 2 or more minutes, etc), while at the same time the old websocket server (when it is running) will instantly ACK (op:result) all my commands (for example give me the order-IDs for new pending orders within fractions of a second even when order-lag is high).  On the socketio stream I can sometimes wait 2 minutes for my order-ID while the order lag reading says "idle" all the time.

Could you please elaborate about significant overhead?

https://github.com/LearnBoost/socket.io/blob/master/History.md - recent releases history. I don't see any signs of it being abandoned.

Yes, you can do this via call operations with the BTC{currency}/fulldepth endpoint.

Note that this is rate-limited and the rate is pretty low. I've gone over it quite easily during testing. Unless you need the entire depth information it's probably worth sticking with partial depth information via BTC{currency}/depth.


I think the server will deliberately disconnect clients after an hour (at least, this is what it looks like from my logs).

I've only had problems with WebSocket connections in the past few days and I can't connect at all at the moment. It looks as though the TCP connection is established but the server won't sent any responses when the TLS session is being established (or even send any ACK replies…). Ditto for unencrypted connections.

The problem is (and has always been) the fallback transports. They are also the reason that socket.io exists. In a world where every browser has RFC 6455 websocket support socket.io has no reason for being. Channels? Meh. Broadcast? Meh. Socket.io with a websocket transport also has significant overhead versus a pure websocket implementation.

Add to that that 0.9 is currently unmaintained (there are over 300 open issues on github). The only way to scale socket.io right now is to throw more machines at it and use RedisStore.

The polling transports (XHR/JSONP) are performance killers but if you disable them you lose IE6-9. They will both leak connections over long periods of time resulting in GC having to manage gigs of useless heap space slowing it down even more.

@MagicalTux:

When will the streaming API leave the beta phase so that it can be used for financial applications?

I don't know, but I heard MagicalTux is planning some updates. He's getting bashed pretty badly in the "wall thread" about the trade engine performance. Maybe he's getting rid of the socket.io (it could be a bottleneck for some reason).

I don't know what it is but there must be some severe problems when you look at the machine specs: https://mtgox.com/img/pdf/20120831/Transparency.008.jpg and compare that to estimates of the order throughput that has to be digested and resulting lag that happens.

is plain websocket still considered deprecated? On https://en.bitcoin.it/wiki/MtGox/API/Streaming I cannot find a word mentioning this (anymore). Did they finally get back some common sense? Now the next step would be to fire the PHP-kiddies and hire some experienced Erlang programmers instead to build a brand new matching engine from scratch (or buy one) *before* it all collapses under the sheer weight of unbelievable 5 orders per second.

For last days, automaticaly subscription for data (ticker,trades,depth) is not performed after connection to socketio.mtgox.com in most attempts. Some times is performed with late, after 10-30 minutes.

yes, its a websocket connection too, but before connecting you first do a http request to get your own session ID and then use that to get your very own websocket stream. This puts a lot of unnecessary load on the server during connection for *nothing*, especially when one needs to reconnect often because it drops the connection often. The first step of allocating this personalized websocket URL alone takes 30 seconds or longer sometimes, one can almost hear the server groaning on the other side while it is lifting heavy stuff around to allocate me a personalized socket.io session before I can even connect to it.

Plain websocket most of the time connects instantly.


No, there is no bug on my side, I'm logging the messaes to the console as they arrive, its lagging behind severely. Just try it yourself, run a minimalistic websocket client that does nothing except printing the messages to the console and at the same time run a minimalistic socket.io client in another console window and print the same type of messages and then compare how the messages that you have seen scrolling by in the websocket client will appear 10 or more seconds delayed in the socket.io client.

I can confirm the connection drops (especiall when a lot of shit happens).

I can assure you the problems have already existed before the switch from websocket to socket.io.

I want to put my 2 cents here. First of all, Mt.Gox's socket.io connection indeed drops very often, I confirm this too. However, as for socket.io vs websocket, the thing is that socket.io may use websocket as the underlying transport layer, so that essentially it turns out to be a websocket connection with a few additional characters added into the websocket stream.

So if data lags by 5 seconds, it might be that there is a bug in your trading client.

At ICBIT, I provide the API fully via socket.io not only for trading bots, but for the web trading too. As far as I can tell (and our users can confirm that), it works without this kind of glitches, very smooth, and indeed compatible with many different platforms.

I am not that skilled in Python, alas, however I could help you debug/test the problem by using ICBIT's socket.io connection and see if your problems appear there.

Problems I have found:

* sometimes a signed call like for example "private/info" or "private/orders" will fail for no apparent reason. At the beginning after connection succeeded and I subscribed to all channels (depth, trade, ticker) I send 3 signed calls ("private/idkey", "private/orders" and "private/info"), most of the time this works perfectly but sometimes one or two of them will be answered with "invalid call". In an attempt to work around this I just send them again when this error occurs, as often a needed until it finally succeeds. I don't know if I am missing something here (the API is poorly documented), I suspect a bug on the server side. "invalid call" just is not the correct answer when the server is busy or whatever might be causing this.

* I have not found any way to request fulldepth through the websocket API, again there is documentation missing, is this possible at all?

* I have managed to send trading commands like order/add and order/cancel through the streaming API, most of the time it works but sometimes they just go missing, maybe similar to the other calls mentioned above. This shoud not happen, there should be a queue on the server and a valid order should always go through. The HTTP API seems more reliable for trading but it would be nice if I could use the streaming API for that also.

* It would be nice if there were a way to request OHLCV data of certain timeframes from the server. I can request the trade history and build them myself but this is not practical if I want for example a daily chart of the last 365 days, this wold be a mega-download, possibly hitting the rate limit and banning me. Currently I have no idea where to get OHLCV data of arbitrary timeframes from, please implement this, it will save you a lot of bandwidth.

* The connection is lost very often, sometimes its even impossible to reconnect for periods of many minutes, please fix this. As a first step to remove load from your servers I would suggest completely abandoning this totally unnecessary Socket.io monstrosity that serves no other purpose than demonstrating that some crazy programmer somewhere was able to build the Frankenstein-Monster equivalent of network-protocols. Please drop this nonsense and revert back to good old plain websockets.

* Please improve the documentation. There are a lot of things that can only be found out by experimenting, I would have edited the wiki myself and added lots of useful information that I collected through experimenting and reverse engineering other snippets of code but the Bitcoin-Wiki authorities don't give me write access and I am NOT willing to pay money for being allowed to work on the wiki, the admins seem to have totally lost the connection to reality.

Maybe a multi-million dollar company like MtGox should be able to publish an *official* API documentation on the *official* MtGox website and not (ab)use some 3rd party wiki for it. Also I would expect a multi-million dollar company like MtGox to be able to afford servers that won't crash or freeze every 10 Minutes.


***

For those who are looking for yet another example: Here is my python client implementation: https://github.com/prof7bit/goxtool  it implements both protocols: plain old websocket and the bizarre socket.io-monstrosity that is always lagging behind the websocket by at least 5 seconds (you can configure which one to use)

for those who are interested there is  a possibility to pass several currencies in the url parameter to subscribe for multiple currencies channels

Hi!

I'm programmer with expirience making trading software for russian market (RTS/FORTS), and want to offer a few improvements to current MtGox API.
1. Optional filtering for depth messages, by price range and minimal volume, aggregating depth messages by price rounding to 0.001 BTC  (this help to significant reduce traffic).
2. Broadcating with depth and ticker messages some additional fields:  ticker.last*.timestamp, depth.timestamp  as double in milliseconds from midnight GMT.

Now I adopts a program TradeStudio to work with the MtGox:

Hi,

my task is to use one socket-io client and subscribe to multiple currencies depth channels.

When I connect with url https://socketio.mtgox.com/mtgox
then I automatically is subscribed to depth, tickers and trades channels.

Depth channel is only for USD messages.

When I connect with https://socketio.mtgox.com/mtgox?Currency=EUR, then automatic subscribtion is done for EUR depth channel
then at the same time I managed also to subscribe mannualy to USD channel using command

so currently I have both EUR and USD messages comming at the same time.

The question is how can I subscribe to more than two currency depth channels?

My current solution does look more like a hack and I cannot understand how to subscriibe for third currecny GBP and fourth CNY
in order to have mdepth messages for 4 currencies.

please advice

Is it just me, or have the event feeds come completely unglued today?

EDIT: Appears to be working again.

I tried for a few hours and crawled the forum for a solution but I'm still not able to fetch websocket depth for EUR.


It works fine for USD with (Ruby-code):


but I wasn't able to get EUR depth with the mentioned solutions.

These are not working:

Could someone please help me with that? Thanks alot!