Topic: Data source url change (ticker, depth, history) - page 2. (Read 17640 times)

anyone wanting the easy PHP code to get a ticker running as trying to find an easy copy and paste on the wiki takes many pages of searching


all you have to do is change the BTCGBP in the url to your desired currency and ofcourse the £ in the echo code needs changing if your not looking for english pounds

MagicalTux, please read the following posts on this thread, and learn from this highly unprofessional failure at Mt. Gox by improving your business practices.  An official apology from Mt. Gox regarding this serious failure over the past few days would help customers to regain confidence.  Hopefully you learn from this and make better business decisions in the future to avoid embarrassing situations like this for Mt. Gox: https://bitcointalksearch.org/topic/mtgox-apis-down-156833

Ah there it is!

https://bitcointalksearch.org/topic/mtgox-apis-down-156833

Well, you suck. Why I should dig forums and newspapers what the fuck is going on with my apps?

Could you please inform about futher similar epic changes like that at least on YOUR FRONT PAGE NEWS? And at most in email newsletter?

Hi Tux,

I think the json data returned by data.mtgox.com could be compressed or reduced. Let's compare the returned json data of regular DepthTable of http V0 version and V2 version:

V0 version:

https://data.mtgox.com/api/0/data/getDepth.php



v2 version

https://data.mtgox.com/api/2/BTCUSD/money/depth



json data size compare:




From above , I suggest that Http v2 version json of depth use the format of http v0 which could reduce more than 90% data size. Or you could place a "switch" like

https://data.mtgox.com/api/2/BTCUSD/money/depth/reduced
or
https://data.mtgox.com/api/2/BTCUSD/money/depth?format=reduced

to let people get the proper data they want.

The website like bitcoin.clarkmoody.com  or info.btc123.com (btc123 caches json data itself for 10 seconds) is designed to maintain a realtime data of ticker , most recent trades and current depth. There could be a very frequent data getting requirement.

Also for ticker and trades:

ticker v0 version:


ticker v2 version:


trade v0 version:


trade v2 version: (The great is that v2 version add trades property "market" "limit" to each transaction.)


From above I think there is a lot of information could be removed from json data, maybe only the yellow background data is essential. And I suggest that the ticker and trades data format could take v0 version depth for reference, or add a switch (or say an option) to provide a version for those who just want to get realtime market information (like info.btc123.com) to get the reduced or compressed json data.

Many Thanks!

No change on the socketio url, however we are testing a new system that'll be much better to handle the load. Tests are currently being done.

Hi, You did not mention socket.io api url changes. Will it change too?

And if yes do you know about current socket.io behaviour? Client connects, connect event invoked, and then no any subscribtion message appear. Just silence.

Can you comment on this effect? What can be done?

Dear god.

Don't know how I missed this thread.

Everything updated.

Will the ticker always stay at HTTP to prevent any overhead of HTTPS (SSL) ?

Thanks.

http://data.mtgox.com/api/1/BTCUSD/ticker

Requesting the same API more than 5 times per hour can be in production mode too - in case when my engine supports more than 5 currencies orderbooks at the same time - when socket reconnects I will need to synchronize all supported orderbooks in the same moment. Then I request full orderbooks for every currency i support. And first test show the first 5 http requests work and the remaining requests get timout response. Thanks to the quite complext logic of managing timeouts for each currency orderbook my engine did not get blocked yet. But it would be good that you give your understanding of this situation and maybe suggest some possible solution in this case?

Is this preferred over websockets? Since the ws server is down more often than its up, Im falling back to "hammering" your system with ticker requests over http. This seems contraproductive ?

Also Im confused about the depth api. How can I request depth changes since a given change or txid?
What depth messages does http://data.mtgox.com/api/2/BTCUSD/money/depth/fetch provide ?

Never mind, figured it out. I missed the  "filter_min_price" and  "filter_max_price" records, that explains. How often can we call this api?

Thanks MagicalTux. I've switched one of my feeds over. Is data.mtgox.com based on another server? What is the latency between data.mtgox.com and mtgox.com?

I also have a suggestion for cutting your bandwidth usage in half... A typical ticker request looks like this:


The most common request must be for an up to date ticker, however I'm willing to bet that the majority of apps and bots will only ever require Buy, Sell and last price.

high, low, avg, vwap, vol, last_orig and last_all will be redundant to a lot of sites.

Following on from prof7bit's suggestions about cutting the data responses down for redundant data, can you produce a second ticker feed with a more condensed dataset? This will speed up response times for sites requesting data too.

Alternatively as a suggestion is there a way to code it so url parameters decide what should be returned? Eg something like http://data.mtgox.com/api/2/BTCUSD/money/ticker/bid/ask/last

It would really be a great help. Imagine a bot that needs to place orders at certain prices if certain conditions are met but under no circumstances may place the same order twice. Ideally it would look up in its own order list whether its orders are there already (sent but not yet acked or acked and pending or open) or whether they are still missing.

Currently this is not easy: One possibility at the moment would be for my bot to generate an ID and send it in the id field of the signed call and then wait for the op:result that returns this id *and* the official mtgox oid and then update my own database to link these two ids together. But unfortunately the op:result is the only message that will ever refer to this user-generated ID, its the only link between my ID and mtgox ID, if its ever lost for some reasons (temporary disconnect between order/add and op:result during busy times, etc) I cannot reconstruct it. did the order/add goo through? is it pending? is it missing? If it suddenly appears 20 minutes later how do I know where it came from? If I query the private/orders it will not contain these user-generated IDs, I cannot know which of the existing orders belong to which of my bots or belong to manual trading.

Ideally I would want to be able implement a send_order() function in my trading client that is non-blocking and immediately returns an (internal) ID that I can rely on that will survive disconnects, restarts, all kinds of errors, etc. and my cancel_order() function and other order functions would always be able to use this internal ID to refer to the order.

Currently the only reliable way to do it is to encode such additional information in the least significant bits of the order volume: Instead of buying 1 BTC my bot would buy 1.00000042 or + , so each of my bots can recognize their own orders and restore this important part of their state from a simple private/orders query. But I consider this a dirty hack. Its only a workaround.

Such an ID field would not have to be large, people would not need to be able to store their entire autobiography in this field, 32 bit unsigned int would be plenty enough for this purpose.

In the last 24 hours, data.mtgox.com has served 143513 requests for a total of 25.9 GB bandwidth.

That's just a fraction of what's being served on mtgox.com, but that helps already quite a lot.


If that's for uses such as development, then it's fine. Some websites actually hit our ticker backend once for every page load on their side.



None, but your suggestions are welcome on [email protected]. I've read what you posted here and will use it. I'm not sure about user-defined order ids, as it's going to be using a lot of storage on our side, and all data is already gzipped if supported by the user agent. We'll be checking your other suggestions.

There are other reasons for requesting the same API more often than only 5 times per hour, for example when developing client software and testing it. Your API should just be able to take such usage without a problem, after all its mostly only reading and not writing, so it should not put any noticeable stress on your servers.

A few suggestions: If you ever decide to make an API V2 then please consider the following:

 * remove all the redundant different representations of the same number (value, display, display_short), only use value_int. There is a lot of totally redundant data in these structures, if you remove it all it would probably cut down bandwidth by 2/3 and your server would not need to waste time with string formatting.
 * gzip all data
 * allow orders to carry a user defined ID number, so I can generate my own order-id *before* submitting it to mtgox. Then when requesting "private/orders" or inside the private:user_order message each order would have two IDs, my own and the official mtgox oid. Then I could use my own ID for all my internal bookkeeping in my own database during the entire lifetime of the order and use the mtgx id only for commands that I send to mtgox.
 * please provide historical OHLCV data of various time frames, they don't all need to have the entire history, just 100 bars each would be enough already, this would remove the need for many people (including me) to request huge amounts trading history if one just quickly want to plot a graph where for example each 15 minutes or 4 hours or an entire week of trading is consolidated into only 5 numbers.

And finally: Is there a thread or a section on this forum specifically for mtgox feature requests/problems/questions/etc?

PS: the websocket server is working again, did you fix it or does it go off and on on its own every Friday? Maybe its the cleaning lady who needs a wall socket for the vacuum cleaner every Friday, maybe you should just lock the door to that room to avoid these problems?

The limits are lifted, however spamming will still end in your IP getting blocked (not by us, but by CloudFlare, using their own method to choose to block you).

Either way getting the same data again and again will not result in new data being returned, remember this.

thanks for the info

IIRC, the "old" mtgox.com (non-socket) API had some "query it 10 times and get blocked for 24h" policy.
Are those limits lifted for the new data.mtgox.com API?

https://mtgox.com/api/1/BTCUSD/fulldepth is actually an alias to https://mtgox.com/api/1/BTCUSD/depth/full

The websocket server does not respond anymore since yesterday. It connects instantly but then its totally silent, not even sending headers.

(Socketio-websocket still "works" but with a delay of 2+ Minutes on top of the usual order-lag, it won't even ack my calls to give me an oid for my pending orders in less than 2 minutes even when order-lag says "idle", this is almost unusable for trading)

Is websocket.mtgox.com affected by the change? Is it a known problem, Is somebody already working on the problem and will it be fixed?