Topic: attn:bitcoin exchanges - request for json api trades by timestamp/tradeid (Read 2345 times)

Originally only suggested or mentioned by Keyur of campbx, but now also agreed upon by Virtex, providing a json api implementation offering results by last trade id or last trade timestamp opens up a kind of cost or risk.

Here's discussion with cavirtex.com:


Perhaps this issue/concern can be discussed a bit to establish a consensually agreed upon idea on how to best implement a secure, stable method of allowing querying by last trade id or last trade timestamp; one that is not susceptible to abuse?

I would like to convince keyur of campbx to implement json api for trades by last trade id or last trade timestamp, however, I am met with a bit of concern from him:


I responded to that statement suggesting:


He responded with:


Discussing in #bitcoinconsultancy about it, joepie91 mentioned memcache, and I agree using it would help, however, I'm fairly confident one could implement an efficient system that doesn't require memcache.  Asking phantomcircuit if he is using memcache for Intersango, he responded he isn't using it since it's not needed yet.

I'm not certain how to respond to keyur, so instead I created this post.  Maybe someone can assist in determining whether my request is asking for too much and that such an implementation is indeed ripe for abuse or not.

Sure.

Example file sizes of json data per exchange market:

bcEUR: 61Kb
bcLREUR: 61Kb
bcLRUSD: 61Kb
bcPGAU: 61Kb
bcmBMUSD: 28Kb
bcmLRUSD: 8Kb
bcmMBUSD: 16Kb
bitchangePLN: 12Kb
bitmarketEUR: 4Kb
bitparkI0C: 1Kb
bitparkIXC: 1Kb
britcoinGBP: 9Kb
btceUSD: 6Kb
btcexUSD: 8Kb
exchbUSD: 38Kb
intrsngEUR: 7Kb
intrsngGBP: 9Kb
intrsngUSD: 3Kb
mrcdBRL: 10Kb
ruxumUSD: 24Kb
thEUR: 1Kb
thUSD: 4Kb

Querying each of these every 30 seconds results in 2,880 queries per day:
@1Kb ~= 2.8Mb/day, 84.3Mb/month
@61Kb ~= 172Mb/day, 5,147Mb/month

This bandwidth consumption is guaranteed to be decreased if showing only latest trades since a specific trade id or date (since the last id or date detected) even with increased query time such a 5 seconds.  Especially in the case of an exchange market having 0 new trades in the last 1 second, then JSON "[]" or similar would be returned, which is only two bytes (1/512th Kb), which additionally greatly reduces bandwidth consumption.

Looking at an example data file above with largest file size (bcEUR): I see the resulting file returns 950 trades, only 29 of them from the last 24 hours, each string of trade data about 64 bytes.  Querying for this data each second showing only latest trades since a particular trade id or date would then return 29 (instances) * 64 bytes of data + (2880 - 29) * ~4 ~= 13Kb/day, 388Kb/month

With these example figures, is your exchange interested in saving approximately 5 gigabytes of bandwidth consumption per month reducing a single instance (for example mine/bitcoinmarkets.com(website coming soon), another maybe bitcoincharts.com) of querying from 5 gigabytes per month to less than half a megabyte (only true with above data, more volume will result in more bandwidth consumption) per month?

Thanks.   Can you add trade ids to the data?


Do you have any JSON APIs for depth/orderbook, ticker, trades?  I didn't see any on your site.

We'd like to work with you on this at https://imcex.com
Please, send me a message.

If I understood you correctly, you want exchanges to provide an api call that lists all trades since a certain timestamp.

Here you are: https://solidcoin24.com/api/trades/btc/s/{timestamp}
Example: https://solidcoin24.com/api/trades/btc/s/1315601224

I opted to post this on the forum instead of emailing each exchange (which I will do anyway to alert them of this thread), which also allows for newly established exchanges to also participate:

I am streaming trades in real-time to #bitcoin-markets on Freenode, similar to #bitcoin-market that I originally established and have since passed on to nanotube (*waves*), but with potentially much faster (the data is multi-threaded and for some markets, trade data results are faster, e.g. 2-10 seconds delay instead of 10-40 seconds) results.  The short delay is true only for exchanges with a JSON trade API implementation that allows for querying all trades since a last timestamp or trade id, therefore guaranteeing downloading only the trade history, reducing bandwidth and processing time.

I have noticed some exchanges that have a discrepancy with sequential trade ids relative to timestamps; some trades have trades ids lesser than other trades but with timestamps more recently/newer than the same trades.  This type of exchange implementation provides a potential issue in the case of querying new trades since last trade id since the particular new trade would be new, but have a much lower trade id than the last trade id from recent queries.  Therefore, if your exchange is like this, it makes much more sense to provide a JSON trade API implementation that references date timestamps instead of last trade id.

If your exchange is interested in adding implementation (or does it already have it?) to query trades either equal to or later than a particular timestamp or greater than a trade id, reply.

Oh, btw, the open source python bot in #bitcoin-markets is still under development.

Also see https://bitcointalksearch.org/topic/json-api-for-trades-45783