MtGox API version 2: Unofficial Documentation - page 4.

nitrous

sr. member

Activity: 246

Merit: 250

Hi zhangweiwu,

With regards to the example, sorry I thought I had tested it. I've pushed a new version that should work and uses the new tonce instead of nonce. Please note also that some API methods must be performed by GET now, and others by POST.

Quote from: zhangweiwu on July 12, 2013, 04:16:29 AM

On another topic, there is little information about how to authenticate against MtGox STREAMING API. Do I must overcome the API2 authentication problem in order to attempt Stream API?

Quote from https://en.bitcoin.it/wiki/MtGox/API/Streaming#Authenticated_commands

Quote

These commands require an API key and secret pair to sign requests. Any of the HTTP API version 1 methods can be called. Responses are op:result

So, if I use Streaming API I cannot use API2, right? I really need the feature like MONEY/ORDER/RESULT which doesn't seem to work in API_1†

--

† with API_1, MONEY/ORDER/RESULT always complain:
{'error': 'No data found', 'result': 'error', 'token': 'unknown_order_id'}

When I know the order ID is prefect existing and open, by double checking result from MONEY/ORDERS, and finding the order ID I request is byte-identical to what MONEY/ORDERS returns. I also know the POST parameters are correct, because if I change post parameter from 'order_id' to 'oid' (as the way it was returned from MONEY/ORDERS) I got missing order_id parameter error.

I've compiled some information about the socket api here https://bitbucket.org/nitrous/mtgox-api/src/master/socket?at=master with example python code.

zhangweiwu

sr. member

Activity: 313

Merit: 250

Quote from: zhangweiwu on July 12, 2013, 04:16:29 AM

with API_1, MONEY/ORDER/RESULT always complain:
{'error': 'No data found', 'result': 'error', 'token': 'unknown_order_id'}

When I know the order ID is prefect existing and open

Turns out this is exactly the problem: if the order is open, there won't be any result of it. Only closed orders can be inquired with this API.

zhangweiwu

sr. member

Activity: 313

Merit: 250

On another topic, there is little information about how to authenticate against MtGox STREAMING API. Do I must overcome the API2 authentication problem in order to attempt Stream API?

Quote from https://en.bitcoin.it/wiki/MtGox/API/Streaming#Authenticated_commands

Quote

These commands require an API key and secret pair to sign requests. Any of the HTTP API version 1 methods can be called. Responses are op:result

So, if I use Streaming API I cannot use API2, right? I really need the feature like MONEY/ORDER/RESULT which doesn't seem to work in API_1†

--

† with API_1, MONEY/ORDER/RESULT always complain:
{'error': 'No data found', 'result': 'error', 'token': 'unknown_order_id'}

When I know the order ID is prefect existing and open, by double checking result from MONEY/ORDERS, and finding the order ID I request is byte-identical to what MONEY/ORDERS returns. I also know the POST parameters are correct, because if I change post parameter from 'order_id' to 'oid' (as the way it was returned from MONEY/ORDERS) I got missing order_id parameter error.

zhangweiwu

sr. member

Activity: 313

Merit: 250

Hi. I have been trying to follow the unofficial documentation for 2 days without success. What frastrate me is even the simpliest case directly taken from example code doesn't work.

The first example code is, forgive me putting up all:

Code:

import hmac, base64, hashlib, urllib2
base = 'https://data.mtgox.com/api/2/'

def makereq(key, secret, path, data):
    hash_data = path + chr(0) + data
    secret = base64.b64decode(secret)
    sha512 = hashlib.sha512
    hmac = str(hmac.new(secret, hash_data, sha512))

    header = {
        'User-Agent': 'My-First-Trade-Bot',
        'Rest-Key': key,
        'Rest-Sign': base64.b64encode(hmac),
        'Accept-encoding': 'GZIP',
        'Content-Type': 'application/x-www-form-urlencoded'
    }

    return urllib2.Request(base + path, data, header)

post_data = 'nonce=123'
request = makreq('abc123..', 'aBc7/+..', 'BTCUSD/money/ticker', post_data)
response = urllib2.urlopen(request, post_data)
# if gzip encoding, decode
# try to decode json into dictionary
# raise exception if response contains error key

And it fail out of the box:

Code:

$ python2.7 test.py 
Traceback (most recent call last):
  File "test.py", line 23, in 
    request = makereq(key, secret, 'BTCUSD/money/ticker', post_data)
  File "test.py", line 8, in makereq
    hmac = str(hmac.new(secret, hash_data, sha512))
UnboundLocalError: local variable 'hmac' referenced before assignment

I "fix" it by renaming the variable hmac to hmac2, and I get:

Code:

$ python2.7 test.py 
Traceback (most recent call last):
  File "test.py", line 24, in 
    response = urllib2.urlopen(request, post_data)
  File "/usr/lib/python2.7/urllib2.py", line 127, in urlopen
    return _opener.open(url, data, timeout)
  File "/usr/lib/python2.7/urllib2.py", line 410, in open
    response = meth(req, response)
  File "/usr/lib/python2.7/urllib2.py", line 523, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python2.7/urllib2.py", line 448, in error
    return self._call_chain(*args)
  File "/usr/lib/python2.7/urllib2.py", line 382, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 531, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
urllib2.HTTPError: HTTP Error 500: Internal Server Error

This is really clueless, if the server just give an 5xxx without any detail. The key/secret used in my case works fine on API_1, and I tried again by applying a new key/secret, which failed with exactly the same 500.

Do you have a code snippet of minimal working code to access an authenticated-user-only feature? E.g. to access wallet information would be perfect. I usually use python3 and am happy to morph a piece of working python2 code to a piece of working python3 code.

The documentation is a rather big topic, worth making up a mailing list or separate board. I feel guilty to make this disordered thread even longer.

The next example I can look up is this:

http://pastebin.com/aXQfULyq

Which also stopped working as soon as I need to do something that requires authentication. From their source it seems they don't even encrypt post data as part of their Sign, thus must fail authentication -> only work for public accessible URIs.

xchrix

hero member

Activity: 905

Merit: 1001

Quote from: nitrous on July 06, 2013, 04:10:25 PM

Great, glad you got it working

mtgox looks like microsoft to me -> trying to make live of devs suck more than others

nitrous

sr. member

Activity: 246

Merit: 250

Quote from: xchrix on July 06, 2013, 03:42:11 PM

got a hint from "Delirium" on mtgox IRC channel. its working now! the "workaround" is to set some curl options which are not documented anywhre at the mtgox API
now i am using this options and everything is working fine:

Code:

		$curl = curl_init();
		curl_setopt($curl, CURLOPT_URL, $url);
		curl_setopt($curl, CURLOPT_RETURNTRANSFER, true);
		curl_setopt($curl, CURLOPT_TIMEOUT, 10);
		curl_setopt($curl, CURLOPT_CONNECTTIMEOUT, 10);
		curl_setopt($curl, CURLOPT_POSTFIELDS, null);
		curl_setopt($curl, CURLOPT_POST, FALSE);
		curl_setopt($curl, CURLOPT_HTTPGET, TRUE);
		
		curl_setopt($curl, CURLOPT_USERAGENT, 'Mozilla/4.0 (compatible; MtGox PHP client; '.php_uname('s').'; PHP/'.phpversion().')');
		curl_setopt($curl, CURLOPT_HEADER, 0);         
		curl_setopt($curl, CURLOPT_FOLLOWLOCATION, 1); 
		curl_setopt($curl, CURLOPT_AUTOREFERER, 1);   
		curl_setopt($curl, CURLOPT_MAXREDIRS, 120); 
		curl_setopt($curl, CURLOPT_SSL_VERIFYPEER, 0);
		curl_setopt($curl, CURLOPT_SSL_VERIFYHOST, 0);
		
		$data = curl_exec($curl);

Great, glad you got it working

xchrix

hero member

Activity: 905

Merit: 1001

got a hint from "Delirium" on mtgox IRC channel. its working now! the "workaround" is to set some curl options which are not documented anywhre at the mtgox API
now i am using this options and everything is working fine:

Code:

		$curl = curl_init();
		curl_setopt($curl, CURLOPT_URL, $url);
		curl_setopt($curl, CURLOPT_RETURNTRANSFER, true);
		curl_setopt($curl, CURLOPT_TIMEOUT, 10);
		curl_setopt($curl, CURLOPT_CONNECTTIMEOUT, 10);
		curl_setopt($curl, CURLOPT_POSTFIELDS, null);
		curl_setopt($curl, CURLOPT_POST, FALSE);
		curl_setopt($curl, CURLOPT_HTTPGET, TRUE);
		
		curl_setopt($curl, CURLOPT_USERAGENT, 'Mozilla/4.0 (compatible; MtGox PHP client; '.php_uname('s').'; PHP/'.phpversion().')');
		curl_setopt($curl, CURLOPT_HEADER, 0);         
		curl_setopt($curl, CURLOPT_FOLLOWLOCATION, 1); 
		curl_setopt($curl, CURLOPT_AUTOREFERER, 1);   
		curl_setopt($curl, CURLOPT_MAXREDIRS, 120); 
		curl_setopt($curl, CURLOPT_SSL_VERIFYPEER, 0);
		curl_setopt($curl, CURLOPT_SSL_VERIFYHOST, 0);
		
		$data = curl_exec($curl);

nitrous

sr. member

Activity: 246

Merit: 250

Quote from: xchrix on July 06, 2013, 02:07:03 PM

thanks for the fast reply! i read your documentation carefully so i know that thing with the tid 218868 and then 1309108565842636. at first i tried to download very slow the whole data but it didnt worked Cheesy

so now i only want the latest trades. but mtgox isnt giving me even this data! if i use since=xxxx (12345 was only a placeholder. didnt used this number) i get NOTHING (all the time)! if i dont use "since" i get sometimes some trades. but most of the time a timeout and zero bytes. i am fetching every 15 minutes!!!! so this shouldnt be a problem.

php curl says

Code:

http://data.mtgox.com/api/2/BTCEUR/money/trades/fetch
Operation timed out after 10001 milliseconds with 0 bytes received

i resisted to add mtgox data to my charts for a long time. but more an more visitors are begging for mtgox charts. every other crap exchange managed to provide an easy and fast API which is working all the time. mtgox isnt able todo this?

Hmm, that's strange that your requests are timing out... the same request worked fine for me. I can only think of two reasons why it might not work. For some API methods, only GET requests are allowed, so make sure you're using that. The other reason is that you might have been blocked, probably when you tried to download the entire database.

Try doing the same request on another IP, and another computer. I know it's not exactly feasible to change the IP of a server, so if it turns out that it has been banned, you might need to go over to #mtgox on freenode IRC and ask if they can unban you (if it was indeed MtGox who did this, it might be cloudflare).

Lastly, if you still want to download the whole data, follow this thread (https://bitcointalksearch.org/topic/new-method-to-pull-mtgox-trade-data-218980). MtGox has setup a Google bigquery database with all data in all currencies up to May 2013, and soon they're going to begin automatically updating this at least every hour. There's some basic instructions at that thread, but I've also written a GUI tool here to download the data. Obviously this isn't suitable for a website, but you might be able to look at the source code for some hints at how to access the bigquery data.

xchrix

hero member

Activity: 905

Merit: 1001

Quote from: nitrous on July 06, 2013, 08:02:58 AM

Hi xchrix,

First of all, please make sure you are not downloading ALL trade data using this method (see https://bitcointalksearch.org/topic/new-method-to-pull-mtgox-trade-data-218980 and https://bitcointalksearch.org/topic/discussion-for-mtgox-trade-data-downloader-221055 for the method to download the majority of trade data. This is not yet completely ready for use, but it will let you download data up to May 2013 at the moment).

MtGox is very particular about what data is available through the trades/fetch api. To get trades since your last fetch, make sure you use the last tid as your since value, e.g. http://data.mtgox.com/api/2/BTCUSD/money/trades/fetch?since=1370025354198298. 123456789 is not a valid tid, and so you will get problems using it. Tids range from 0 to 218868, and then they start again from 1309108565842636.

If you really can't get through to the server, perhaps you have made too many requests? You should limit your api requests to a maximum of 10 requests every 10 seconds, or else you risk being blocked.

thanks for the fast reply! i read your documentation carefully so i know that thing with the tid 218868 and then 1309108565842636. at first i tried to download very slow the whole data but it didnt worked Cheesy

so now i only want the latest trades. but mtgox isnt giving me even this data! if i use since=xxxx (12345 was only a placeholder. didnt used this number) i get NOTHING (all the time)! if i dont use "since" i get sometimes some trades. but most of the time a timeout and zero bytes. i am fetching every 15 minutes!!!! so this shouldnt be a problem.

php curl says

Code:

http://data.mtgox.com/api/2/BTCEUR/money/trades/fetch
Operation timed out after 10001 milliseconds with 0 bytes received

i resisted to add mtgox data to my charts for a long time. but more an more visitors are begging for mtgox charts. every other crap exchange managed to provide an easy and fast API which is working all the time. mtgox isnt able todo this?

nitrous

sr. member

Activity: 246

Merit: 250

Quote from: xchrix on July 06, 2013, 07:52:50 AM

thank you for your great documentation. i am haveing problem with the "trades/fetch" API. maybe somebody knows something about it. i want to integrate the data into cryptocoincharts.info

when i am doing this:
http://data.mtgox.com/api/2/BTCUSD/money/trades/fetch
-> everything works fine

when i want to optimize the query and only get the trades since my last fetch
http://data.mtgox.com/api/2/BTCUSD/money/trades/fetch?since=123456789
-> i get a timeout after some seconds

i think mtgox or cloudflare is blocking this requests from my php curl. the funny thing is when i open the URL with my browser also the "since" url is working!

Hi xchrix,

First of all, please make sure you are not downloading ALL trade data using this method (see https://bitcointalksearch.org/topic/new-method-to-pull-mtgox-trade-data-218980 and https://bitcointalksearch.org/topic/discussion-for-mtgox-trade-data-downloader-221055 for the method to download the majority of trade data. This is not yet completely ready for use, but it will let you download data up to May 2013 at the moment).

MtGox is very particular about what data is available through the trades/fetch api. To get trades since your last fetch, make sure you use the last tid as your since value, e.g. http://data.mtgox.com/api/2/BTCUSD/money/trades/fetch?since=1370025354198298. 123456789 is not a valid tid, and so you will get problems using it. Tids range from 0 to 218868, and then they start again from 1309108565842636.

If you really can't get through to the server, perhaps you have made too many requests? You should limit your api requests to a maximum of 10 requests every 10 seconds, or else you risk being blocked.

xchrix

hero member

Activity: 905

Merit: 1001

thank you for your great documentation. i am haveing problem with the "trades/fetch" API. maybe somebody knows something about it. i want to integrate the data into cryptocoincharts.info

when i am doing this:
http://data.mtgox.com/api/2/BTCUSD/money/trades/fetch
-> everything works fine

when i want to optimize the query and only get the trades since my last fetch
http://data.mtgox.com/api/2/BTCUSD/money/trades/fetch?since=123456789
-> i get a timeout after some seconds

i think mtgox or cloudflare is blocking this requests from my php curl. the funny thing is when i open the URL with my browser also the "since" url is working!

nitrous

sr. member

Activity: 246

Merit: 250

Update to the API examples (https://bitbucket.org/nitrous/mtgox-api/src/master/examples?at=master):

There are now VB.Net, Python, Node.js (Ameen), Java (advanced) and Ruby (Renich) examples, thanks to all the contributors, and definitely check them out if you're writing your own tool or want a starting point

nitrous

sr. member

Activity: 246

Merit: 250

I've been asked about the correct procedure for sending authenticated requests to the socket API, and so I spent a little bit of time translating the PHP example, trying it out, and writing up some detailed instructions on correct usage here:

https://bitbucket.org/nitrous/mtgox-api/src/master/socket?at=master

nitrous

sr. member

Activity: 246

Merit: 250

Hi everyone, forgot to update here but if you haven't heard, there is now a resolution to the trade data problem!
See here for more info - https://bitcointalksearch.org/topic/new-method-to-pull-mtgox-trade-data-218980

nitrous

sr. member

Activity: 246

Merit: 250

Ok, it uploaded quicker than I thought

, here's the link:
https://drive.google.com/folderview?id=0B3hexlKVFpMpTHFjbU5WVUYxVHc&usp=sharing

The file is archive.db, and it is a sqlite3 archive, with tables:

meta
MtGox_USD
MtGox_EUR
MtGox_GBP
~~MtGox_BTC~~

meta gives the last tid downloaded for each table. MtGox_USD, _EUR, _GBP contain all the trade data I have been able to download (should be complete for USD and EUR, definitely incomplete for GBP though). Please ignore MtGox_BTC, that was a bug in my program and I forgot to delete it before uploading! Fields for the trade data tables are:

date (primary, int -- microstamp)
price (int -- amount of auxiliary currency per bitcoin, see docs for info in integer amounts)
quantity (int -- amount of btc traded, see docs again)
type (int -- 0:bid, 1:ask)
primary (int -- 0:N, 1:Y)

Keep in mind that the primary field may not be accurate for MtGox_USD as I didn't think to include it when I downloaded the USD data, and so I set them all to Y. The EUR data should be accurate though.

I won't be downloading any other currencies, and I doubt I'll keep this uploaded archive up to date, however it does have data up to 1369314732232756 / Thu, 23 May 2013 13:12:12 GMT. If this dump isn't months out of date by the time you download this, feel free to keep it up to date using money/trades/fetch though

nitrous

sr. member

Activity: 246

Merit: 250

Regarding the trade data API, MagicalTux told me the following over IRC:

Quote from: MagicalTux

this api change was made because of the very high load caused by all the people downloading the trade history that way. we've been considering different solutions for this, and there's already a very wide variety of available dumps for any currency available on mtgox.

I haven't seen any announcements or dumps anywhere around, however he told me he 'will prepare something' so that we can access these official dumps. In the meantime, please avoid downloading large quantities of data using money/trade/fetch. If enough people are interested before MtGox prepares their dumps, I have an (as of now) up to date dump for USD and EUR (and the very first 187 GBP trades Tongue

). These include the microtime, the price (int), the type (0-bid, 1-ask), the quantity (int), and whether or not it's primary (0-no, 1-yes) - note that for USD, I didn't download whether or not each trade was primary, so I just assumed they all were, this may not be correct. Anyway, my internet connection is quite slow, but I'll endeavour to upload it somewhere given enough demand

nitrous

sr. member

Activity: 246

Merit: 250

Quote from: btc_lurker on May 21, 2013, 09:43:54 AM

I suggest giving their IRC channel a try. Last time I went there, MagicalTux answered the questions I had. But this is a tad different, I'm not sure they are willing to change their API to the earlier behavior -- but hopefully it was just a mistake somewhere, and then the situation will be reverted.

MagicalTux told me that the API change was intentional, and was because all of us are trying to download all their trade data! There are some database dumps though which he is going to prepare some links to, I'll update the docs and give some more info in about 8 hours time.

nitrous

sr. member

Activity: 246

Merit: 250

Quote from: btc_lurker on May 21, 2013, 09:43:54 AM

I suggest giving their IRC channel a try. Last time I went there, MagicalTux answered the questions I had. But this is a tad different, I'm not sure they are willing to change their API to the earlier behavior -- but hopefully it was just a mistake somewhere, and then the situation will be reverted.

Ok, thanks. I tried now but it seems he's offline, not surprising considering its 12am there Tongue

I'll try to contact him tomorrow morning if I get the opportunity, I might be able to catch him around 6pm JST.

btc_lurker

member

Activity: 78

Merit: 10

I suggest giving their IRC channel a try. Last time I went there, MagicalTux answered the questions I had. But this is a tad different, I'm not sure they are willing to change their API to the earlier behavior -- but hopefully it was just a mistake somewhere, and then the situation will be reverted.

nitrous

sr. member

Activity: 246

Merit: 250

Quote from: btc_lurker on May 11, 2013, 05:01:27 PM

It looks like MtGox changed its behaviour regarding fetch of old trades. We all know there is a jump at transaction id 218868, but I'm pretty sure it was possible to continue fetching data by specifying ?since=218868 and so on. But now MtGox is returning no trades at that point, and it is necessary to manually to do the jump.

This was reported to me at https://github.com/knowitnothing/btcx/issues/1 and can be seen by visiting https://data.mtgox.com/api/2/BTCUSD/money/trades/fetch?since=218868

Do you know the best way to contact the MtGox API team? I've been able to download all USD trade data by performing the jump manually, but this fails for some currencies (e.g. GBP), because the data seems to be very dispersed, and so there are numerous jumps at unpredictable places.

Also, for anyone who wants to download EUR data, it seems that you need to start downloading from 1315289573000000 (I'm not sure if there is any earlier data). I hope there's a way around this, as otherwise anyone who wants to download less popular currency trades will have to do small test jumps repeatedly until they reach the next data 'pocket'!

Topic: MtGox API version 2: Unofficial Documentation - page 4. (Read 62428 times)