Discussion for MtGox trade data downloader

VinCeCream

member

Activity: 89

Merit: 10

Quote from: BitOnyx on March 11, 2014, 06:00:33 AM

Quote from: blankia on March 11, 2014, 05:55:25 AM

Im getting an error when loading the dump:

Traceback ( most recent call last):
File "app.py", line 97 in thread_bootstrao
File "mtgox.pyc", line 120, in run
File "bq.pyc", line 103, in gen2
KeyError:'pageToken'

maybe because mt.gox is down...

EPIC!

blankia

newbie

Activity: 27

Merit: 0

Quote from: intron on March 11, 2014, 10:39:53 AM

Quote from: BitOnyx on March 11, 2014, 06:00:33 AM

Quote from: blankia on March 11, 2014, 05:55:25 AM

Im getting an error when loading the dump:

Traceback ( most recent call last):
File "app.py", line 97 in thread_bootstrao
File "mtgox.pyc", line 120, in run
File "bq.pyc", line 103, in gen2
KeyError:'pageToken'

maybe because mt.gox is down...

LOL:-)

Hahaha nice one

Thanks for pointing that out. I thought the old data was still hosted on Google BigQuery?

intron

sr. member

Activity: 427

Merit: 251

- electronics design|embedded software|verilog -

Quote from: BitOnyx on March 11, 2014, 06:00:33 AM

Quote from: blankia on March 11, 2014, 05:55:25 AM

Im getting an error when loading the dump:

Traceback ( most recent call last):
File "app.py", line 97 in thread_bootstrao
File "mtgox.pyc", line 120, in run
File "bq.pyc", line 103, in gen2
KeyError:'pageToken'

maybe because mt.gox is down...

LOL:-)

BitOnyx

member

Activity: 112

Merit: 10

Cryptocurrencies Exchange

Quote from: blankia on March 11, 2014, 05:55:25 AM

Im getting an error when loading the dump:

Traceback ( most recent call last):
File "app.py", line 97 in thread_bootstrao
File "mtgox.pyc", line 120, in run
File "bq.pyc", line 103, in gen2
KeyError:'pageToken'

maybe because mt.gox is down...

blankia

newbie

Activity: 27

Merit: 0

Im getting an error when loading the dump:

Traceback ( most recent call last):
File "app.py", line 97 in thread_bootstrao
File "mtgox.pyc", line 120, in run
File "bq.pyc", line 103, in gen2
KeyError:'pageToken'

toniquone

newbie

Activity: 4

Merit: 0

Quote

Well bid/ask is a whole new story. But if you want to have whole Orderbook at any given point of time that is gigantic masses of data. That is not managable i guess. Even for stock market, this is not commonly used.

don't know about whole orderbook, but nitrous db has this column (not included in standard export), and it's very handy for backtesting - strategy chooses more correct price on buy/sell actions.

BNO

full member

Activity: 157

Merit: 100

Quote

I see only one problem with the bitcoincharts data - there is no bid/ask column Sad
In other ways data seems to be enough accurate.

Well bid/ask is a whole new story. But if you want to have whole Orderbook at any given point of time that is gigantic masses of data. That is not managable i guess. Even for stock market, this is not commonly used.

toniquone

newbie

Activity: 4

Merit: 0

I see only one problem with the bitcoincharts data - there is no bid/ask column Sad

In other ways data seems to be enough accurate.

BNO

full member

Activity: 157

Merit: 100

Hi nitrous,

good to hear from you again.
O.k. that sounds good with the quality. Even if order in the 1second Intervall is mixed up, its not that bad, its just a second i guess.

O.K. have look for myself and post if i find something.

If someone else has some more info on this - don't be shy Wink

greetings to all of ya..

nitrous

sr. member

Activity: 246

Merit: 250

Quote from: BNO on January 17, 2014, 12:15:13 PM

Hi all,

at one point nitrous posted this link:

Quote

http://api.bitcoincharts.com/v1/csv/

Can anyone comment on dataquality/accuracy known problems/oddities of the bitcoinchartsdata for mtgox, bitstamp and btc-e?
Nitrous do you have an opinion about this?

Greetings to all of you!

Difficult to tell, it seems good, you'll need to judge that for yourself though I think. I have only looked at the bitstamp data, I can't comment on the other exchanges, but for bitstamp the one oddity I found is that the trade times are given with second resolution, and so it's difficult to tell the order when multiple trades happened in the same time period. It's likely that they are given in the correct order, but I'm taking a VWAP of each second to avoid any potential problems with that.

BNO

full member

Activity: 157

Merit: 100

Hi all,

at one point nitrous posted this link:

Quote

http://api.bitcoincharts.com/v1/csv/

Can anyone comment on dataquality/accuracy known problems/oddities of the bitcoinchartsdata for mtgox, bitstamp and btc-e?
Nitrous do you have an opinion about this?

Greetings to all of you!

nitrous

sr. member

Activity: 246

Merit: 250

Quote from: kyotoku on December 31, 2013, 04:04:22 PM

Hi nitrous,
I tried running the app in OS X 1.6.8 (snow leopard). But it crashes after I select the dump file name in (File->new dump). It prints segmentation fault at the prompt.

I was able to track the error to self.update() call at the end Application.load_dump function at gui.py file (line 217). It actually inconsistent, when I place some print around update() call, it works some time. Anyway, it ran fine when I commented the self.update() line out. I think this function updates window visual after update_button state is set to normal, isn't it? I don't have a better idea of what it could be. But I am happy to test on OS X 1.6.8 if you have any idea.

If anyone is interested I posted a version with the alteration described above
https://www.dropbox.com/sh/cyql48n834n9sfq/VWuOLxYA79

Cheers

Ahh, thank you for posting a working version. Yeah, I used Tkinter but it doesn't work well with multithreading, hence the numerous errors we've all encountered with this. It doesn't seem to work with Mavericks either. Unfortunately, I don't really have time to maintain this anymore, but hopefully it should continue to work with the dump I posted for now.

Thanks again for the new update

!

kyotoku

newbie

Activity: 32

Merit: 0

Hi nitrous,
I tried running the app in OS X 1.6.8 (snow leopard). But it crashes after I select the dump file name in (File->new dump). It prints segmentation fault at the prompt.

I was able to track the error to self.update() call at the end Application.load_dump function at gui.py file (line 217). It actually inconsistent, when I place some print around update() call, it works some time. Anyway, it ran fine when I commented the self.update() line out. I think this function updates window visual after update_button state is set to normal, isn't it? I don't have a better idea of what it could be. But I am happy to test on OS X 1.6.8 if you have any idea.

If anyone is interested I posted a version with the alteration described above
https://www.dropbox.com/sh/cyql48n834n9sfq/VWuOLxYA79

Cheers

nitrous

sr. member

Activity: 246

Merit: 250

Quote from: BNO on December 31, 2013, 07:45:15 AM

...

Oh when i became interested in BTC this year again one thing i was really clear about was: I won't put again money in Mtgox. And i was happy for bitcoin that now there are three Big USD BTC Exchanges: btc-e, Bitstamp and Mtgox. I tried bitstamp (really like look and feel of the site, customer service replies fast, cashing out no problem so far) btc-e (service is a bit harsh) and bitfinex. Have to say Kraken looks nice maybe i give it shot. Volume is probably low so and prices are a bit higher, withdrawal problems?

The reason why i wan't to use Mtgox Data is no one goes back so far. And since Mtgox has a significant higher price than other exchanges its maybe not so wise to mix the data between exchanges.

...

Do you know something about timestamp with Bitstamp? Just downloaded the data, i assumend unix timestam. i converted with like this: A1/(24*3600)+25569 gives me as first trade:
13.09.2011 13:53:36

is that right Bitstamp was already back then in 2011 around - wow should have known that back then would maybe have saved some money Wink

That's fair enough. You should probably be careful about using strategies developed for MtGox with other exchanges. Simple strategies are probably fine, but you might find subtle differences. And these will probably be amplified for more complex strategies. You should be able to compensate for this with some minor tweaks though, so using MtGox data initially should still be fine.

Yeah, Kraken is quite new. Sometimes their 24h XBTEUR volume peaks above 1000 though. I think there have been a few introductory problems as they've grown in popularity, but I think they're getting through this now.

Yes, they are unix timestamps and that was their first trade (or, at least, the first one recorded by bitcoincharts).

Haha, hindsight is a wonderful thing Wink

BNO

full member

Activity: 157

Merit: 100

Quote

Hmm, well the last timestamp from the 2nd dump is consistent with the time I made it, so it looks like recent data is at least consistent with UTC.

I'll just trust this data, hope i won't regret this one day.

Quote

Personally I've been looking at other exchanges more recently, so you might want to consider other exchanges too, especially as you're worried about whether you can trust Mark. I'm quite interested in Kraken as it looks like it has a lot of potential, but I'm primarily using Bitstamp at the moment. There's some published CSVs for many exchanges here http://api.bitcoincharts.com/v1/csv/, although obviously then you have to trust bitcoincharts to have collected the data properly (I haven't analysed their data yet, but I assume it's fairly reliable; from a quick look the bitstamp one definitely seems to include the major price spike at roughly the right time).

Oh when i became interested in BTC this year again one thing i was really clear about was: I won't put again money in Mtgox. And i was happy for bitcoin that now there are three Big USD BTC Exchanges: btc-e, Bitstamp and Mtgox. I tried bitstamp (really like look and feel of the site, customer service replies fast, cashing out no problem so far) btc-e (service is a bit harsh) and bitfinex. Have to say Kraken looks nice maybe i give it shot. Volume is probably low so and prices are a bit higher, withdrawal problems?

The reason why i wan't to use Mtgox Data is no one goes back so far. And since Mtgox has a significant higher price than other exchanges its maybe not so wise to mix the data between exchanges.

Quote

http://api.bitcoincharts.com/v1/csv

Did i already say: That's a good link. Thanks.

Do you know something about timestamp with Bitstamp? Just downloaded the data, i assumend unix timestam. i converted with like this: A1/(24*3600)+25569 gives me as first trade:
13.09.2011 13:53:36

is that right Bitstamp was already back then in 2011 around - wow should have known that back then would maybe have saved some money Wink

nitrous

sr. member

Activity: 246

Merit: 250

Quote from: BNO on December 30, 2013, 06:00:51 PM

Quote

Have you found an inconsistency?

No i was just thinking about it since its very destructive if you look at trading data and do not realize that the data might be looking 1 or 2h in the future.

Quote

UTC is pretty easy to get right, his servers should just automatically sync with a reliable timeserver in order to get reliable UTC.

Hope/guess you are right, but when i look at the "track record" of Mark Karpeles, i'm not sure. What he fucked up in the last 2 years is an achievement in itself.

Good nite its late here...

Hmm, well the last timestamp from the 2nd dump is consistent with the time I made it, so it looks like recent data is at least consistent with UTC. I suppose the only advice I could give is to do a dry run of any algorithm you develop first, just to see whether it makes the right choices in a period of, say, a week, before risking any real assets.

Personally I've been looking at other exchanges more recently, so you might want to consider other exchanges too, especially as you're worried about whether you can trust Mark. I'm quite interested in Kraken as it looks like it has a lot of potential, but I'm primarily using Bitstamp at the moment. There's some published CSVs for many exchanges here http://api.bitcoincharts.com/v1/csv/, although obviously then you have to trust bitcoincharts to have collected the data properly (I haven't analysed their data yet, but I assume it's fairly reliable; from a quick look the bitstamp one definitely seems to include the major price spike at roughly the right time).

Good night

BNO

full member

Activity: 157

Merit: 100

Quote

Have you found an inconsistency?

No i was just thinking about it since its very destructive if you look at trading data and do not realize that the data might be looking 1 or 2h in the future.

Quote

UTC is pretty easy to get right, his servers should just automatically sync with a reliable timeserver in order to get reliable UTC.

Hope/guess you are right, but when i look at the "track record" of Mark Karpeles, i'm not sure. What he fucked up in the last 2 years is an achievement in itself.

Good nite its late here...

nitrous

sr. member

Activity: 246

Merit: 250

Quote from: BNO on December 30, 2013, 06:22:25 AM

How was the timestamp of the data again created. I know i read it somwhere but don't remember. Was it UTC timestamp is there a problem with daylight savings time of Japan? As far as i understand UTC is a constant "world time" and all timezones of the earth a derived from UTC.

Is something known, that there is some problem with the timestamps like Mark K. changed (for whatsoever reasons) , the timesettings of servers or something like this?

Have you found an inconsistency?

I'm pretty sure they should be consistent with UTC, I take the timestamps directly from that returned by each one of the APIs, and I don't think there are any problems. If you think there are though, tell me which ones and maybe I can have a quick look. If you're just speculating though, then you should be fine. UTC is pretty easy to get right, his servers should just automatically sync with a reliable timeserver in order to get reliable UTC.

BNO

full member

Activity: 157

Merit: 100

How was the timestamp of the data again created. I know i read it somwhere but don't remember. Was it UTC timestamp is there a problem with daylight savings time of Japan? As far as i understand UTC is a constant "world time" and all timezones of the earth a derived from UTC.

Is something known, that there is some problem with the timestamps like Mark K. changed (for whatsoever reasons) , the timesettings of servers or something like this?

gigica viteazu`

sr. member

Activity: 458

Merit: 250

beast at work

thank you

Topic: Discussion for MtGox trade data downloader (Read 14374 times)