Topic: Discussion for MtGox trade data downloader - page 4. (Read 14394 times)

very nice tool for running scenarios! thanks!

I tried again today but I didn't get an answer. Tomorrow I'll try to contact him earlier, hopefully during Japanese business hours.

Any idea when the trade data will be updated?
It would be interesting to backtest trade strategies against the bear market after May 23rd.

Glad you've found it useful Yes, that's right. Here's an example of one script that's already been created - http://cahier2.ww7.be/bitcoinmirror/phantomcircuit/. This is a different format though, and doesn't store all the data (I did however port this to an export format). You'll need to determine which field in the API maps to which field in the BQ database, and fill in some that isn't included in the API (Bid_User_Rest_App__ for example).

There are some discrepancies with the API data - the missing fields, data being split into individual currencies, currencies occasionally being silently added, gaps in the API data, etc. Whilst this shouldn't be too much of a problem for your actual data processing, it could cause problems if you ever want to update from BQ again. I suggest that you instead insert the data from the API into a new table (say, dump_live). You can then create a view from a union between dump and dump_live, delete the obsolete index on dump, and index the view (Money_Trade__ ASC,[Primary] DESC,Currency__ ASC). This will allow you to still access all the data fairly fast, but without corrupting the BQ dump. If you don't mind an extra 200mb, you can leave the index on dump. My tool will recreate this index if it doesn't exist, so it might be easier and more efficient to just leave it.

I've actually given thought to putting this in the official tool, hence the insights I had above (I'm not sure when I'll have enough time to do that though, it would mean changing quite a lot of the tool), so if you want any help with your script let me know, especially as I've also written documentation on the MtGox API v2, so I can help you map the API data to the BQ schema and get around the data jumps

So I am teaching myself how to use Postgresql (among other things), and I thought "what better way could there be than finally getting around to messing with the mtgox historical trade data?" (I had been wanting to analyze this data for sometime). After searching around for a while, I came upon this thread.

Thank you so much for this awesome tool! You have saved me much time/effort It was very easy to start a dump, and the green bar is moving to the right currently so it looks like it is working! Just curious, the easiest way to fill in the missing data (as the Big Query db is not yet being updated regularly) would be to fetch trades with IDs after the ID of the last trade in the dump, yes? I was thinking writing a small python script using the API commands as specified in this post: https://bitcointalksearch.org/topic/m.2067244

Ok, I've now written an in-depth readme that should explain how to do everything possible with the tool . There are screenshots for Mac, but the Windows interface is exactly the same so that shouldn't cause any problems. You can download it here:

https://bitbucket.org/nitrous/bq/downloads/Readme.pdf

I have also updated the landing page readme.md with a quickstart guide.

Yeah, sorry I haven't got around to doing the readme yet. Hopefully I won't be busy tomorrow and I'll get on it.

err..
a dummie's guide to step-by-stepping to create a dump, first^^

Ok, I'll have to think of another way to get the progress bar working.


Awesome!

Thanks Epoch and Diabolicus for your patience and prompt testing

Not working on Win7 64Bit. Want me to test it with aero or something else disabled?
Cannot test 32Bit before monday, I'm not at the office any more.

However, this version
is right now exporting 15min candles since 2011 like a boss (Win 7, 64 Bit)

I will check thoroughly later, but it seems to work. Great job!

Yes, finally! I'm tentatively hopeful that this success will continue I'll be exploring possible solutions to the progress bar problem, as exports from a full dump can take a very long time; in the meantime, you can check the filesize of the export to see that it is in fact increasing, although that won't tell you how much longer there is to go.

I just tried the latest Windows binary (without status bar) and can confirm it has successfully exported 3600 second candles. I"ll try some other values as well and report if I encounter any errors. Win7/64.

For some reason bitbucket didn't replace the version with the new one I uploaded, and now it's bugging out so I can't upload the new version. For now, I'm going to just host it on Dropbox until I can make sure bitbucket is working properly. Here's the new version. To verify it, you should get some red text at the top of the export window on Windows. The red text says that I've disabled the progress bar on Windows because I believe that is responsible for most of these crashes. Hopefully I can reenable it at some point in the future, but for now try these links. I am really sorry about all this hassle, and thank you for your patience. I really hope that this version does actually work.

Mac - https://dl.dropboxusercontent.com/u/1760961/MTT/v1.0.3/Binary%20(Mac).zip
Windows - https://dl.dropboxusercontent.com/u/1760961/MTT/v1.0.3/Binary%20(Windows).zip
BitBucket seems to be working again now, so you should be able to download from here, the links above are still available if not though.

If you are doing a large export and would like to have a progress bar, try this version and please tell me if it works for you:
Windows - https://dl.dropboxusercontent.com/u/1760961/MTT/v1.0.2/Binary%20(Windows).zip
(The latest mac version already includes the progress bar)

Windows 7, 64 Bit:
When trying to build 1h candles from the whole 640MB dump.sql, after about 10-15% done it crashes with the following error:
Windows 7, 32 Bit:
Runtime error after ~20% done:

Nope, same problem as the previous build. It crashes on export and does not generate any .log file. I'm running Win7/64.

Ok, I'm now cautiously optimistic that the new version I just pushed should now work on windows 7 (at least 32 bit). I quickly tested it right now though (OS X 10.8, XP 32bit, 7 32bit) and it seems to be fully functional, touch wood.

Also, I've only experienced this a couple times, but when the Google library fails to get a connection to the Google servers quickly enough it can sometimes crash, I'll see if I can stop it but in the meantime just open the program again and it should be fine. You'll get a logfile if this happens.

Ok, my current conclusion is that Tkinter is broken in windows 7 I thought I found the (sort-of) problem, I fixed it, and then another bug came up, entirely due to tkinter. (Tkinter is the graphical library I use, and I use it for around a third of my code!) I believe I have now isolated the bug though and I think I've got around it satisfactorily. Sorry about that.

There are some other changes I also need to make (you still get an error when trying to update a CSV (price/volume) [ISO], for example), so expect a new release tomorrow, and I'll try to do all I can to make Tkinter work. In future, you'll know if it crashes because of Tkinter because you won't get a traceback, just something like 'python.exe has stopped working'. If you get that, please tell me the exact circumstances that led you to get that error so I can try to reproduce it, as without a traceback I have no idea where the bug is coming from.

Weird, it worked fine on my XP but I just tested it on windows 7 and I see what you mean. For some reason, Tkinter (the library responsible for the graphics of the app) is causing a crash, and I'm not sure why at the moment. I'll keep looking into it. Also, I have access to OS X 10.8, Windows XP and Windows 7, does this problem occur in Vista as well?

Same here, still crashes and no log.