Topic: Discussion for MtGox trade data downloader - page 5. (Read 14374 times)

I've just downloaded and tried the new version; still crashes on CSV export. However, this time it did not generate any .log file (the previous version created a logfile in the same directory as the executable). Any special steps needed to get the debugging info?

Ok, I've released a new version It should fix the bugs that have been reported and give more helpful error messages in the cases identified.

Ok, that bug should be fixed. Expect a new release later today (I'll post again when it's ready). Improved error logging will only apply to the bits of code found to have the potential to cause an error, unexpected errors will still cause a crash unfortunately. When I have enough time I may read through all the code to try to find any more that could happen.

Gekko should work with the format set to 'CSV Candle [Unix]', and volume unchecked. I think in the future he plans to include volume in his candles. Technically, Gekko should still work with volume left checked as his backtest loader script currently ignores any extra fields, but that could change. Make sure to select only one currency, and the other settings are up to you. Also, WhyDifficult's published CSVs included non-primary trades. I usually don't include them, but again that's up to you.

I'm also getting a crash under Win7/64 when trying to export a CSV (candles, UNIX time, do not include volume, 1800s duration ... haven't tried the others). Log says:

Once a new release is out with improved error logging, I'll run it again to see what it shows.

A question: (assuming the crash is fixed in the next release) does anyone know what settings should be chosen for compatibility with WhyDifficult's trading bot?

Ok, thanks, I will test again as soon as you release the update.

Hi Diabolicus,

Thanks, the 'time' error was a bug (though it only occurs after the program crashes). Your other errors appear to be you trying to update a CSV file which was created with a different format. There are five types of CSV file that can currently be produced:

CSV (price/volume) [ISO]
CSV Candle [ISO] - no volume
CSV Candle [ISO] - with volume
CSV Candle [Unix] - no volume
CSV Candle [Unix] - with volume

None of these are compatible with each other. It should work as long as you remember which options you used (I recommend saving your export settings with the Save button in the export window and then reloading them with the Load button so that the settings are exactly as they were before).

I've now fixed the time bug, and I've made the app produce some user-friendly error messages instead of crashing on a CSV file error. I haven't updated the binaries yet though in case there are any changes to make.

Hopefully there aren't any bugs specific to Windows 7, but please let me know as I've only personally tested it on OS X Mountain Lion and Windows XP.

It crashes when trying to export, no matter what settings I choose.
Tested on Windows 7 64bit and 32bit.
Log says:
or

Great Let me know and I'll see what I can do.

trying thiso out ASAP.
might have some feature requests, happy to donate.

Many thanks. Please let us know when you know something 

Sorry, not yet. I've asked a couple times but I've yet to receive an official response.

Fantastic.


Do you perhaps know when MtGox will start updating the database?

Ok everyone, I've completed the tool!

You can download a binary for mac or windows here:
https://bitbucket.org/nitrous/bq/downloads

The source is still available here:
https://bitbucket.org/nitrous/bq
And you can build your own binary if you want.


The tool is ready to download from the bigquery database, and to export to a number of formats.

• CSV (price/volume) [ISO]
• CSV Candle [ISO]
• CSV Candle [Unix]
• Sqlite3 PhantomCircuit

The first option is the format Loozik needed, a csv file with headers date,time,price,volume. Date and time are in ISO 8601 format.

The next two generate candlestick data using a port of https://bitcointalksearch.org/topic/downloadable-mtgox-historical-candles-239815. The first one uses ISO date formats and the second, Unix timestamps.

The last should be compatible with PhantomCircuit's script http://cahier2.ww7.be/bitcoinmirror/phantomcircuit/.


Feel free to download, try it out, and let me know if you find any bugs. Please note, however, that the bigquery database is not yet being updated, nor is it sorted. When it is, any current dumps will probably be incompatible, so I suggest you don't download a full dump yet unless you need/want the data. When automatic bigquery updates do start, you will need to generate a new dump (File->New Dump...).

I haven't gotten around to documenting the usage of this tool yet, but hopefully it's pretty self-explanatory.

---

Windows DLL requirements:

There are a number of DLL files which py2exe seems to require. I think those that are come preloaded with Windows, except for the MSVC runtime. Try running the app first without downloading any new stuff. If it doesn't work, first you should try installing the following:

http://www.microsoft.com/en-us/download/details.aspx?id=29

If you still can't run it, tell me and I'll try to see what the problem is.

It seems that some windows users have been experiencing trouble with the initial authentication, so I've overridden part of the google code to create a GUI method that should work cross-platform. Please let me know if there are still problems authenticating with Google on any platform, or if the instructions aren't clear enough.

In addition, it seems that you do not need to supply your own client_secrets.json. Apparently, it is up to me as the app developer to do this, so there is no longer any need to dig through Google's API Console as I have incorporated the codes into the program, you just need to complete the authorisation in a browser.

Thanks for updating.

Looks great, will test it out tonight!


Yes of course! It would be much better to be part of your tool instead of being a separate script (relying on a separate dev environment). It is a pretty simple script but if you need help porting it just let me know. I would have done it myself if I was able to code something in Python (learning it is on my list).

--

Thanks for your contributions to the community We definitely need more initiatives like this.

Hi everyone,

While I was away SerialVelocity helped to develop the start of a GUI, and I've now filled in some basic update functionality Here are some screenshots of where we're at now:



There is still much to do, however:

• Custom dump location
• Dump statistics
• Graphical error handling
• Exporting the data to different formats
• Choosing what data to export
• Saving export preferences so that exports can be routinely updated
• Testing on multiple platforms
• Packaging the app into a self-contained bundle

As you can see, there's a fair bit to do, but hopefully it shouldn't take too long.

Thanks,
nitrous

EDIT: To use this, run:

Sorry to hijack this thread but:
WHEN ON EARTH WILL YOU FIX THE ANDROID APP?

Thank you, that's great work I'll look into porting it to python and including it in the downloader soon if that's ok with you.

Just an update for anyone interested in this project: I've just finished two weeks of exams, and I'm now going on holiday, but I'll be back by the beginning of July and then I'll be able to put in some more work on this, thank you for your patience.

Sorry for hijacking your topic but people using this tool might find this handy:

Using the database downloaded by the trade data downloader I wrote a tiny candleCalculator which calculates candles based on the trades in this database. I already calculated all hourly candles from Jun 26 2011 19:00:00 up to May 23 2013 16:00:00 (for BTCUSD, 16,699 candles), you can download them here.

If anyone wants to calculate their own candles check out the thread for the script.

(it's a small and easy script but you could always port it to python to make it part of the trade data downloader.)