Downloadable MtGox Historical Candles - page 2.

nitrous

sr. member

Activity: 246

Merit: 250

Nice work

Just a heads up - your JPY data is off by a factor of 100. See https://en.bitcoin.it/wiki/MtGox/API#Number_Formats for the constants you need to divide each currency by.

Quote from: Epoch on July 08, 2013, 02:31:39 PM

Anyone taken the time to calculate 30-min and/or 15-min candle .CSV's (in the same format as whydifficult provided in his OP) for this dataset and care to share with the group? Thanks in advance.

Epoch, my GUI tool (https://bitcointalksearch.org/topic/m.2669142) has a port of WhyDifficult's script and you can then calculate any candles you want (export, set currency, set formatter to CSV Candle [Unix], set candle duration in seconds, don't include volume). If there are a few specific ones you're interested in though I could maybe do them for you

Quote from: medji on July 04, 2013, 03:17:02 PM

I think that if during a time period there are no trades then the open,close,high,low is equal to the previous period close price.
This chart was during the time mtgox was down for a few days, and its just as I described.
http://bitcoincharts.com/charts/mtgoxUSD#rg360zigDailyzczsg2011-06-10zeg2011-07-05ztgSzm1g10zm2g25zv

In my same tool, you can turn Don't include missing candle data off and get the data the way you want.

Epoch

legendary

Activity: 922

Merit: 1003

Anyone taken the time to calculate 30-min and/or 15-min candle .CSV's (in the same format as whydifficult provided in his OP) for this dataset and care to share with the group? Thanks in advance.

medji

newbie

Activity: 31

Merit: 0

I think that if during a time period there are no trades then the open,close,high,low is equal to the previous period close price.
This chart was during the time mtgox was down for a few days, and its just as I described.
http://bitcoincharts.com/charts/mtgoxUSD#rg360zigDailyzczsg2011-06-10zeg2011-07-05ztgSzm1g10zm2g25zv

whydifficult

sr. member

Activity: 287

Merit: 250

Quote from: daybyter on July 01, 2013, 06:30:31 AM

I wondered if this script could be used to visualize trades in a phonegap app?

The script outputs plain numbers to a text file. I guess this data could be used to serve as input for some graphing library. I would suggest to rewrite it to output JSON instead of csv though, or you could convert it to something else after running it.

I'm going to update the script soon to also include volume (and thus store as OHLCV candles).

daybyter

legendary

Activity: 965

Merit: 1000

I wondered if this script could be used to visualize trades in a phonegap app?

whydifficult

sr. member

Activity: 287

Merit: 250

Quote from: pulsecat on June 23, 2013, 03:51:06 AM

I'm wondering what is common practice for handling missing candles? For sure, the candlestick generator can simply ignore them, but wouldn't it cause errors when data is fed to technical analysis/charts algorithms?

Gekko uses these candles when you want to backtest. this is how I solved it for now:

Quote

When there are missing candles Gekko will act as if the whole duration of the missing candle never happened.

whydifficult

sr. member

Activity: 287

Merit: 250

Quote from: pulsecat on June 23, 2013, 03:51:06 AM

I'm wondering what is common practice for handling missing candles? For sure, the candlestick generator can simply ignore them, but wouldn't it cause errors when data is fed to technical analysis/charts algorithms?

That's a good question. I am building a list of candle files so they can be used to backtest with Gekko, not sure how Gekko will solve this yet. Also not sure how to do TA on candles with gaps. Luckely the USD market has a lot of liquidity, so I guess this won't make much of a difference.

I don't know how charting software normally deals with it but I now emitted the lines from the files. If someone runs into trouble running this and needs special syntax for gaps just let me know and I will re run the script.

pulsecat

full member

Activity: 220

Merit: 100

I'm wondering what is common practice for handling missing candles? For sure, the candlestick generator can simply ignore them, but wouldn't it cause errors when data is fed to technical analysis/charts algorithms?

RagnarDanneskjold

full member

Activity: 144

Merit: 100

Very cool. I will use this.

whydifficult

sr. member

Activity: 287

Merit: 250

I've added download links to all USD hourly candles (16k+) and all daily candles for USD and EUR (600+).

If you need the CSV files structured differently let me know.

whydifficult

sr. member

Activity: 287

Merit: 250

I am working on backtesting for my trade bot Gekko and ~~I created a simple script to calcuate candles on MtGox data. This script will output a csv file with all candles.~~ Nitrous has created a great tool which can download all trade data and calculate candles based on this trade data. Check it out in the Mt. Gox data downloader thread.

Downloadable candles

I've ran the script myself for a couple of candle durations on a couple of markets, here are the download links:

Hourly

Hourly BTC/USD candles from Jun 26 2011 19:00:00 to May 23 2013 16:00:00. The CSV file has 16699 candles (note that 27 candles/hours are missing, during these hours I could not find any trades).
Hourly BTC/EUR candles from Aug 27 2011 10:00:00 to May 23 2013 16:00:00. The CSV file has 12613 candles (note that a lot of candles/hours are missing, during these hours I could not find any trades).
Hourly BTC/GBP candles from Sep 06 2011 08:00:00 to May 23 2013 16:00:00. The CSV file has 11423 candles (note that 3586 of candles/hours are missing, during these hours I could not find any trades).
Hourly BTC/AUD candles from Sep 02 2011 12:00:00 to May 23 2013 16:00:00. The CSV file has 5830 candles (note that 9271 of candles/hours are missing, during these hours I could not find any trades).
Hourly BTC/JPY candles from Aug 30 2011 11:00:00 to May 23 2013 15:00:00. The CSV file has 3101 candles (note that 15173 of candles/hours are missing, during these hours I could not find any trades).

4 Hourly

4 Hourly BTC/USD candles from Jun 26 2011 17:00:00 to May 23 2013 16:00:00. The CSV file has 4182 candles (note that 1 candle/hour is missing, during these 4 hours I could not find any trades).
4 Hourly BTC/EUR candles from Aug 27 2011 09:00:00 to May 23 2013 13:00:00. The CSV file has 3683 candles (note that 129 of candles/hours are missing, during these hours I could not find any trades).

Daily

Daily BTC/USD candles from Jun 26 2011 to May 23 2013. The CSV file has 697 candles.
Daily BTC/EUR candles from Aug 27 2011 to May 23 2013. The CSV file has 630 candles (note that 6 candles/days are missing).
Daily BTC/GBP candles from Sep 06 2011 to May 23 2013. The CSV file has 622 candles (note that 2 candles/days are missing).

Other timeframes (provided by Nitrous)

Calculate your own candles

EDIT2:

If you have a copy of Microsoft excel you can use the data from Bitcoincharts to create your own candles for a number of exchanges and makets. Read the whole method here. (big thanks to TradesLikeAPotato)

EDIT:

Nitrous has created a great tool which can download all trade data and calculate candles based on this trade data. Check it out in the Mt. Gox data downloader thread. If you want to use my older script here is the howto:

You can also use the script to calculate your own candles (for example: weekly, daily, 4hourly, 15min, 5 min, 1min, 514 seconds, etc. candles):

The simple script requires a local copy of a database with MtGox trades. To download this database you should use Nitrous' excellent trade data downloader.

Once you have this database:

Install nodejs
Install npm (you can skip this on Windows and OS X if you used the installers)
Download the script and put it somewhere close to database dump (from the trade data downloader).
Go to the directory in command line or terminal.
Open candleCalculator.js in a texteditor and edit the candle vars to your needs (adjust the startTime, endTime and candleDuration)* also check to see if `mtgoxDump` points to the correct file (it is as long as it's called dump.sql and is in the same directory)
type in: npm install moment underscore sqlite-wrapper
type in: node candleCalculator.js
watch it go! Once every 10 candles it will write them to the csv file (candles-[script start timestamp].csv).

Notes:

It is advised to index the date column (Money_Trade__) of the database before you start calculating candles. This will extremely speed up the time it takes to calculate the candles: without it takes 40+ hours on my VPS (with SSD) and with it it takes less than 5 minutes to calculate all hourly USD candles.
As of now the script will report the following OHCL for a candle that did not have any trades: open: NaN, high: -Infinity, low: Infinity, close: NaN. If you don't want them strip them out after run or add a simple check at line 48.

*If you're not sure how to edit the dates and durations please see the docs of momentjs.

Topic: Downloadable MtGox Historical Candles - page 2. (Read 7951 times)