Downloadable MtGox Historical Candles

TradesLikeAPotato

newbie

Activity: 41

Merit: 0

Quote from: whydifficult on August 21, 2013, 04:10:13 AM

Quote from: TradesLikeAPotato on August 20, 2013, 10:44:14 PM

Well, any ideas whydifficult?

Hmm that's very strange, I did heard some similar things before though.

Unfortunately I am not able to reproduce it, could you please send me the candle file you used + the EMA settings? That way I can debug the behaviour more easy without staring in the dark.

Of course, I should've done it from the start. EMA 5 short, 1 long, .25 threshold, .2 fee.

https://mega.co.nz/#!2Et0GLbT!YATk7xQsBKrVSSZmF9_8DQ0Mrcld9GDlhZMPOEogC9M (URL won't hyperlink for some reason ¯\_(ツ)_/¯)

whydifficult

sr. member

Activity: 287

Merit: 250

Quote from: TradesLikeAPotato on August 20, 2013, 10:44:14 PM

Well, any ideas whydifficult?

Hmm that's very strange, I did heard some similar things before though.

Unfortunately I am not able to reproduce it, could you please send me the candle file you used + the EMA settings? That way I can debug the behaviour more easy without staring in the dark.

TradesLikeAPotato

newbie

Activity: 41

Merit: 0

Well, any ideas whydifficult?

nitrous

sr. member

Activity: 246

Merit: 250

Quote from: TradesLikeAPotato on August 20, 2013, 01:45:15 PM

Quote from: nitrous on August 20, 2013, 01:09:26 PM

Quote from: TradesLikeAPotato on August 20, 2013, 12:15:55 PM

Make sure you select custom time, I have candles from your start date up to yesterday because I copied 5 month intervals into my Excel sheet.

Thanks. Ok I believe I see the reason now - bitcoincharts reuses the closing price of the last datum for the opening price of the next, whereas I only included trades within the range t and t+i (where t is the unix timestamp at the start of the candle and i is the candle interval). Both ways make sense - bitcoincharts gives the theoretical price for a trade if you were to do it at t, whereas whydifficult's and mine give the price for the first trade - however our way conforms to the official use of the term 'opening price' http://www.investopedia.com/terms/o/openingprice.asp. Except for the opening price, most of my data agrees with that of bitcoincharts', except for where the opening price is the same as the high, low or closing, in which case my values differ in about 5% of cases.

Make of this what you will. I don't think it makes too much of a difference anyway - backtesting is an approximation, and during real trading your actual trade prices are likely to be different from those given by the candles anyway. All of these datasets seem to be accurate, albeit inconsistent with each other, but these inconsistencies seem easily explained.

Thanks for taking a look! I've got one more issue. When I test with the BTCe candles, I end up with a profit of 41664823177.120% on ema 1/5 hourly. I've got no idea what's causing that! Anyone have any ideas?

Here is the log: http://pastebin.com/M061Q03M

The perfect trade strategy? xD

I don't know, sorry. I'm sure whydifficult can help you though.

TradesLikeAPotato

newbie

Activity: 41

Merit: 0

Quote from: nitrous on August 20, 2013, 01:09:26 PM

Quote from: TradesLikeAPotato on August 20, 2013, 12:15:55 PM

Make sure you select custom time, I have candles from your start date up to yesterday because I copied 5 month intervals into my Excel sheet.

Thanks. Ok I believe I see the reason now - bitcoincharts reuses the closing price of the last datum for the opening price of the next, whereas I only included trades within the range t and t+i (where t is the unix timestamp at the start of the candle and i is the candle interval). Both ways make sense - bitcoincharts gives the theoretical price for a trade if you were to do it at t, whereas whydifficult's and mine give the price for the first trade - however our way conforms to the official use of the term 'opening price' http://www.investopedia.com/terms/o/openingprice.asp. Except for the opening price, most of my data agrees with that of bitcoincharts', except for where the opening price is the same as the high, low or closing, in which case my values differ in about 5% of cases.

Make of this what you will. I don't think it makes too much of a difference anyway - backtesting is an approximation, and during real trading your actual trade prices are likely to be different from those given by the candles anyway. All of these datasets seem to be accurate, albeit inconsistent with each other, but these inconsistencies seem easily explained.

Thanks for taking a look! I've got one more issue. When I test with the BTCe candles, I end up with a profit of 41664823177.120% on ema 1/5 hourly. I've got no idea what's causing that! Anyone have any ideas?

Here is the log: http://pastebin.com/M061Q03M

nitrous

sr. member

Activity: 246

Merit: 250

Quote from: whydifficult on August 19, 2013, 06:17:59 PM

On a sidenote, I am working on my own platform which will offer candles for a lot of different exchanges / markets in a similar format as the candles above. If you want to get notified you can signup on our website:

wizbit

Nice, wizbit look's great! Will you support different formats, date ranges and intervals? I know a few people using forex tools would like ISO dates (ideally separate date/time fields).

nitrous

sr. member

Activity: 246

Merit: 250

Quote from: TradesLikeAPotato on August 20, 2013, 12:15:55 PM

Make sure you select custom time, I have candles from your start date up to yesterday because I copied 5 month intervals into my Excel sheet.

Thanks. Ok I believe I see the reason now - bitcoincharts reuses the closing price of the last datum for the opening price of the next, whereas I only included trades within the range t and t+i (where t is the unix timestamp at the start of the candle and i is the candle interval). Both ways make sense - bitcoincharts gives the theoretical price for a trade if you were to do it at t, whereas whydifficult's and mine give the price for the first trade - however our way conforms to the official use of the term 'opening price' http://www.investopedia.com/terms/o/openingprice.asp. Except for the opening price, most of my data agrees with that of bitcoincharts', except for where the opening price is the same as the high, low or closing, in which case my values differ in about 5% of cases.

Make of this what you will. I don't think it makes too much of a difference anyway - backtesting is an approximation, and during real trading your actual trade prices are likely to be different from those given by the candles anyway. All of these datasets seem to be accurate, albeit inconsistent with each other, but these inconsistencies seem easily explained.

TradesLikeAPotato

newbie

Activity: 41

Merit: 0

Make sure you select custom time, I have candles from your start date up to yesterday because I copied 5 month intervals into my Excel sheet.

nitrous

sr. member

Activity: 246

Merit: 250

Quote from: TradesLikeAPotato on August 19, 2013, 08:14:13 PM

I'm looking forward to it. In the meantime, the candles I'm downloading seem to have a slight variation from the ones you've calculated, leading to completely different backtesting results. Would you mind taking a look when you get a chance? I'm not sure where the variation stems from.

Ok, I looked at my data too and unfortunately I can't compare the two datasets. For example, my 1800s candles (due to the currently available bq data) only goes up to 23 May 2013, whilst the bitcoincharts data will only start at 22 Jun 2013, so unfortunately I can't really compare the two.

nitrous

sr. member

Activity: 246

Merit: 250

Quote from: TradesLikeAPotato on August 19, 2013, 08:14:13 PM

I'm looking forward to it. In the meantime, the candles I'm downloading seem to have a slight variation from the ones you've calculated, leading to completely different backtesting results. Would you mind taking a look when you get a chance? I'm not sure where the variation stems from.

Hi, I had a look at the difference between whydifficult's daily BTCUSD data and the data from bitcoincharts, and I noticed that whilst bitcoincharts data is aligned to midnight GMT for each datum, whydifficult's data is aligned to 17:00 GMT for each day, so that is likely the source of the differences wrt his data. This difference alignment shouldn't really cause a problem though. As for my data, I'll have a look at that now (remember I used 900s/1800s intervals, not 900m/1800m).

whydifficult

sr. member

Activity: 287

Merit: 250

When there are no trades during a giving candle interval the candles are removed from the files. While instead they should have an OHLC of close of the previous candle, I guess this could be the cause.

EDIT: I will update the candles including empty candles ASAP.

TradesLikeAPotato

newbie

Activity: 41

Merit: 0

I'm looking forward to it. In the meantime, the candles I'm downloading seem to have a slight variation from the ones you've calculated, leading to completely different backtesting results. Would you mind taking a look when you get a chance? I'm not sure where the variation stems from.

whydifficult

sr. member

Activity: 287

Merit: 250

Quote from: TradesLikeAPotato on August 19, 2013, 02:40:35 PM

I just created a new topic with another technique to getting candles:

https://bitcointalk.org/index.php?topic=277704.new#new

That's awesome! I've added this link to the OP.

--

On a sidenote, I am working on my own platform which will offer candles for a lot of different exchanges / markets in a similar format as the candles above. If you want to get notified you can signup on our website:

wizbit

TradesLikeAPotato

newbie

Activity: 41

Merit: 0

I just created a new topic with another technique to getting candles:

https://bitcointalk.org/index.php?topic=277704.new#new

Epoch

legendary

Activity: 922

Merit: 1003

Quote from: nitrous on July 10, 2013, 01:06:12 PM

USD/900 - https://dl.dropboxusercontent.com/u/1760961/Candles/USD-900-Unix-novolume.csv
USD/1800 - https://dl.dropboxusercontent.com/u/1760961/Candles/USD-1800-Unix-novolume.csv

Appreciate the links ... didn't see them the first time I read the post!

nitrous

sr. member

Activity: 246

Merit: 250

Quote from: whydifficult on July 10, 2013, 04:18:08 PM

Quote from: nitrous on July 10, 2013, 01:06:12 PM

The tool does now index the databases, so it didn't take 48 hours, but I guess because I didn't do it on a VPS it took 50 minutes instead of 5 for each.

Thanks! I changed the OP and the topic name to reflect that it hosts a list of downloadable candles (I also included yours, let me know if you mind). I've pointed to your tool for everyone looking for a calculator. I still need some time to check it out by the way.

The 5 minutes was on a VPS with an SSD, so that boost in IO probably did speed it up a lot.

I don't mind

At the moment there are still some bugs with the tool, especially on windows 7, but hopefully I'll finally have a working cross-platform build sometime tomorrow, so there's no pressure to check it out yet Wink

Edit: btw, those are 900s and 1800s, not minutes!

whydifficult

sr. member

Activity: 287

Merit: 250

Quote from: nitrous on July 10, 2013, 01:06:12 PM

The tool does now index the databases, so it didn't take 48 hours, but I guess because I didn't do it on a VPS it took 50 minutes instead of 5 for each.

Thanks! I changed the OP and the topic name to reflect that it hosts a list of downloadable candles (I also included yours, let me know if you mind). I've pointed to your tool for everyone looking for a calculator. I still need some time to check it out by the way.

The 5 minutes was on a VPS with an SSD, so that boost in IO probably did speed it up a lot.

nitrous

sr. member

Activity: 246

Merit: 250

Quote from: Epoch on July 10, 2013, 09:46:57 AM

Nitrous, I did try your GUI tool (nice initiative, BTW), downloaded the MtGox data (638kB .sql file), but the tool crashes for me when I try to export to CSV.

I'll try out your next released version, but in the meantime if you happen to have 1800-second and 900-second CSVs (for gekko) that you could post it would keep me occupied until then. Wink

Oh yes, sorry I forgot you posted in the other thread Tongue

I've released a new version, and I'm about to generate those CSVs for USD. I'll update this post when I've done that.

Edit:

USD/900 - https://dl.dropboxusercontent.com/u/1760961/Candles/USD-900-Unix-novolume.csv
USD/1800 - https://dl.dropboxusercontent.com/u/1760961/Candles/USD-1800-Unix-novolume.csv

The tool does now index the databases, so it didn't take 48 hours, but I guess because I didn't do it on a VPS it took 50 minutes instead of 5 for each.

dirtscience

newbie

Activity: 28

Merit: 0

Thanks for the information.

Epoch

legendary

Activity: 922

Merit: 1003

Nitrous, I did try your GUI tool (nice initiative, BTW), downloaded the MtGox data (638kB .sql file), but the tool crashes for me when I try to export to CSV.

I'll try out your next released version, but in the meantime if you happen to have 1800-second and 900-second CSVs (for gekko) that you could post it would keep me occupied until then. Wink

Topic: Downloadable MtGox Historical Candles (Read 7951 times)