Goomboo's Journal - page 6. | Bitcointalksearch.org

mmccll

newbie

Activity: 7

Merit: 0

For those using php here's a little script I made which generates candle files from bitcoincharts.com (using for example wget http://api.bitcoincharts.com/v1/csv/btceUSD.csv). Alter the $candlesize parameter to set the desired candlesize.
Rename the file you get from bitcoincharts to btc.csv and run the script.

Code:

$handle = fopen('btc.csv',"r");
$candlesize=600;
$i=0;$j=0;
while (($line = fgetcsv($handle,0,",")) != FALSE) {
$timestamp=$line[0];
if(!isset($nextstamp)){$nextstamp=$timestamp+$candlesize;}
$value=round($line[1],4);
if (($timestamp < $nextstamp)){
$candle[$i]=$value;
$i++;$j++;
}
else
{
if(isset($candle)){
$AVG=array_sum($candle)/count($candle);
$OPEN=$candle[0];
$HIGH=max($candle);
$LOW=min($candle);
$CLOSE=$candle[$i-1];
$TRADES=$i;
}
else {
$AVG=0;
}
print $nextstamp . "," . $OPEN . "," . $HIGH . "," . $LOW . "," . $CLOSE . "," . $TRADES ."\n";
$i=0;
$nextstamp=$nextstamp+$candlesize;
if(isset($candle)){unset($candle);}
}
}
fclose($handle);
?>

Output is a csv which can be used for Gekko backtesting.

Queeq

sr. member

Activity: 427

Merit: 250

Quote from: helluvaname on December 21, 2013, 04:54:00 AM

I have run my MT4 script on EMA 21,11 on the same timeframe, here is the report:

...

starting with $100 on november 1st it appears to end up with $440 at 21:00 December 20

Now it's very close to my result. Here's mine for 00:00 01.11.2013 - 00:00 21.12.2013:

Queeq

sr. member

Activity: 427

Merit: 250

Quote from: helluvaname on December 21, 2013, 04:54:00 AM

And, could you please share the way you extract candles from that csv bitcoincharts file, maybe you could post the part of the code responsible for doing that?

I don't really understand what you mean by "extracting candles". I don't use candles, only closing prices.
The algorithm for finding closing price is pretty simple. Firstly, script calculates number of seconds for given interval, 3600 for 1h, for example. Then, when iterating over lines, the following actions are made:
1. Remember current timestamp and price as "last line".
2. Calculate timestamp for start of the next interval.
3. Move on to the next line.
4. If timestamp on this line is greater or equal to calculated "next interval timestamp" - put data from "last line" to database, calculate next interval start and remember current "last line". If current timestamp is smaller - go to step 1 and then to 3.

The code is available at Github. Data manipulation and analysis is made mostly in analysis/analysis.py file.

helluvaname

newbie

Activity: 28

Merit: 0

Quote from: Queeq on December 20, 2013, 08:55:39 AM

So here's the result using corrected algorithm for data from 1st of the November:

I have run my MT4 script on EMA 21,11 on the same timeframe, here is the report:

https://drive.google.com/file/d/0B3f4gdrrQPzZSXF1NFZhM21CVXM/edit?usp=sharing
https://docs.google.com/document/d/1_8Cm3wC-dQsrecPOqlLmQgLlR49hrhp7DUncCYcF_j4/edit?usp=sharing

It is an .htm file with a gif, can be opened with any browser

starting with $100 on november 1st it appears to end up with $440 at 21:00 December 20

I don't really know how to make those heatcharts with MT4, and I am sick and tired of my excel script, it is so ugly and I just want to forget about it Grin

And, could you please share the way you extract candles from that csv bitcoincharts file, maybe you could post the part of the code responsible for doing that?

Queeq

sr. member

Activity: 427

Merit: 250

I think I found an error in my EMA calculation. Different examples use numpy convolution function to do that. However I found that EMA changes more slowly despite that it must be more sensitive. Reversing the weights did the trick. So here's the result using corrected algorithm for data from 1st of the November:

My algorithm tracks each pair of MAs and does buy or sell simulation when they cross. Selection of buying or selling depends on which one goes below (if fast is going below slow - sell, and vice versa). I also subtract 0.2% fee from each operation.
I use an array of closing prices for each time interval (1 hour or other) and another array of corresponding values of MAs for these prices. Prices used for buy/sell simulation is the closing price of such interval.

helluvaname

newbie

Activity: 28

Merit: 0

Quote from: Queeq on December 19, 2013, 04:46:29 AM

If you zoom in you can see MA lengths in braces. I purposely turned off the axis as they don't help much when you look somewhere in the middle of a map.

Oh now I see.

Quote from: Queeq on December 19, 2013, 04:46:29 AM

Bitcoincharts have some gaps in their data, especially in the beginning. They can be easily seen on their graphs. My algorithm just fills these gaps with last known closing price.

So do mine.

Quote from: Queeq on December 19, 2013, 04:46:29 AM

So the results are mostly different. What algorithm have you used?

Well, I made it the hard way Grin

I have loaded all hourly "raw data" from bitcoincharts (there is a button under a candle graph) to excel, then wrote a VBA script imitating buy and sell.
I forgot to tell that I have included a 0,25% fee for each trade.
Now I am moving from excel to MT4 available at btc-e, but it doesn't contain historical data earlier than approx 1st of november 2013 that would be valid for backtesting it with MT4 expert advisor.

My algorythm acts when a new candle is opened, it checks if EMA1-EMA2 was > 0 on the previous candle and EMA1-EMA2 was < 0 on the cande before that or vice versa.

But my excel algorythm and MT4 expert advisor give similar results on timeframe from 1 november 2013 till now.

Right now I am working on feeding MT4 with historical data so I can backtest my expert advisor on all historical data and check if my excel algo is right or wrong.

I have loaded my 1 hour frame from excel to MT4, but I am getting this ugly graphs:

http://s11.postimg.org/ku7mhwuf3/123123_1.jpg

where blue and red lines intersection is a buy/sell point

My guess is that MT4 needs not only 1H scale but all the smaller ones including 1M. Or maybe it needs all ticks, but I can't find aa way to load ticks into it.

I wonder what algorythm do you use to extract candels from that csv data file?

Queeq

sr. member

Activity: 427

Merit: 250

Quote from: helluvaname on December 18, 2013, 04:08:03 PM

Hi Queeq!
Could you please add some axis on your graphs? It is very hard to understand what MA numbers are on your pictures.

By the way, I have done a little research of my own, it is EMA on 1 hour scale from 08.09.2011 till 08.12.13, btc-e data. I wonder, Queeq, if you could run a similar conditions test so we could compare the results?
The numbers are total balance at the final date, starting with 100$

the picture (I have some "disabled for security errors" so cant post a thumbnail):

http://postimg.org/image/5m32aninp/

And, while getting btc-e data from bitcoincharts I have found several "infinite" prices, which vere replaced vith valid prices taken from previous candles.

If you zoom in you can see MA lengths in braces. I purposely turned off the axis as they don't help much when you look somewhere in the middle of a map.

My numbers are relative percent of the starting sum, so they would be pretty much similar, except that if sum goes under 100 they would be negative.

Bitcoincharts have some gaps in their data, especially in the beginning. They can be easily seen on their graphs. My algorithm just fills these gaps with last known closing price.

Here's the result of my backtesting on the given interval (08.09.2011 - 08.12.2013) with BTC-e data as well. I haven't included longer MAs as they're all red and image becomes insanely big.

So the results are mostly different. What algorithm have you used?

helluvaname

newbie

Activity: 28

Merit: 0

Quote from: Queeq on December 18, 2013, 03:18:42 AM

Here are the results from 11th of December till now.

Hi Queeq!
Could you please add some axis on your graphs? It is very hard to understand what MA numbers are on your pictures.

By the way, I have done a little research of my own, it is EMA on 1 hour scale from 08.09.2011 till 08.12.13, btc-e data. I wonder, Queeq, if you could run a similar conditions test so we could compare the results?
The numbers are total balance at the final date, starting with 100$

the picture (I have some "disabled for security errors" so cant post a thumbnail):

http://postimg.org/image/5m32aninp/

And, while getting btc-e data from bitcoincharts I have found several "infinite" prices, which vere replaced vith valid prices taken from previous candles.

Queeq

sr. member

Activity: 427

Merit: 250

According to my calculations, Goomboo's strategy is fairly unprofitable in downtrend. That's not surprising at all as signals are generated with lag, thus buying signals are generated with price higher than selling signals. I wonder of how could we minimize lost in such situations. There may be some additional indicators. Could someone point them out so I could add them to backtesting model?

Here are the results from 11th of December till now.

5 min

15 min

30 min

1 hour

2 hours

solidshotnosh

sr. member

Activity: 307

Merit: 250

Quote from: notme on December 18, 2013, 01:31:29 AM

Daily Goomboos should have sold around $700. Not bad for a few hours, not that Goomboos care for quick profits.

The last sell signal I had was @ 857, what interval are you using?

notme

legendary

Activity: 1904

Merit: 1002

Daily Goomboos should have sold around $700. Not bad for a few hours, not that Goomboos care for quick profits.

Queeq

sr. member

Activity: 427

Merit: 250

Quote from: lx on December 15, 2013, 11:29:15 AM

I just wanted your opinion on the fact that the data we use for backtesting is 'polluted' by events that will most likely not happen again (bitcoin skyrocketing to > $1000 etc).
In the end backtesting is about the relevancy of the data you use to test your strategy, and I just feel like the data is not relevant to determine what's coming.

You're free to backtest on any time periods you want. You may exclude periods of fast growth to have a picture of what will be going on when market is steady. However you may potentially lose profits if price goes up/down again rapidly. So I do not only test on the large periods, but also on smaller ones, especially when price goes down to see how much would I lose in such market conditions and to be prepared for that.

lx

newbie

Activity: 7

Merit: 0

Hi everyone and thanks for this amazing thread.
I just wanted your opinion on the fact that the data we use for backtesting is 'polluted' by events that will most likely not happen again (bitcoin skyrocketing to > $1000 etc).
In the end backtesting is about the relevancy of the data you use to test your strategy, and I just feel like the data is not relevant to determine what's coming.

I've been playing around with Gekko and it seems that EMAs settings have to be adjusted (inverted ?) at some point ; which is not very natural (be gentle, I'm just a geek playing wall street)

mmccll

newbie

Activity: 7

Merit: 0

Checked for 10/21 and 21/10.
Heatmap. Notice the cells with bold lines around them. Showing calculated profit of 11.2% and 8.1%.
http://canalphotos.nl/gekko/upload/Afbeelding3.jpg

And the Gekko output to verify:

Code:

##### SHORT: 10 LONG: 21 #####
2013-12-15 13:52:18 (INFO): ADVICE is to SELL @ 728.000 (-10.036) at 2013-12-07 12:59:57
2013-12-15 13:52:18 (INFO): ADVICE is to BUY @ 820.000 (0.767) at 2013-12-09 00:59:58
2013-12-15 13:52:18 (INFO): ADVICE is to SELL @ 938.000 (-0.661) at 2013-12-11 14:59:59
2013-12-15 13:52:18 (INFO): ADVICE is to BUY @ 950.000 (0.591) at 2013-12-13 05:59:46
2013-12-15 13:52:18 (INFO): ADVICE is to SELL @ 905.000 (-0.287) at 2013-12-13 20:59:54

2013-12-15 13:52:18 (INFO): (PROFIT REPORT) start time: 2013-12-07 11:59:59
2013-12-15 13:52:18 (INFO): (PROFIT REPORT) end time: 2013-12-13 22:55:35
2013-12-15 13:52:18 (INFO): (PROFIT REPORT) timespan: 6 days

2013-12-15 13:52:18 (INFO): (PROFIT REPORT) start price: 1065.1
2013-12-15 13:52:18 (INFO): (PROFIT REPORT) end price: 891
2013-12-15 13:52:18 (INFO): (PROFIT REPORT) Buy and Hold profit: -16.345883%

2013-12-15 13:52:18 (INFO): (PROFIT REPORT) amount of trades: 5
2013-12-15 13:52:18 (INFO): (PROFIT REPORT) original simulated balance: 1000.000000 USD
2013-12-15 13:52:18 (INFO): (PROFIT REPORT) current simulated balance: 1081.025961 USD
2013-12-15 13:52:18 (INFO): (PROFIT REPORT) simulated profit: 81.025961 USD (8.102596%)
2013-12-15 13:52:18 (INFO): (PROFIT REPORT) simulated yearly profit: 4807.540376 USD (480.754038%)

##### SHORT: 21 LONG: 10 #####
2013-12-15 13:54:15 (INFO): ADVICE is to BUY @ 728.000 (10.036) at 2013-12-07 12:59:57
2013-12-15 13:54:15 (INFO): ADVICE is to SELL @ 820.000 (-0.767) at 2013-12-09 00:59:58
2013-12-15 13:54:15 (INFO): ADVICE is to BUY @ 938.000 (0.661) at 2013-12-11 14:59:59
2013-12-15 13:54:15 (INFO): ADVICE is to SELL @ 950.000 (-0.591) at 2013-12-13 05:59:46
2013-12-15 13:54:15 (INFO): ADVICE is to BUY @ 905.000 (0.287) at 2013-12-13 20:59:54

2013-12-15 13:54:15 (INFO): (PROFIT REPORT) start time: 2013-12-07 11:59:59
2013-12-15 13:54:15 (INFO): (PROFIT REPORT) end time: 2013-12-13 22:55:35
2013-12-15 13:54:15 (INFO): (PROFIT REPORT) timespan: 6 days

2013-12-15 13:54:15 (INFO): (PROFIT REPORT) start price: 1065.1
2013-12-15 13:54:15 (INFO): (PROFIT REPORT) end price: 891
2013-12-15 13:54:15 (INFO): (PROFIT REPORT) Buy and Hold profit: -16.345883%

2013-12-15 13:54:15 (INFO): (PROFIT REPORT) amount of trades: 5
2013-12-15 13:54:15 (INFO): (PROFIT REPORT) original simulated balance: 1000.000000 USD
2013-12-15 13:54:15 (INFO): (PROFIT REPORT) current simulated balance: 1111.949512 USD
2013-12-15 13:54:15 (INFO): (PROFIT REPORT) simulated profit: 111.949512 USD (11.194951%)
2013-12-15 13:54:15 (INFO): (PROFIT REPORT) simulated yearly profit: 6642.337697 USD (664.233770%)

And Queeg/Bulbasur you're right! It swaps the BUY/SELL moments as you can see above, effectively trading against the crossovers. Resulting in higher profits in the specific market dynamics of the specified time period. I'll be running some tests on other data sets (varying from/to and interval) to see how this holds against other periods. Might be interesting to see the results...

mmccll

newbie

Activity: 7

Merit: 0

I'm checking my scripts very carefully now, just to make sure that there are no mistakes in it. But basically what it does is looping over the gekko config file altering those parameters:

Code:

// EMA weight (α)
// the higher the weight, the more smooth (and delayed) the line
short: 10,
long: 21,
// amount of candles to remember and base initial EMAs on

xybersurfer

member

Activity: 72

Merit: 10

mmccll, i think you need to check to make sure you didn't accidentally reverse your short and long axii because they seem to be the complete opposite of what one would expect (not just in the normally missing area). mmccll i do enjoy seeing what the heatmap looks like on your more normal axii.

also the csv file from mmccll goes from:
2013-12-06 12:59:28 to 2013-12-13 21:55:35

while the data from Queeq goes from:
2013-12-05 00:00:00 to 2013-12-12 00:00:00

but i doubt this makes that much difference

Bulbasaur

newbie

Activity: 13

Merit: 0

Queeq is right, whether an average is fast or slow depends entirely on the number of periods, so to say that the fast one has a higher number than the slow means that it's no longer really the fast one in practice.

I would be hesitant to trade on any such strategy. You would expect it would probably just do as Queeq suggests, buying and selling on the 'wrong' crossovers, but it might also do entirely unexpected things since there's some chance that the code expects the 'fast' input to always actually be faster than the 'slow' one and hasn't been debugged for the reversed case.

On the other hand, your plot does look basically like a mirror image, so where the normal crossover strategy is unprofitable, the corresponding reversed crossover shows a fairly proportional profit. This does seem to make sense, if we assume the losses are due mainly to 'wrong' guesses rather than commission costs. Switching all the trades would transform all the 'wrong' guesses to 'right' guesses.

I would guess that what's happening is, the indicators are lagging enough that the crossover doesn't hit until the trend is over, at which point you would be taking a position just when the trend is reversing. So if you trade against the crossover, you (maybe) catch the new trend rather than the old one. But now you're getting away from the original idea of using the averages as a signal of recent momentum/current trend and relying more on hoping that the trend reversals will happen to align with your choice of lag time.

Queeq

sr. member

Activity: 427

Merit: 250

Quote from: mmccll on December 14, 2013, 07:56:42 AM

That would be in your graphs any values below the diagonal line (which is in your graph blanked out).
Might be interesting to modify your script to see what is shown in this now white area.

That would mean to make my script act in another way round: buy when fast goes below slow and sell when it goes above. That is completely opposite of what Goomboo taught us

But for the sake of experiment, here's the result

mmccll

newbie

Activity: 7

Merit: 0

That's interesting. I'm using Whydifficults Gekko trading bot which can be found at:

https://bitcointalksearch.org/topic/gekko-a-javascript-trading-bot-and-backtesting-platform-209149

(which is by the way pretty amazing stuff!)

Basically it lets you set the EMA pair and some thresholds. Input is a csv file with timestamp, open, high, low and close values. But i assume it does roughly the same as your exponential algorithm.

While making a php script to loop over various settings of Gekko I noticed that there are some very profitable results when you take the short average with a higher setting compared to the long average. That would be in your graphs any values below the diagonal line (which is in your graph blanked out).
Might be interesting to modify your script to see what is shown in this now white area.

Below is the candle-file which I used to make this graph:

http://www.canalphotos.nl/gekko/upload/candles.csv

Queeq

sr. member

Activity: 427

Merit: 250

Here's my result for almost the same period.

I believe we have different algorithms of backtesting.

In short, mine is the following:
Build an array of closing prices for, say, 1 hour resolution (configurable).
Build a number of simple and exponential moving averages from 1 to 50 (configurable) for these closing prices.
Find all intersections of slow and fast MAs for each unique pair.
Simulate sell or buy at that points (use closing price) depending on which MA goes below which: if fast goes below slow - sell, and vice versa. Substract a fee from each operation. 0.2% by default (configurable).
See what amount of initial sum (in percent) is left.
Show the result for each pair on heatmap.

Topic: Goomboo's Journal - page 6. (Read 281467 times)