Gekko - a javascript trading bot and backtesting platform - page 25.

whydifficult

sr. member

Activity: 287

Merit: 250

Quote from: genydeny on August 02, 2013, 09:47:29 AM

Hi, thank you for your wonderful project!

I have a small question. How often should Gekko to send email ?
I have received only one letter after the start.

Gekko sends an email when a change in the market is detected. With the default setup it uses EMA to detect such a change on an hourly basis. When there is a lot of volatility Gekko will probably report a couple of times a day (max) but more realistically you should expect it once a day.

If something is wrong Gekko currently only logs it (when you enable debug = true, as you did), if you are using forever use `forever list` to find the logfile, you can read it by typing `tail [name of logfile]` (or use `cat` if you want to read the whole thing).

Quote from: juve4v on August 02, 2013, 11:36:47 AM

I'm trying to start it and here's what I get: "Failed to load historical trades from Bitstamp". Any suggestions?

I am currently working on a solution for this as this is a big problem, what you can try for now: lessen the `candles` in the settings, lessen the `interval` in the settings. Note that setting `candles` to a lower number will decrease the trend detection performance in the start (until it build up the history it can't fetch right now). If you are only using paper trading this means it's better to ignore the first couple of days. If you lessen the `interval` Gekko will look at smaller timeframes (so it will detect trends on a smaller timescale so to say).

This problem is not present at Mt. Gox right now btw.

juve4v

hero member

Activity: 505

Merit: 500

I'm trying to start it and here's what I get: "Failed to load historical trades from Bitstamp". Any suggestions?

genydeny

newbie

Activity: 50

Merit: 0

Hi, thank you for your wonderful project!

I have a small question. How often should Gekko to send email ?
I have received only one letter after the start.

email

Code:

Gekko is watching the bitcoin market and has detected a new trend, advice is to SELL
The current BTC price is 95.520

Additional information:

@ 95.520 (-0.445)

config.js

Code:

// Exponential Moving Averages settings:
config.EMA = {
// timeframe per candle
interval: 60, // in minutes
// EMA weight (α)
// the higher the weight, the more smooth (and delayed) the line
short: 10,
long: 21,
// amount of candles to remember and base initial EMAs on
candles: 30,
// the difference between the EMAs (to act as triggers)
sellTreshold: -0.25,
buyTreshold: 0.25
};
....

// Monitor the live market
config.normal = {
enabled: true,
exchange: 'BTCe', // 'MtGox', 'BTCe' or 'Bitstamp'
currency: 'USD',
asset: 'BTC',
tradingEnabled: false,
key: '',
secret: '',
}
...
config.debug = true; // for additional logging / debugging

EDIT:
Gekko is running now through forever on my server with Ubuntu 12.04, node v0.10.5

Code:

data: [1] euny /usr/local/bin/node gekko.js 4628 4630 /home/jeka/.forever/euny.log 0:7:24:8.894

whydifficult

sr. member

Activity: 287

Merit: 250

Quote from: Ronan- on July 30, 2013, 03:53:47 PM

Does this still use BTC-e historical data from bitcoincharts?

My concern is Bitcoincharts downtime/recently they missed a bunch of data from BTC-e

At the moment it does, however I am in the process of storing all trade data on my server, precalculating most standard candles and exposing them through an API. Though this still is a central point of failure.

Sneak peak:

Code:

> db.stats()
{
"db" : "coindata",
"collections" : 144,
"objects" : 151908,
"avgObjSize" : 85.47955341390842,
"dataSize" : 12985028,
"storageSize" : 32378880,
"numExtents" : 329,
"indexes" : 285,
"indexSize" : 12476576,
"fileSize" : 201326592,
"nsSizeMB" : 16,
"dataFileVersion" : {
"major" : 4,
"minor" : 5
},
"ok" : 1
}

(currently watching different 144 markets, got 150k+ trades stored). I suspect it may take a while before this is stable enough to be included in Gekko.

Ronan-

newbie

Activity: 52

Merit: 0

Does this still use BTC-e historical data from bitcoincharts?

My concern is Bitcoincharts downtime/recently they missed a bunch of data from BTC-e

whydifficult

sr. member

Activity: 287

Merit: 250

Quote from: lky_svn on July 30, 2013, 09:47:20 AM

Hi again,

Firstly, thanks for your help on this!
Realise that Gekko is dependent on external sources - don't get me wrong, I wasn't complaining about Gekko at all, just thought I'd help with a bugreport... Fully understand that it might be external sources. But since it works with USD, it is a bit strange, I thought...

As I said, it's working absolutely fine with USD... Only when I change the currency to EUR, then I get this issue.
Tried it with JPY and SEK as well (just changing the conf) to try out other pairs - doesn't seem to be working either, only USD so far for me...

The issue already appears at the start of gekko. Interval is at 30mins at the moment. Running well with USD...
Let me know if I can help somehow to enable/simplify/troubleshoot a fix. I'm on windows, if it helps. Can't find the logs anywhere though...

Hosting standard candle intervals would be interesting. But realise that this might cost you time/money/effort, so thanks for considering it!

Hi,

Sorry if I came across as trying to push the issues to external dependencies, I meant that in most markets besides USD there is little volatility. Currently when Gekko tries to calculate a candle which doesn't have any trade, you'll get NaN for the rest of the time (so also for all other candles that come after it). I'll fix this bug but I don't know how to handle situations where candles don't have any trades at all.

Right now you could set the interval to something higher than 30 minutes to lessen the chance that a candle will have 0 trades, but I'll patch it ASAP so that it will still work.

lky_svn

newbie

Activity: 45

Merit: 0

Quote from: whydifficult on July 30, 2013, 03:34:53 AM

What is the interval you have set? The volatility is very low at the moment, any time Gekko encouters an interval in which there were no trades this happens. So al small interval will trigger this sooner. See below:

Hi again,

Firstly, thanks for your help on this!
Realise that Gekko is dependent on external sources - don't get me wrong, I wasn't complaining about Gekko at all, just thought I'd help with a bugreport... Fully understand that it might be external sources. But since it works with USD, it is a bit strange, I thought...

As I said, it's working absolutely fine with USD... Only when I change the currency to EUR, then I get this issue.
Tried it with JPY and SEK as well (just changing the conf) to try out other pairs - doesn't seem to be working either, only USD so far for me...

The issue already appears at the start of gekko. Interval is at 30mins at the moment. Running well with USD...
Let me know if I can help somehow to enable/simplify/troubleshoot a fix. I'm on windows, if it helps. Can't find the logs anywhere though...

Hosting standard candle intervals would be interesting. But realise that this might cost you time/money/effort, so thanks for considering it!

ErebusBat

hero member

Activity: 560

Merit: 500

I am the one who knocks

Quote from: whydifficult on July 30, 2013, 07:57:27 AM

This will be an optional solution for Gekko, in no way will this be forced upon Gekko's users because nobody can easily check whether the candle data is not messed with.

Thanks for the info whydifficult.... however some of us would still like the option, especially if you decide that it isn't worth hosting one day.

Quote from: whydifficult on July 30, 2013, 07:57:27 AM

Thanks. I am planning on hosting all standard candle intervals (15min, 1h, 4h, 1d, etc.) for all those markets and providing them for free (as long as I can afford it). I could open source it all but storing all the data is not something as small as running a Gekko instance, there is a lot of trade data flowing from all exchanges. Take a look at the mtgox sqlite database for example, it is over 4GB already.

It isn't the standard candles that I am worried about

Nor am I worried about data storage. However my interest really only expands *if* this data could/would be used by the backtesting system. Who knows what odd candle times might give us. But I guess that depends on how your system works... if it does not store the raw trade data then calculating off-standard candles will be difficult.

I see it very much like an electrum server... they are a PIA to setup and take up a lot of space, but some users would like to do that.

That being said.... it is your code and I respect your decision to do with it as you see fit, you have treated us great so far. At the very lease would you consider releasing a bare bones reference implementation so others could do their own implementation?

whydifficult

sr. member

Activity: 287

Merit: 250

Quote from: ErebusBat on July 30, 2013, 07:29:37 AM

This is very interesting indeed... Will the server be open source too so we can run our own instances? I think this is especially important for people who want to use non-standard candles. Also will it be able to be used for the backtesting too?

Thanks. I am planning on hosting all standard candle intervals (15min, 1h, 4h, 1d, etc.) for all those markets and providing them for free (as long as I can afford it). I could open source it all but storing all the data is not something as small as running a Gekko instance, there is a lot of trade data flowing from all exchanges. Take a look at the mtgox sqlite database for example, it is over 4GB already.

I am now thinking of maybe turning this into a service where all standard candles for all those markets can be retrieved by everyone (free). And maybe I can see what other stuff can be calculated easily. I am thinking about things like spread graphs between different BTC/LTC (or other pure crypto) markets for arbitrage or maybe realtime datastreams to popular charting software like sierrachart. Right now I am just exploring all the different things I can do with this stuff.

--

This will be an optional solution for Gekko, in no way will this be forced upon Gekko's users because nobody can easily check whether the candle data is not messed with.

ErebusBat

hero member

Activity: 560

Merit: 500

I am the one who knocks

whydifficult

sr. member

Activity: 287

Merit: 250

Quote from: lky_svn on July 30, 2013, 03:05:02 AM

Hi,

Using MtGox, I get NaN errors when using EUR...
USD seems to be working fine.

Enabled verbose to check what's going on. With USD I get:

Code:

2013-07-30 09:59:29 (DEBUG): calced EMA properties for new candle:
2013-07-30 09:59:29 (DEBUG): short ema 101.048
2013-07-30 09:59:29 (DEBUG): long ema 101.037
2013-07-30 09:59:29 (DEBUG): diff ema 0.010
2013-07-30 09:59:29 (DEBUG): fetching new trades for new bucket at Mt. Gox
2013-07-30 09:59:31 (DEBUG): calculated candle: 0
2013-07-30 09:59:31 (DEBUG): calced EMA properties for new candle:
2013-07-30 09:59:31 (DEBUG): short ema 101.038
2013-07-30 09:59:31 (DEBUG): long ema 101.033
2013-07-30 09:59:31 (DEBUG): diff ema 0.005
2013-07-30 09:59:31 (DEBUG): we are currently not in an up or down trend @ 100.981 (0.005)

When changing to EUR, I get something like:

Code:

2013-07-30 10:04:03 (DEBUG): calculated candle: 1
2013-07-30 10:04:03 (DEBUG): calced EMA properties for new candle:
2013-07-30 10:04:03 (DEBUG): short ema NaN
2013-07-30 10:04:03 (DEBUG): long ema NaN
2013-07-30 10:04:03 (DEBUG): diff ema NaN
2013-07-30 10:04:03 (DEBUG): fetching new trades for new bucket at Mt. Gox
2013-07-30 10:04:04 (DEBUG): calculated candle: 0
2013-07-30 10:04:04 (DEBUG): calced EMA properties for new candle:
2013-07-30 10:04:04 (DEBUG): short ema NaN
2013-07-30 10:04:04 (DEBUG): long ema NaN
2013-07-30 10:04:04 (DEBUG): diff ema NaN
2013-07-30 10:04:04 (DEBUG): we are currently not in an up or down trend @ 75.500 (NaN)

This is the config I changed:

Code:

config.normal = {
  enabled: true,
  exchange: 'MtGox', // 'MtGox', 'BTCe' or 'Bitstamp'
  currency: 'EUR',
  asset: 'BTC',
  tradingEnabled: false,

Simply changed the currency in there - and then the error appears.

Anybody had this issue?

What is the interval you have set? The volatility is very low at the moment, any time Gekko encouters an interval in which there were no trades this happens. So al small interval will trigger this sooner. See below:

Quote from: ErebusBat on July 29, 2013, 06:34:16 AM

Quote from: ewibit on July 29, 2013, 05:24:21 AM

Code:

realtime-candle-fetcher.js:98
throw 'Failed to load historical trades from ' + this.watcher.name;
^
Failed to load historical trades from bitcoincharts

I have been getting this all weekend for Bitstamp too.

Gekko is currently not dependent on external factors (besides the exchange APIs and bitcoincharts for BTC-e), this is great and I want to keep it like this for as long as possible. But with the recent problems I am facing I am going to calculate candles on my server expose those to Gekko/the world (it stays optional).

This solves a lot of small issues:

No 'failed to load data from exchange X'
No sudden API changes exposed to Gekko
Able to keep track of a lot of exchanges, even those without a historical data API. This finally means altcoins as well!
Way more efficient: This way the candles are only calculated once and everyone uses that directly

Problems:

If for whatever reason I can't fetch the candles no one using Gekko is able to (paper) trade
Central point of failure
I have to pay for the server / bandwidth, if I can't it will be shutdown
It's going to be hard to offer different non-standard intervals

I set up a script that is watching a bunch of exchanges with different currency pairs at the moment.Those can all be exposed to Gekko (assuming the exchanges have trading APIs) I am currently watching:

Code:

> show collections
bit2c_BTC/NIS
bitstamp_BTC/USD
btcchina_BTC/CNY
btce_BTC/EUR
btce_BTC/RUR
btce_BTC/USD
btce_EUR/USD
btce_FTC/BTC
btce_LTC/BTC
btce_LTC/RUR
btce_NMC/BTC
btce_NVC/BTC
btce_PPC/BTC
btce_TRC/BTC
btce_USD/RUR
bter_BQC/BTC
bter_BQC/CNY
bter_BQC/LTC
bter_BTB/BTC
bter_BTC/CNY
bter_CNC/BTC
bter_CNC/CNY
bter_CNC/LTC
bter_FRC/BTC
bter_FRC/CNY
bter_FRC/LTC
bter_FTC/BTC
bter_FTC/CNY
bter_FTC/LTC
bter_LTC/BTC
bter_LTC/CNY
bter_MYMINER/BTC
bter_NMC/BTC
bter_NME/LTC
bter_PPC/BTC
bter_PPC/CNY
bter_PPC/LTC
bter_TRC/BTC
bter_TRC/CNY
bter_TRC/LTC
bter_WDC/BTC
bter_WDC/CNY
bter_WDC/LTC
bter_YAC/BTC
bter_YAC/CNY
cryptsy_ALF/BTC
cryptsy_AMC/BTC
cryptsy_ANC/BTC
cryptsy_ARG/BTC
cryptsy_BQC/BTC
cryptsy_BTB/BTC
cryptsy_BTE/BTC
cryptsy_BTG/BTC
cryptsy_CAP/BTC
cryptsy_CGB/BTC
cryptsy_CNC/BTC
cryptsy_CRC/BTC
cryptsy_CSC/BTC
cryptsy_DBL/LTC
cryptsy_DGC/BTC
cryptsy_DMD/BTC
cryptsy_DVC/LTC
cryptsy_ELC/BTC
cryptsy_EMD/BTC
cryptsy_EZC/LTC
cryptsy_FLO/LTC
cryptsy_FRC/BTC
cryptsy_FRK/BTC
cryptsy_FST/BTC
cryptsy_FTC/BTC
cryptsy_GLD/LTC
cryptsy_HYC/BTC
cryptsy_IFC/LTC
cryptsy_IXC/BTC
cryptsy_JKC/LTC
cryptsy_KGC/BTC
cryptsy_LKY/BTC
cryptsy_LTC/BTC
cryptsy_MEC/BTC
cryptsy_MEM/LTC
cryptsy_MNC/BTC
cryptsy_MST/LTC
cryptsy_NAN/BTC
cryptsy_NBL/BTC
cryptsy_NMC/BTC
cryptsy_NRB/BTC
cryptsy_NVC/BTC
cryptsy_PPC/BTC
cryptsy_PXC/BTC
cryptsy_QRK/BTC
cryptsy_RYC/LTC
cryptsy_SBC/BTC
cryptsy_TRC/BTC
cryptsy_WDC/BTC
cryptsy_XNC/LTC
cryptsy_XPM/BTC
cryptsy_YAC/BTC
fxbtc_BTC/CNY
fxbtc_LTC/BTC
fxbtc_LTC/CNY
mtgox_BTC/AUD
mtgox_BTC/BTC
mtgox_BTC/CAD
mtgox_BTC/CHF
mtgox_BTC/CNY
mtgox_BTC/CZK
mtgox_BTC/DKK
mtgox_BTC/EUR
mtgox_BTC/GBP
mtgox_BTC/HKD
mtgox_BTC/JPY
mtgox_BTC/NOK
mtgox_BTC/NZD
mtgox_BTC/PLN
mtgox_BTC/RUB
mtgox_BTC/SEK
mtgox_BTC/SGD
mtgox_BTC/THB
mtgox_BTC/USD
mtgox_USD/BTC
vircurex_BTC/DGC
vircurex_BTC/DVC
vircurex_BTC/FRC
vircurex_BTC/FTC
vircurex_BTC/IXC
vircurex_BTC/LTC
vircurex_BTC/NMC
vircurex_BTC/NVC
vircurex_BTC/PPC
vircurex_BTC/RUR
vircurex_BTC/TRC
vircurex_BTC/USD
vircurex_BTC/XPM
virwox_BTC/SLL

It's been running for a day or something and I currently have 120k trades stored (this does not include historical data).

lky_svn

newbie

Activity: 45

Merit: 0

Hi,

Using MtGox, I get NaN errors when using EUR...
USD seems to be working fine.

Enabled verbose to check what's going on. With USD I get:

Code:

2013-07-30 09:59:29 (DEBUG): calced EMA properties for new candle:
2013-07-30 09:59:29 (DEBUG): short ema 101.048
2013-07-30 09:59:29 (DEBUG): long ema 101.037
2013-07-30 09:59:29 (DEBUG): diff ema 0.010
2013-07-30 09:59:29 (DEBUG): fetching new trades for new bucket at Mt. Gox
2013-07-30 09:59:31 (DEBUG): calculated candle: 0
2013-07-30 09:59:31 (DEBUG): calced EMA properties for new candle:
2013-07-30 09:59:31 (DEBUG): short ema 101.038
2013-07-30 09:59:31 (DEBUG): long ema 101.033
2013-07-30 09:59:31 (DEBUG): diff ema 0.005
2013-07-30 09:59:31 (DEBUG): we are currently not in an up or down trend @ 100.981 (0.005)

When changing to EUR, I get something like:

Code:

2013-07-30 10:04:03 (DEBUG): calculated candle: 1
2013-07-30 10:04:03 (DEBUG): calced EMA properties for new candle:
2013-07-30 10:04:03 (DEBUG): short ema NaN
2013-07-30 10:04:03 (DEBUG): long ema NaN
2013-07-30 10:04:03 (DEBUG): diff ema NaN
2013-07-30 10:04:03 (DEBUG): fetching new trades for new bucket at Mt. Gox
2013-07-30 10:04:04 (DEBUG): calculated candle: 0
2013-07-30 10:04:04 (DEBUG): calced EMA properties for new candle:
2013-07-30 10:04:04 (DEBUG): short ema NaN
2013-07-30 10:04:04 (DEBUG): long ema NaN
2013-07-30 10:04:04 (DEBUG): diff ema NaN
2013-07-30 10:04:04 (DEBUG): we are currently not in an up or down trend @ 75.500 (NaN)

This is the config I changed:

Code:

config.normal = {
enabled: true,
exchange: 'MtGox', // 'MtGox', 'BTCe' or 'Bitstamp'
currency: 'EUR',
asset: 'BTC',
tradingEnabled: false,

Simply changed the currency in there - and then the error appears.

Anybody had this issue?

San1ty

sr. member

Activity: 462

Merit: 250

Quote from: ErebusBat on July 29, 2013, 06:34:16 AM

Quote from: ewibit on July 29, 2013, 05:24:21 AM

Code:

realtime-candle-fetcher.js:98
throw 'Failed to load historical trades from ' + this.watcher.name;
^
Failed to load historical trades from bitcoincharts

I have been getting this all weekend for Bitstamp too.

That's normal, Bitstamp fucked up their API without any notice.

ErebusBat

hero member

Activity: 560

Merit: 500

I am the one who knocks

Quote from: ewibit on July 29, 2013, 05:24:21 AM

Code:

realtime-candle-fetcher.js:98
throw 'Failed to load historical trades from ' + this.watcher.name;
^
Failed to load historical trades from bitcoincharts

I have been getting this all weekend for Bitstamp too.

ewibit

legendary

Activity: 2955

Merit: 1049

Code:

realtime-candle-fetcher.js:98
throw 'Failed to load historical trades from ' + this.watcher.name;
^
Failed to load historical trades from bitcoincharts

ErebusBat

hero member

Activity: 560

Merit: 500

I am the one who knocks

Quote from: Diabolicus on July 24, 2013, 01:53:48 AM

May I ask how far are you with the ta-lib implementation?
Also, could you make it somewhat similar to the code that cryptotrader.org accepts?
It would be so much easier if one could just simply copy&paste coffescript snippets back and forth.

FYI... Apparently you can use CoffeeScript pretty easy in node: http://stackoverflow.com/questions/4679782/can-i-use-coffeescript-instead-of-js-for-node-js#4683431

conquerorofbits

newbie

Activity: 4

Merit: 0

Gekko doesn't show Profit Reports, so I think there is a bug in logger.js:

logger.js, Line 37 ...

Code:

// log advice
Logger.prototype.inform = function(what, price, meta) {
  if(!this.verbose && what !== 'SELL' && !this.config.backtest)
   return;

  if(!this.verbose && what === 'HOLD' && this.config.backtest)
   return;

should be:

Code:

// log advice
Logger.prototype.inform = function(what, price, meta) {
  if(!this.verbose && what !== 'SELL' && !this.config.backtest.enable)
   return;

  if(!this.verbose && what === 'HOLD' && this.config.backtest.enabled)
   return;

Same thing at Line 94 ...

whydifficult

sr. member

Activity: 287

Merit: 250

Quote from: Diabolicus on July 24, 2013, 01:53:48 AM

Quote from: whydifficult on June 17, 2013, 06:15:38 PM

This is a big step towards implementing a system that supports over a 100 different indicators (ta-lib) because it needs to get candles fed.

May I ask how far are you with the ta-lib implementation?
Also, could you make it somewhat similar to the code that cryptotrader.org accepts?
It would be so much easier if one could just simply copy&paste coffescript snippets back and forth.

I was pretty far. But because of the API changes in both bitcoincharts (BTC-e data) and bitstamp I probably have to change core. I want to have the core stabilised before I'm adding all the advanced stuff.

I was planning to do something similar for a while, I am not a fan of coffeescript myself. I think plain javascript is a lot easier for most people, as it's very similar to a lot of big languages like Java and C. But adding coffeescript ofcourse is a piece of cake since Gekko is a js project, so this will make it in.

I don't think I can make it exactly the same (with all function calls, etc). But I'll see what I can do

Diabolicus

member

Activity: 90

Merit: 10

Quote from: whydifficult on June 17, 2013, 06:15:38 PM

This is a big step towards implementing a system that supports over a 100 different indicators (ta-lib) because it needs to get candles fed.

May I ask how far are you with the ta-lib implementation?
Also, could you make it somewhat similar to the code that cryptotrader.org accepts?
It would be so much easier if one could just simply copy&paste coffescript snippets back and forth.

whydifficult

sr. member

Activity: 287

Merit: 250

Quote from: jago25_98 on July 20, 2013, 06:31:57 AM

Anyone had any wins with EMA?

Well I think a trading bot following a strategy is way better than its human counter part. But finding a good strategy is pretty hard. I would suggest using the backtesting to verify that your current settings would have made profit in the past.

You can also use Gekko to do paper/simulation trading (with a bunch of settings simultaneously for example), this way you know whether a strategy is currently effective. Do note that you are always looking a the success after a while, in which case the success lies in the history and won't guarantee anything anymore for the future.

EMA is a indicator that you can use to ride the trend (buy low - sell high), however because it uses moving averages to verify the trend it is always lagging behind. This in combination that trends can reverse at any time makes it so that you don't always make profit while trying to ride the trend.

To answer your question: I am currently on a 10% profit measured in BTC in a little over one month.

Topic: Gekko - a javascript trading bot and backtesting platform - page 25. (Read 147899 times)