ncurses based MtGox live monitor and trading-bot-framework - page 5.

prof7bit

hero member

Activity: 938

Merit: 500

https://youengine.io/

goxtool.py can now display a depth chart. Also it has grouping in the orderbook and optionally adding up the totals:

git pull

there are a lot of new keyboad shortcuts to control all this,
I have added them to the manual: http://prof7bit.github.io/goxtool/

Screenshot:

Ban Curtain

member

Activity: 63

Merit: 10

Quote from: prof7bit on April 27, 2013, 09:03:50 AM

Quote from: Ban Curtain on April 27, 2013, 08:46:14 AM

Is it possible to get access to another strategies' instances?

Not in the current implementation, not when passing a list of strategies to the command line. But what you could do is to explicitly load a strategy from within another strategy. This "master strategy" could then load "slave strategies" in an exactly defined order and pass them references to itself or to each other in their constructors so they can refer to each other and then you could have your strategies communicate with custom signals or directly calling each other.

Edit: Another (maybe even better) solution would be that you pass the strategies to the command line in the right order so that the first strategy would be the one that wants to emit signals would simply create a new signal in the gox instance and all the other strategies that load after it would check if a member of that name exists in the gox instance and if it exists connect a slot to it.

Emitting Strategy:
In the constructor __init__ of the emitting strategy (that one should be loaded first) do the following to create the new the "signal_foo":

Code:

gox.signal_foo = goxapi.Signal()

and to emit the signal you do something like this:

Code:

self.gox.signal_foo(self, (42, 23))

Receiving Strategy:
in the constructor __init__ of the receiving strategies (these must appear *after* the above one on the command line) you add the following to detect the existence of the signal and connect a slot to it to it:

Code:

if hasattr(gox, 'signal_foo'):
gox.signal_foo.connect(self.slot_foo)

and a slot:

Code:

def slot_foo(self, sender, data):
   """slot for custom foo signal"""
   (a, b) = data
   self.debug("foo signal detected: %g %g" % (a, b))

That should do the trick, I just tested it and it works very well. If the emitter was not loaded then the slot will just never be called but it won't crash (you can add an else to the if to print a debug warning if that happens)

Thank you, I gonna try the solution you recommend.

prof7bit

hero member

Activity: 938

Merit: 500

https://youengine.io/

Quote from: Ban Curtain on April 27, 2013, 08:46:14 AM

Is it possible to get access to another strategies' instances?

Not in the current implementation, not when passing a list of strategies to the command line. But what you could do is to explicitly load a strategy from within another strategy. This "master strategy" could then load "slave strategies" in an exactly defined order and pass them references to itself or to each other in their constructors so they can refer to each other and then you could have your strategies communicate with custom signals or directly calling each other.

Edit: Another (maybe even better) solution would be that you pass the strategies to the command line in the right order so that the first strategy would be the one that wants to emit signals would simply create a new signal in the gox instance and all the other strategies that load after it would check if a member of that name exists in the gox instance and if it exists connect a slot to it.

Emitting Strategy:
In the constructor __init__ of the emitting strategy (that one should be loaded first) do the following to create the new the "signal_foo":

Code:

gox.signal_foo = goxapi.Signal()

and to emit the signal you do something like this:

Code:

self.gox.signal_foo(self, (42, 23))

Receiving Strategy:
in the constructor __init__ of the receiving strategies (these must appear *after* the above one on the command line) you add the following to detect the existence of the signal and connect a slot to it to it:

Code:

if hasattr(gox, 'signal_foo'):
gox.signal_foo.connect(self.slot_foo)

and a slot:

Code:

def slot_foo(self, sender, data):
   """slot for custom foo signal"""
   (a, b) = data
   self.debug("foo signal detected: %g %g" % (a, b))

That should do the trick, I just tested it and it works very well. If the emitter was not loaded then the slot will just never be called but it won't crash (you can add an else to the if to print a debug warning if that happens)

Ban Curtain

member

Activity: 63

Merit: 10

Id like to create hierarchy of strategies. The first strategy observes for a complex event and then emits signal which should be handled by slot of the second strategy. How is it better to implement?

As far as I understand to connect to a signal of a dummy emitter I must have access to instance of emitter, but while initializing strategies I have only instance of gox. Is it possible to get access to another strategies' instances?

c0inbuster

member

Activity: 105

Merit: 10

Maybe price should be stored in Decimal http://docs.python.org/2/library/decimal.html

prof7bit

hero member

Activity: 938

Merit: 500

https://youengine.io/

Quote from: c0inbuster on April 26, 2013, 12:48:30 PM

I'm not very interested by you "speaker" strategy

maybe others are - for various other reasons.

Quote from: c0inbuster on April 26, 2013, 12:48:30 PM

I was very surprised to see that candles are stored in a list...
why not using a Numpy Array ?

The entire history was initially only a quick hack to be able to plot the recent 24 hours prices into the chart view. Ideally i would eventually implement somethiing more sophisticated that would provide more history, more different timeframes and then maybe also with numpy arrays. I'm waiting until magicaltux implements the API to query trade history of several different (most commonly used) timeframes in OHLCV format. Until then it remains only a quick and dirty hack to be able to plot the price chart.

Until then your bot needs to construct its Numpy arrays itself (maybe even from a local database or some other web service) and keep it updated with the trade signals. You could write a class that does all this, connects to needed gox signals and has a well architected and intuitive interface for usage by bots and also integrates TA-lib and then instantiate this entire thing from your bot during __init__. Once it is working I could look at it for some inspiration of how to design my own new history class once mtgox has the history api ready.

Quote from: c0inbuster on April 26, 2013, 12:48:30 PM

I was also surprised to see output for price such as

Code:

2013-04-26 18:02:23,152:DEBUG:test_strategy.Strategy:[ 13399024. 13590000. 13680000. 13610000. 13649999. 13680000.

In my mind (but I ever said something similar) in strategy" I should have "real" values.

The format is the one used by mtgox api, all currency values are integers. BTC is an integer of satoshi (1e8), JPY is 1e3, USD and all others are 1e5.

c0inbuster

member

Activity: 105

Merit: 10

I'm not very interested by you "speaker" strategy ... for now... but maybe one day it could be useful...

I tried to play with a Python wrapper for TA-Lib
http://ta-lib.org/
http://mrjbq7.github.io/ta-lib/
this wrapper is total...
you just need to install cython, TA-Lib, and this wrapper

I just did a test strategy for now

Code:

"""
trading robot breadboard
"""

import goxapi
import datetime
import numpy as np
import talib
#import pandas as pd

class Strategy(goxapi.BaseObject):
# pylint: disable=C0111,W0613,R0201

def __init__(self, gox):
goxapi.BaseObject.__init__(self)
self.signal_debug.connect(gox.signal_debug)
gox.signal_keypress.connect(self.slot_keypress)
gox.signal_strategy_unload.connect(self.slot_before_unload)
gox.signal_ticker.connect(self.slot_tick)
gox.signal_depth.connect(self.slot_depth)
gox.signal_trade.connect(self.slot_trade)
gox.signal_userorder.connect(self.slot_userorder)
gox.orderbook.signal_owns_changed.connect(self.slot_owns_changed)
gox.signal_wallet.connect(self.slot_wallet_changed)

gox.signal_orderlag.connect(self.slot_orderlag)

self.gox = gox
self.name = "%s.%s" % \
(self.__class__.__module__, self.__class__.__name__)
self.debug("%s loaded" % self.name)

self.is_lagging = True

def __del__(self):
self.debug("%s unloaded" % self.name)

def slot_before_unload(self, _sender, _data):
self.debug("%s before unload" % self.name)

def slot_keypress(self, gox, (key)):
if key == ord("t"):
#self.debug("Test")
len_hist = gox.history.length()
self.debug("Test history with {N} candles".format(N=len_hist))
#z = np.zeros(len)
#self.df = pd.DataFrame(z)

#opn = np.array(gox.history.candles)
#iterable = (gox.history.candles[i].tim for i in range(len_hist))
#a_tim = np.fromiter(iterable, ???)

rng = range(len_hist)
iterable = (gox.history.candles[i].opn for i in rng)
a_opn = np.fromiter(iterable, np.float)

iterable = (gox.history.candles[i].hig for i in rng)
a_hig = np.fromiter(iterable, np.float)

iterable = (gox.history.candles[i].low for i in rng)
a_low = np.fromiter(iterable, np.float)

iterable = (gox.history.candles[i].cls for i in rng)
a_cls = np.fromiter(iterable, np.float)

iterable = (gox.history.candles[i].vol for i in rng)
a_vol = np.fromiter(iterable, np.float)

a_sma = talib.SMA(a_opn)

"""
self.tim = tim
self.opn = opn
self.hig = hig
self.low = low
self.cls = cls
self.vol = vol
"""
#self.debug(gox.history.candles[len-1].cls)
#self.debug(self.df)

self.debug(a_opn)
#self.debug(a_sma)

else:
self.debug("someone pressed the %s key" % chr(key))

def slot_tick(self, gox, (bid, ask)):
pass

def slot_depth(self, gox, (typ, price, volume, total_volume)):
pass

def slot_trade(self, gox, (date, price, volume, typ, own)):
"""a trade message has been received. Note that this might come
before the orderbook.owns list has been updated, don't rely on the
own orders and wallet already having been updated when this fires."""
pass

def slot_userorder(self, gox, (price, volume, typ, oid, status)):
"""this comes directly from the API and owns list might not yet be
updated, if you need the new owns list then use slot_owns_changed"""
pass

def slot_owns_changed(self, orderbook, _dummy):
"""this comes *after* userorder and orderbook.owns is updated already"""
pass

def slot_wallet_changed(self, gox, _dummy):
"""this comes after the wallet has been updated"""
pass

def slot_orderlag(self, dummy_sender, (usec, text)):
"""slot for order_lag messages"""
lag = datetime.timedelta(microseconds=usec)
lag_max = datetime.timedelta(seconds=60)
#self.debug(usec)
if lag>lag_max:
self.debug("MtGox is lagging {lag}".format(lag=lag))
self.is_lagging = True
else:
self.is_lagging = False

I was very surprised to see that candles are stored in a list...
why not using a Numpy Array ?

It will much efficient to implement indicators on top of a Numpy array than on top
of a Python list.

You should add a length for array as Numpy arrays have a fixed size (contrary to python list).

I was also surprised to see output for price such as

Code:

2013-04-26 18:02:23,152:DEBUG:test_strategy.Strategy:[ 13399024. 13590000. 13680000. 13610000. 13649999. 13680000.

In my mind (but I ever said something similar) in strategy" I should have "real" values.

I think that a "layer" is actually missing but I don't know how to implement it.

prof7bit

hero member

Activity: 938

Merit: 500

https://youengine.io/

Here is something not really serious but someone might still find it useful:

_speaker.py

Code:

"""
_speaker.py - A simple fun bot (no real usage) for goxtool.

This bot will read big market buys and sells loudly, a female voice will read
all buys and a male voice will read all sells that are above VOL_MIN. A 3rd
voice will read own orders that have been filled. It will attempt to sum up
partial trades by waiting until either a new tick message arrives or a trade
in opposite direction.

This will only work if espeak is installed.
On Ubuntu this is apt-get install espeak
"""
import os
import goxapi
import strategy

# minimum trade volume, market orders below that
# volume will be ignored. value is in BTC.
VOL_MIN = 10

# see the manual of espeak to understand how to tweak the voices.
VOICE_BUY = "-ven+f5"
VOICE_SELL = "-ven"
VOICE_OWN = "-ven+f5 -p10 -s150"

# if mbrola us voices are installed these also work
# VOICE_BUY = "-vmb-us1 -s150 -p70"
# VOICE_SELL = "-vmb-us2 -s150"
# VOICE_OWN = "-ven-us1 -s130 -p0"

def say(txt, voice="-ven"):
   """say the text (needs espeak installed)"""
   os.system("espeak %s '%s' 2>/dev/null 1>/dev/null &" % (voice, txt))

class Strategy(strategy.Strategy):
   """text to speech bot"""
   def __init__(self, gox):
   strategy.Strategy.__init__(self, gox)
   self.vol = 0
   self.action = 0

   def slot_trade(self, gox, (date, price, volume, typ, own)):
   prc = goxapi.int2float(price, gox.currency)
   vol = goxapi.int2float(volume, "BTC")

   if own:
   say("Order filled: %0.2f at %0.2f." % (vol, prc), VOICE_OWN)
   else:
   action = {"bid": "buy", "ask": "sell"}[typ]
   if action != self.action:
   self.say_trade()
   self.action = action
   self.vol += vol

   def slot_tick(self, gox, (bid, ask)):
   self.say_trade()

   def say_trade(self):
   """read the last trade (if any). a female voice will read
   the buy orders, a male voice the sell orders"""
   if self.vol >= VOL_MIN:
   if self.action == "buy":
   voice = VOICE_BUY
   else:
   voice = VOICE_SELL
   say("%s %0.0f." % (self.action, self.vol), voice)
   self.vol = 0

ErebusBat

hero member

Activity: 560

Merit: 500

I am the one who knocks

Quote from: prof7bit on April 26, 2013, 07:55:51 AM

I have added a command line option --password that can be used when starting it from within a script (don't use it on the command line unless you understand the implications)

Code:

./goxtool.py --password=YOURPASSWORD

I would HIGHLY recommend that you store your password on disk (with appropriate file permissions) then you can launch the bot with this command line, keeping the password out of your history file:

Code:

./goxtool.py --password=`cat ~/.mygoxtoolpassword`

Note that those are backticks (same key as the tilde, left of the 1 on most keyboards) NOT single quotes!

supert

full member

Activity: 160

Merit: 100

Quote from: smooth on April 24, 2013, 03:36:00 PM

I'm very surprised by a load of ~1 for this on a rpi. A (possible) memory leak surprises me less. I'll test this myself soon, but I think something is likely going wrong in your configuration.

having run it a bit longer it seems about 0.5 load most of the time and 55%-75% memory (256MB rpi).
The memory usage seems stable so it doesn't seem there is a (serious) leak.

prof7bit

hero member

Activity: 938

Merit: 500

https://youengine.io/

I have added a command line option --password that can be used when starting it from within a script (don't use it on the command line unless you understand the implications)

Code:

./goxtool.py --password=YOURPASSWORD

I have also added two new ini settings (they will be added to the .ini with default values after you start it the first time after update):

Code:

[goxtool]
orderbook_group = 0
orderbook_sum_total = False

The dafault values will display the orderbook exactly like it was in earlier versions of goxtool all the time already, so if you leave them at defaults then nothing will change. orderbook_group will have the same effect as it is used on the clarkmoody website, if you set it for example to 0.1 it will bin the price levels into $0.10 intervals (it won't display empty bins). 0 means no grouping at all, display every price level separately. orderbook_sum_total = True would sum up the total amount of BTC that needs to be bought or sold to reach and break that level instead of printing only the amount at that particular level like it was in earlier versions.

Please git pull to update.

prof7bit

hero member

Activity: 938

Merit: 500

https://youengine.io/

Quote from: K1773R on April 26, 2013, 06:51:38 AM

how about reading from STDIN?

Code:

python goxtool.py <<< YOURPASSWORDHERE

Doesn't work for me. I'm not a bash expert, so I'm not sure why it won't work (maybe the reason is how readline internally operates or the python getpass module). Also this would still leave the password in the .bash_history even if it worked.

K1773R

legendary

Activity: 1792

Merit: 1008

/dev/null

Quote from: prof7bit on April 26, 2013, 06:35:50 AM

Quote from: c0inbuster on April 26, 2013, 05:16:11 AM

What is the "best" way to do this (even if that's a bit insecure) ?
because when I run goxtool at startup I always get this message

Code:

enter passphrase for secret:

I would like to bypass this step

You are not the first one to ask me and I am going to implement one of these two options (not sure which one, probably the second one):

(1) A command line option (with a huge warning) to pass the pasword on the command line. It is insecure because the password will end up in the ~/.bash_history if you do this.

(2) The other alternative would be to make a separate .ini setting for the unencrypted secet that is automatically used (and the password question bypassed) when this setting is found in the ini.

how about reading from STDIN?

Code:

python goxtool.py <<< YOURPASSWORDHERE

prof7bit

hero member

Activity: 938

Merit: 500

https://youengine.io/

Quote from: c0inbuster on April 26, 2013, 05:16:11 AM

What is the "best" way to do this (even if that's a bit insecure) ?
because when I run goxtool at startup I always get this message

Code:

enter passphrase for secret:

I would like to bypass this step

You are not the first one to ask me and I am going to implement one of these two options (not sure which one, probably the second one):

(1) A command line option (with a huge warning) to pass the pasword on the command line. It is insecure because the password will end up in the ~/.bash_history if you do this.

(2) The other alternative would be to make a separate .ini setting for the unencrypted secet that is automatically used (and the password question bypassed) when this setting is found in the ini.

c0inbuster

member

Activity: 105

Merit: 10

@hugolp
you should have a look at PushOver API https://pushover.net/, it works with both Android and iOS.
You just have to install PushOver app on your Android device and use their HTTP API
https://play.google.com/store/apps/details?id=net.superblock.pushover&hl=fr
It's a paid apps but it worth it
you can find some inspiration here
https://code.google.com/p/working4arbitrage/source/browse/trunk/src/pushover_notifier.py
Have a look at priority
https://pushover.net/api#priority

@prof7bit
I'm looking for a way to run goxtool at startup (connected to my MtGox account) in a GNU Screen
I tried

Code:

$ python goxtool --add-secret

I set API and SECRET key
but I'm asked for a password... I can't set a blank password
What is the "best" way to do this (even if that's a bit insecure) ?
because when I run goxtool at startup I always get this message

Code:

enter passphrase for secret:

I would like to bypass this step

hugolp

legendary

Activity: 1148

Merit: 1001

Radix-The Decentralized Finance Protocol

Excuse if this is offtopic, but I want my goxtool bot to notify my Android phone, and even sound an alarm (in case Im sleeping) when certain conditions are met. Whats the best way of achieving this? I know how to program for Android (but would be happy if I can avoid having to create my own program just for this) and own Tasker. One option seems to use Google Cloud Messaging, anyone has any other suggestion?

ghostshirt

full member

Activity: 216

Merit: 100

Quote from: prof7bit on April 25, 2013, 05:08:27 PM

Quote from: ghostshirt on April 25, 2013, 01:29:53 PM

prof7bit, thanks for this great tool. I've just set it up and started to experiment with a bit of amount. I am intending to run it for a week. If I git pull during this time, and restart goxtool, should I press 'b' before 'p', or let it rebalance on its own with the price movement?

The balancer does not need to be re-initialized after you update/restart goxtool, it does not have any state, it can trade from the information about currently open orders and account balance alone, it will continue where it left automatically.

Are you using the newest version of the balancer that I posted on the separate thread that I started for the balancer? Questions about it should go to this thread.

Also please note (if you haven't seen it already) that since a few days (since a server upgrade at mtgox took place) the most reliable connection to mtgox is achieved with these command line options:
--protool=websocet --use-http
I'm mentioning this again because it is very important.

Sorry about the wrong thread.
I used the other protocol unfortunately. But the _balancer.py is the right one. I left my desktop running at office without a remote access.
Strangely, I seem to be on profit side in terms of BTC and a very small loss of $ after the crash we saw a few hours ago. newbie luck probably

prof7bit

hero member

Activity: 938

Merit: 500

https://youengine.io/

Quote from: ghostshirt on April 25, 2013, 01:29:53 PM

prof7bit, thanks for this great tool. I've just set it up and started to experiment with a bit of amount. I am intending to run it for a week. If I git pull during this time, and restart goxtool, should I press 'b' before 'p', or let it rebalance on its own with the price movement?

The balancer does not need to be re-initialized after you update/restart goxtool, it does not have any state, it can trade from the information about currently open orders and account balance alone, it will continue where it left automatically.

Are you using the newest version of the balancer that I posted on the separate thread that I started for the balancer? Questions about it should go to this thread.

Also please note (if you haven't seen it already) that since a few days (since a server upgrade at mtgox took place) the most reliable connection to mtgox is achieved with these command line options:
--protool=websocet --use-http
I'm mentioning this again because it is very important.

ghostshirt

full member

Activity: 216

Merit: 100

prof7bit, thanks for this great tool. I've just set it up and started to experiment with a bit of amount. I am intending to run it for a week. If I git pull during this time, and restart goxtool, should I press 'b' before 'p', or let it rebalance on its own with the price movement?

Quote from: prof7bit on April 20, 2013, 09:26:35 AM

And as i have mentioned before already: make sure you ALWAYS have an up to date version of goxtool. git pull (and restart if there were updates) at least once a day, I'm still fixing things in goxtool itself occasionally that make it more reliable.

hugolp

legendary

Activity: 1148

Merit: 1001

Radix-The Decentralized Finance Protocol

Quote from: ErebusBat on April 24, 2013, 02:48:40 PM

The code that handles that error is ~953:

Code:

def reload(self):
"""reload and re-initialize the strategy module"""
self.unload()
for name in self.strategy_name_list:
name = name.replace(".py", "").strip()

try:
strategy_module = __import__(name)
try:
reload(strategy_module)
strategy_object = strategy_module.Strategy(self.gox)
self.strategy_object_list.append(strategy_object)

# pylint: disable=W0703
except Exception:
self.gox.debug(traceback.format_exc())

except ImportError:
self.gox.debug("### could not import %s.py" % name)

I would change that log message to output more information if I were you... by the looks of it I would guess (not being a python person at all) that your class name doesn't match your file name.

Thanks, that helped. I had an typo in one of my import statements so the module refused to load. prof7bit, it would be better if the log signaled the particular error otherwise you are in the dark making it hard to debug. Basically this change:

Code:

except ImportError as e:
self.gox.debug("### could not import %s.py, reason: %s" % (name,e))

Topic: ncurses based MtGox live monitor and trading-bot-framework - page 5. (Read 33839 times)