Pages:
Author

Topic: ncurses based MtGox live monitor and trading-bot-framework - page 6. (Read 33839 times)

legendary
Activity: 2968
Merit: 1198
Fantastic work prof7bit.

It runs OK on a raspberry pi raspbian / wheezy (wanted an always-on low power bot server) at a load average of ~0.99 and most of the memory. Hopefully there is no memory leak.

I'm not running X and in the console the xterm title (price/bid/ask) seems to cause some trouble appearing at wherever the cursor was last.

With screen over ssh in an xterm it looks fine though.


I'm very surprised by a load of ~1 for this on a rpi.  A (possible) memory leak surprises me less.  I'll test this myself soon, but I think something is likely going wrong in your configuration.

hero member
Activity: 938
Merit: 500
https://youengine.io/
Code:
2013-04-24 20:21:58,510:DEBUG:goxapi.Gox:### could not import mybot.py

Make sure the current working directory is the goxtool directory and the mybot.py is in the same directory too.
hero member
Activity: 560
Merit: 500
I am the one who knocks
Im writting a bot (Strategy class) and Im having the problem that goxtool sometimes will just quit leaving a grey screen with no error message, making it hard to debug where I have fucked up in the code. Sometimes the error will show but others it just completely kills the program with no explanation. Is there a way to find out what happen or a way to solve it so one can get a debug message?

Im hitting this problem again and now I just get a message saying:

Code:
2013-04-24 20:21:58,510:DEBUG:goxapi.Gox:### could not import mybot.py

The program keeps running but without running the strategy obviously. And yes, the file mybot.py exists and is the right one. No other indication as to what might be happening. Any idea of what I could do to know where I got the code wrong?

The code that handles that error is ~953:
Code:
    def reload(self):
        """reload and re-initialize the strategy module"""
        self.unload()
        for name in self.strategy_name_list:
            name = name.replace(".py", "").strip()

            try:
                strategy_module = __import__(name)
                try:
                    reload(strategy_module)
                    strategy_object = strategy_module.Strategy(self.gox)
                    self.strategy_object_list.append(strategy_object)

                # pylint: disable=W0703
                except Exception:
                    self.gox.debug(traceback.format_exc())

            except ImportError:
                self.gox.debug("### could not import %s.py" % name)

I would change that log message to output more information if I were you... by the looks of it I would guess (not being a python person at all) that your class name doesn't match your file name.
legendary
Activity: 1148
Merit: 1001
Radix-The Decentralized Finance Protocol
Im writting a bot (Strategy class) and Im having the problem that goxtool sometimes will just quit leaving a grey screen with no error message, making it hard to debug where I have fucked up in the code. Sometimes the error will show but others it just completely kills the program with no explanation. Is there a way to find out what happen or a way to solve it so one can get a debug message?

Im hitting this problem again and now I just get a message saying:

Code:
2013-04-24 20:21:58,510:DEBUG:goxapi.Gox:### could not import mybot.py

The program keeps running but without running the strategy obviously. And yes, the file mybot.py exists and is the right one. No other indication as to what might be happening. Any idea of what I could do to know where I got the code wrong?
full member
Activity: 160
Merit: 100
You can add (or change) the following in your ini file and then the title will not update Smiley

Code:
set_xterm_title = False

thanks.

Is there a way to moderate what is logged / verbosity (I guess I can comment out the code)
-- would also be a nice .ini option
hero member
Activity: 560
Merit: 500
I am the one who knocks
Fantastic work prof7bit.

It runs OK on a raspberry pi raspbian / wheezy (wanted an always-on low power bot server) at a load average of ~0.99 and most of the memory. Hopefully there is no memory leak.

I'm not running X and in the console the xterm title (price/bid/ask) seems to cause some trouble appearing at wherever the cursor was last.

With screen over ssh in an xterm it looks fine though.



You can add (or change) the following in your ini file and then the title will not update Smiley

Code:
set_xterm_title = False
full member
Activity: 160
Merit: 100
Fantastic work prof7bit.

It runs OK on a raspberry pi raspbian / wheezy (wanted an always-on low power bot server) at a load average of ~0.99 and most of the memory. Hopefully there is no memory leak.

I'm not running X and in the console the xterm title (price/bid/ask) seems to cause some trouble appearing at wherever the cursor was last.

With screen over ssh in an xterm it looks fine though.

hero member
Activity: 560
Merit: 500
I am the one who knocks
I thought I would share several of my helper scripts I have written to help with goxtool

Bot Start Script
This script is used to rotate the log file and start the bot with the appropriate connection params, as well as a log of where and when it was said to use those.
Code:
#!/bin/zsh
# https://bitcointalk.org/index.php?topic=148462.msg1854636#msg1854636
# https://gist.github.com/prof7bit/5395900
# https://bitcointalk.org/index.php?topic=148462.msg1886914#msg1886914

# "p": place 2 orders above and below price.
# "c": cancel orders and effectively stop the bot
# "u": update own order list, depth, history, wallet and everything
# "i": display how much it is out of balance at current price (negative: must sell, positive: must buy)
# "b":balance immediately (cancel orders and then buy/sell at market to rebalance)

# Backup log file
if [[ -f goxtool.log ]]; then
  mv goxtool.log goxtool_$(date '+%Y%m%d-%H%M%S').log
fi
# Per https://bitcointalk.org/index.php?topic=181584.msg1923260#msg1923260
./goxtool.py --protocol=websocket --strategy=_balancer.py


botlog.sh - View/watch bot's log
This filters out the 'noise' and lets me see just the output I want from the log.
As a side effect of the above log rotation you will need to restart this every time you restart the tool with the script above

Code:
#!/bin/zsh
tail -n500000 -f ~/goxtool/goxtool.log | egrep "(Strategy|got ack|got|have)"

startbot.sh - Enforce Screen Session
Forces that you are in a screen session so you don't start the bot and then logout and think it is running when it is not  Shocked
Code:
#!/bin/zsh
if [[ -z $STY ]] then
echo "Not in screen! Run ~/botwindow.sh"
else
cd ~/goxtool
./bot_balancer.sh
fi

botwindow.sh
Starts (or re-connects to) screen with the same params every time
Code:
#!/bin/sh
screen -D -R -A -S bot
newbie
Activity: 24
Merit: 0
+1 for 2.6
Debian squeeze has 2.5, 2.6 and 3.0 in repo.

I just installed goxtool on a squeeze box and I made this script to help anyone else in the same situation.

Code:
# Installing python2.7 on debian stable and running goxtool
# Tested on virgin debian squeeze install with sudo installed

####################################################
# Part 1: Install python2.7

# First get some tools
sudo apt-get install build-essential
# and some libs
sudo apt-get install libsqlite3-dev zlib1g-dev libncurses5-dev libgdbm-dev libbz2-dev libreadline5-dev libssl-dev libdb-dev
 
# Optional bits and bobs
#sudo apt-get build-dep python

# starting at home
cd ~

# Get and compile Python-2.7.4
wget wget http://python.org/ftp/python/2.7.4/Python-2.7.4.tar.bz2
tar xf Python-2.7.4.tar.bz2
cd Python-2.7.4/
./configure --prefix=/usr/local
make

# This make takes about 5 mins on an old virtual machine

# Now you have 2 options:
# 1. install the new python
#  sudo make altinstall  
# 2. create a new debian package and install python 2.7 into debian package system
#  sudo checkinstall make altinstall

# I prefer checkinstall, though it will take longer
sudo apt-get install checkinstall

# There is a bug and checkinstall will fail because some directories could not be created.
# So just create them manually right now.
sudo mkdir -p /usr/local/lib/pkgconfig
sudo mkdir -p /usr/local/lib/python2.7/config

# Install and create the debian package
sudo checkinstall --pkgname=python2.7 --pkgversion=2.7.4 make altinstall

# checkinstall took about 45 minutes on my slow VM

# have a look at the new python
python2.7 --version
# and still we have the original 2.6 as the system wide default
python --version

# get back to home
cd ~

# install a python package manager
# this will install the script 'easy_install-2.7'
wget https://pypi.python.org/packages/source/d/distribute/distribute-0.6.36.tar.gz
tar xvf distribute-0.6.36.tar.gz
cd distribute-0.6.36/
sudo python2.7 setup.py install

####################################################
# Part 2: Install goxtool

# get back to home
cd ~

# install a means to segregate python applications
sudo easy_install-2.7 virtualenv

# get latest goxtool
git clone https://github.com/prof7bit/goxtool.git

# create a new python2.7 environment for goxtool
virtualenv-2.7 --distribute goxtool

# switch to the new environment
source goxtool/bin/activate

# check it worked - you should see 2.7.4 as the version number
python --version

# get the packages required for goxtool
easy_install-2.7 pycrypto

# fire her up and get trading
python goxtool/goxtool.py --protocol=socketio --use-http


I've only just discovered this project with it's balance bot and I love it and have been running it for the last day and a half on the 7% default.
I'm going to try some different margins on other accounts with like half a coin balance and see how they each perform over time.
My first trading bot.

One issue.  During setup I balanced the account and then hit 'p' to place the buy/sell orders but I think I went too fast because the price was around 123 and the bot miscalculated my center and put in buys and sells at 80/90 something.  MtGox ignored the low sell but I did panic.

Also the graph is using a lot of screen space so could do with a toggle for switching between other information sets.  I'm a C++ programmer by trade but I'm keen to help out with this projects so I'm currently looking at the code.
Graph area could switch between log, graph, more detailed order book stats, strategy provided info.
Strategy provided would be the obvious solution to all as then you could have empty strategies that only provide information screens.  Then just have a mechanism to cycle through them.  So even the current graph would end up in a strategy.

Thanks for your work prof7bit.
hero member
Activity: 938
Merit: 500
https://youengine.io/
Is there a plan to implement such a test-mode into goxtool, so you are able to test your trading strategy?

That would certainly be a nice thing to have. I'm not yet sure if it should be part of goxtool or a separate application that just exposes the same interface and can load the same strategy files. I cannot promise anything yet (don't have unlimited time), maybe also someone else might want to write something that could be used for this purpose.
hero member
Activity: 560
Merit: 500
I am the one who knocks
I almost did not want to answer for the embarasment. Yes, it was there. Thank you.
NP... we all have our moments so it is good to ask Smiley  Plus your question will help others.
legendary
Activity: 1148
Merit: 1001
Radix-The Decentralized Finance Protocol
Im writting a bot (Strategy class) and Im having the problem that goxtool sometimes will just quit leaving a grey screen with no error message, making it hard to debug where I have fucked up in the code. Sometimes the error will show but others it just completely kills the program with no explanation. Is there a way to find out what happen or a way to solve it so one can get a debug message?
STUPID QUESTION:  Nothing in goxtool.log?

I almost did not want to answer for the embarasment. Yes, it was there. Thank you.
hero member
Activity: 560
Merit: 500
I am the one who knocks
Im writting a bot (Strategy class) and Im having the problem that goxtool sometimes will just quit leaving a grey screen with no error message, making it hard to debug where I have fucked up in the code. Sometimes the error will show but others it just completely kills the program with no explanation. Is there a way to find out what happen or a way to solve it so one can get a debug message?
STUPID QUESTION:  Nothing in goxtool.log?
legendary
Activity: 1148
Merit: 1001
Radix-The Decentralized Finance Protocol
Im writting a bot (Strategy class) and Im having the problem that goxtool sometimes will just quit leaving a grey screen with no error message, making it hard to debug where I have fucked up in the code. Sometimes the error will show but others it just completely kills the program with no explanation. Is there a way to find out what happen or a way to solve it so one can get a debug message?
sr. member
Activity: 360
Merit: 250
Have you ever try to implement MA crossover strategy ?
I experimented with this back in my old forex days and never found something that would work really reliable when I simulated it with new (not seen before) test data. All the myriads of MA-crossover strategies (or whatever-indicator-cross-this-or-that-strategies) that looked promising on the data they were developed on failed miserably when simulating them on new test data. All of them! Every time! Even the most sophisticated stuff with neural networks and other machine learning technologies trying to predict the trend that seemed to work well on the training set failed miserably and immediately every time when confronted with new data, so I won't even try to do that anymore.

It looks like you have large experience with automatic trading technologies. I have only some basic knowledge. One of the things I know about it, is that it's important to test/fine-tune your trading strategy with historical or synthesizer new data before going live. I think this is also what you are talking about your experience with various MA-crossover strategies. Is there a plan to implement such a test-mode into goxtool, so you are able to test your trading strategy?
hero member
Activity: 938
Merit: 500
https://youengine.io/
Have you ever try to implement MA crossover strategy ?
I experimented with this back in my old forex days and never found something that would work really reliable when I simulated it with new (not seen before) test data. All the myriads of MA-crossover strategies (or whatever-indicator-cross-this-or-that-strategies) that looked promising on the data they were developed on failed miserably when simulating them on new test data. All of them! Every time! Even the most sophisticated stuff with neural networks and other machine learning technologies trying to predict the trend that seemed to work well on the training set failed miserably and immediately every time when confronted with new data, so I won't even try to do that anymore.
member
Activity: 105
Merit: 10
Thanks for this !

Have you ever try to implement MA crossover strategy ?
http://www.onlinetradingconcepts.com/TechnicalAnalysis/MASimple2.html

How can I use indicator MACD, RSI... ?
hero member
Activity: 938
Merit: 500
https://youengine.io/
Do you know how to load list of strategies ?
_loader.py
[ ...code...]
This is obsolete now. After pulling the latest revision you can now just pass a comma separated list of modules on the command line, no need to load them from a separate loader strategy, this is much more flexible now:

Code:
./goxtool --strategy=foo.py,bar.py,baz.py
member
Activity: 105
Merit: 10
Nice!

I prefer not to get away from your code...
so I will git pull when it will be ready

I think it's both a good idea :
- to avoid breaking a large part of code (and that's what you are doing elegantly)
- to have notifications when auto trading (I prefer to know very quickly [bad] surprises)

You should also have a look at my suggestion
https://bitcointalksearch.org/topic/m.1911355
it will allow us to see several symbols (BTCUSD, BTCEUR...)
hero member
Activity: 938
Merit: 500
https://youengine.io/
Do you know how to load list of strategies ?

_loader.py (untested code, just wrote it down here without testing but should work this way unless I have forgotten something, for example not tested what happens on reload during runtime)
(use it like this: ./goxtool --strategy=_loader.py)

(my [personal] convention is to prefix all strategy module filenames (and other eperimental stuff in this folder) with _ to not confuse git too much with unversioned files in the same folder, I have put _*.* into my .gitignore file)
Code:
import strategy

import _notifierstrat
import _foostrat
import _barstrat
import _whateverstrat

class Strategy(strategy.Strategy):
    def __init__(self, gox):
        strategy.Strategy.__init__(self, gox)

        self.notifierstrat  =  _notifierstrat.Strategy(gox)
        self.foostrat       =       _foostrat.Strategy(gox)
        self.barstrat       =       _barstrat.Strategy(gox)
        self.whateverstrat  =  _whateverstrat.Strategy(gox)
Pages:
Jump to: