Pages:
Author

Topic: ncurses based MtGox live monitor and trading-bot-framework - page 4. (Read 33839 times)

sr. member
Activity: 288
Merit: 250
ManualMiner
Hy prof7bit,
maybe i am blind, but i left the parts as they were in the rebalancing bot.

what i saw now from looking into
http://www.rugatu.com/questions/14260/python-trading-bot-configuration-file

is that the rebalancing bot does
import strategy
and in strategy.py there is a function slot_keypress
in the bot of tarzan he doesnt import strategy and defines the class Strategy(goxapi.BaseObject):
- but if thats the problem, how does the balancing bot work?
thx
hero member
Activity: 938
Merit: 500
https://youengine.io/
hy
i started to look into goxbot and i must admit, i dont get anything, but anyway i will reach the goal of my own little bot. Wink plz help!

in my strategy module i got (copied most of it of the 50/50-balance boot)

Code:
import strategy
import goxapi


class Strategy(strategy.Strategy):

    """a protfolio rebalancing bot"""
    def __init__(self, gox):
        strategy.Strategy.__init__(self, gox)
        self.temp_halt = False


    def slot_before_unload(self, _sender, _data):
        pass

    def slot_keypress(self, gox, (key)):
if key == ord("b"):
    self.debug("canceling all rebalancing orders")

This can't work. Please fix the formatting and indentations. And set your editor to 4 spaces, no tabs (recommended style, used by most others too) and dont ever mix tabs and spaces. You should first make yourself comfortable with some essential concepts of the Python programming language with simpler examples and simple isolated standalone programs, follow some tutorials. You should not attempt to ride a motorbike until you can safely ride a bicycle. This will avoid many frustrations.
sr. member
Activity: 288
Merit: 250
ManualMiner
thx, so your log shows the message, mine doesnt; there is something wrong with my setup, but i cant figure out what.
i.e. this script prints out the messages:
http://www.rugatu.com/questions/14260/python-trading-bot-configuration-file

my bot  does load - debug message appears:

2013-04-30 15:30:24,177:DEBUG:testbot.Strategy:testbot.Strategy unloaded

and keystroke is registered, but the self.debug message doesnt print.

If I use it i.e. with an non existent bot file (goxtool.py --strategy=anything.py

it does not recognize the key pressed.
I am stuck with this
newbie
Activity: 24
Merit: 0
hy
i started to look into goxbot and i must admit, i dont get anything, but anyway i will reach the goal of my own little bot. Wink plz help!

in my strategy module i got (copied most of it of the 50/50-balance boot)

Code:
import strategy
import goxapi


class Strategy(strategy.Strategy):

    """a protfolio rebalancing bot"""
    def __init__(self, gox):
        strategy.Strategy.__init__(self, gox)
        self.temp_halt = False


    def slot_before_unload(self, _sender, _data):
        pass

    def slot_keypress(self, gox, (key)):
if key == ord("b"):
     self.debug("canceling all rebalancing orders")






If key "b" is pressed goxtool displays: someone pressed key b,
But it doesnt display "canceling all rebalancing orders".
so, how to write messages to the log output?


btw: is there an easy way to get rid of the Winconsole output in the main window? I am running tail on the log in a second window, so its just superfluid for me?

Your strategy file works perfectly though it won't pay the bills.
Make sure you are loading it.  e.g.
Code:
python ./goxtool.py --protocol=websocket --strategy=_test.py
And you should see the load and unload in goxtool.log
Code:
2013-05-01 14:47:32,433:DEBUG:_test.Strategy:_test.Strategy loaded
...
2013-05-01 14:47:38,099:DEBUG:_test.Strategy:canceling all rebalancing orders
...
2013-05-01 14:47:40,974:DEBUG:_test.Strategy:_test.Strategy unloaded
sr. member
Activity: 288
Merit: 250
ManualMiner
hy
i started to look into goxbot and i must admit, i dont get anything, but anyway i will reach the goal of my own little bot. Wink plz help!

in my strategy module i got (copied most of it of the 50/50-balance boot)

Code:
import strategy
import goxapi


class Strategy(strategy.Strategy):

    """a protfolio rebalancing bot"""
    def __init__(self, gox):
        strategy.Strategy.__init__(self, gox)
        self.temp_halt = False


    def slot_before_unload(self, _sender, _data):
        pass

    def slot_keypress(self, gox, (key)):
if key == ord("b"):
    self.debug("canceling all rebalancing orders")






If key "b" is pressed goxtool displays: someone pressed key b,
But it doesnt display "canceling all rebalancing orders".
so, how to write messages to the log output?


btw: is there an easy way to get rid of the Winconsole output in the main window? I am running tail on the log in a second window, so its just superfluid for me?
newbie
Activity: 24
Merit: 0
EDIT: MtQuid tried your code and its working perfectly. Im sending you a small tip.

Tip recieved. 
Thanks hugolp Smiley
legendary
Activity: 1148
Merit: 1001
Radix-The Decentralized Finance Protocol
MtQuid that looks terrific. Hopefully prof7bit can include that or something similar in the source.

Another solution might be being able to select what goes into the debug at runtime, so one could select to only have in the debug file the messages from the modules s/he wants. Im fine with any of both solutions.

EDIT: MtQuid tried your code and its working perfectly. Im sending you a small tip.
newbie
Activity: 24
Merit: 0
EDIT: Also, can anyone give me a hint on how to create a separate debug file for my bot? The regular debug goxtool file gets too big and its very cumbersome to check.

To reduce debug clutter for writing strategies I use a crafty tail.  I did have a cut on the end but it lagged
Code:
tail -f goxtool.log| grep -E 'Strat|Trace|^[^0-9]'

Yeah, thats an option, but I still would preffer to have a different file so I can check it from time to time.

And how are the results?.

I just loaded the chart along with current data from bitcoin charts and they are pretty similar. 
http://bitcoincharts.com/charts/chart.json?m=mtgoxUSD&r=1&i=15-min
I don't know what weighted moving average function they are using but the data is close enough for my use though I probably should check it out on a few more days before I invest the yacht Cheesy
If you use this data you will need to alter the timestamp for HighStock.js  add on 000 (three zeros)

EDIT: I haven't checked for the 1 hour EMAs but will do some more testing and experimenting later.  Using the TA-Lib you can specify the timeperiod and then ignore those first timeperiod output values as the results settle down.  I've used the default values from bitcoincharts.  Simple at 10 and Weighted at 25.

Thanks for the direction, I will try using ta-lib between today and tomorrow, hopefully it gives better results that my manually created one (which its not difficult...).

I just put this together.
This will create a log file for each loaded strategy as well as sending log messages to the main log.
Maybe prof7bit could check it over and put it into the main source.
It's a quick hack but it works.
And you call self.debug() just like before and there is a new function self.info() for different log level.

Change Strategy.py to look like this.

Code:
"""
trading robot breadboard
"""

import goxapi
import logging


class Strategy(goxapi.BaseObject):
    # pylint: disable=C0111,W0613,R0201

    def __init__(self, gox):
        goxapi.BaseObject.__init__(self)
        self.signal_debug.connect(gox.signal_debug)
        gox.signal_keypress.connect(self.slot_keypress)
        gox.signal_strategy_unload.connect(self.slot_before_unload)
        gox.signal_ticker.connect(self.slot_tick)
        gox.signal_depth.connect(self.slot_depth)
        gox.signal_trade.connect(self.slot_trade)
        gox.signal_userorder.connect(self.slot_userorder)
        gox.orderbook.signal_owns_changed.connect(self.slot_owns_changed)
        gox.history.signal_changed.connect(self.slot_history_changed)
        gox.signal_wallet.connect(self.slot_wallet_changed)
        self.gox = gox
        self.name = "%s.%s" % \
            (self.__class__.__module__, self.__class__.__name__)

        self.logger = logging.getLogger(self.name)
        self.fh = logging.FileHandler(self.name + '.log')
        formatter = logging.Formatter('%(asctime)s:%(levelname)s:%(message)s')
        self.fh.setFormatter(formatter)
        self.fh.setLevel(logging.DEBUG)
        self.logger.addHandler(self.fh)

        self.info("%s loaded" % self.name)

    def info(self, *args):
        """write info event to log.  use our logger then send to the base logger"""
        msg = " ".join([str(x) for x in args])
        self.logger.info(msg)
        goxapi.BaseObject.debug(self, *args)


    def debug(self, *args):
        """write debug event to log.  use our logger then send to the base logger"""
        msg = " ".join([str(x) for x in args])
        self.logger.debug(msg)
        goxapi.BaseObject.debug(self, *args)

    def __del__(self):
        """the strategy object will be garbage collected now, this mainly
        only exists to produce the log message, so you can make sure it
        really garbage collects and won't stay in memory on reload. If you
        don't see this log mesage on reload then you have circular references"""
        self.debug("%s unloaded" % self.name)
        self.logger.removeHandler(self.fh)

    def slot_before_unload(self, _sender, _data):
        """the strategy is about to be unloaded. Use this signal to persist
        any state and also use it to forcefully destroy any circular references
        to allow it to be properly garbage collected (you might need to do
        this if you instantiated linked lists or similar structures, the
        symptom would be that you don't see the 'unloaded' message above."""
        pass

    def slot_keypress(self, gox, (key)):
        """a key in has been pressed (only a..z without "q" and "l")"""
        self.debug("someone pressed the %s key" % chr(key))

    def slot_tick(self, gox, (bid, ask)):
        """a tick message has been received from the streaming API"""
        pass

    def slot_depth(self, gox, (typ, price, volume, total_volume)):
        """a depth message has been received. Use this only if you want to
        keep track of the depth and orderbook updates yourself or if you
        for example want to log all depth messages to a database. This
        signal comes directly from the streaming API and the gox.orderbook
        might not yet be updated at this time."""
        pass

    def slot_trade(self, gox, (date, price, volume, typ, own)):
        """a trade message has been received. Note that this signal comes
        directly from the streaming API, it might come before orderbook.owns
        list has been updated, don't rely on the own orders and wallet already
        having been updated when this is fired."""
        pass

    def slot_userorder(self, gox, (price, volume, typ, oid, status)):
        """this comes directly from the API and owns list might not yet be
        updated, if you need the new owns list then use slot_owns_changed"""
        pass

    def slot_owns_changed(self, orderbook, _dummy):
        """this comes *after* userorder and orderbook.owns is updated already.
        Also note that this signal is sent by the orderbook object, not by gox,
        so the sender argument is orderbook and not gox. This signal might be
        useful if you want to detect whether an order has been filled, you
        count open orders, count pending orders and compare with last count"""
        pass

    def slot_wallet_changed(self, gox, _dummy):
        """this comes after the wallet has been updated. You can access the
        new balance like so: gox.wallet["BTC"] or gox.wallet[gox.currency]"""
        pass

    def slot_history_changed(self, history, _dummy):
        """this is fired whenever a new trade is inserted into the history,
        you can also use this to query the close price of the most recent
        candle which is effectvely the price of the last trade message.
        Contrary to the slot_trade this also fires when streaming API
        reconnects and re-downloads the trade history, you can use this
        to implement a stoploss or you could also use it for example to detect
        when a new candle is opened"""
        pass

EDIT1: Changed formatter to
Code:
formatter = logging.Formatter('%(asctime)s:%(levelname)s:%(message)s')
EDIT2: Changed call to base debug to use *arg instead of arg so is now
Code:
goxapi.BaseObject.debug(self, *args)
No more edits.  Code is flawless Cheesy
EDIT3: Need to remove file handler on strategy unload so that it is not reloaded twice or more during a strategy reload
I think that should do it now.  Code is never flawless Sad
Price is also going down.... Time for tea
legendary
Activity: 1148
Merit: 1001
Radix-The Decentralized Finance Protocol
EDIT: Also, can anyone give me a hint on how to create a separate debug file for my bot? The regular debug goxtool file gets too big and its very cumbersome to check.

To reduce debug clutter for writing strategies I use a crafty tail.  I did have a cut on the end but it lagged
Code:
tail -f goxtool.log| grep -E 'Strat|Trace|^[^0-9]'

Yeah, thats an option, but I still would preffer to have a different file so I can check it from time to time.

And how are the results?.

I just loaded the chart along with current data from bitcoin charts and they are pretty similar. 
http://bitcoincharts.com/charts/chart.json?m=mtgoxUSD&r=1&i=15-min
I don't know what weighted moving average function they are using but the data is close enough for my use though I probably should check it out on a few more days before I invest the yacht Cheesy
If you use this data you will need to alter the timestamp for HighStock.js  add on 000 (three zeros)

EDIT: I haven't checked for the 1 hour EMAs but will do some more testing and experimenting later.  Using the TA-Lib you can specify the timeperiod and then ignore those first timeperiod output values as the results settle down.  I've used the default values from bitcoincharts.  Simple at 10 and Weighted at 25.

Thanks for the direction, I will try using ta-lib between today and tomorrow, hopefully it gives better results that my manually created one (which its not difficult...).
newbie
Activity: 24
Merit: 0
MtQuid...
Would you be willing to share a basic demo of the Ta-lib integration?  I do not live in python so it would be very helpful for me, and I am sure others. 

Ok.

We will use TA-Lib for bython from these lovely chaps
http://mrjbq7.github.io/ta-lib/index.html

Assuming you are working on a debian system you will need to do the following to install ta-lib for python
If on windows or mac then check http://mrjbq7.github.io/ta-lib/install.html

Code:
sudo apt-get install python-dev
sudo easy_install cython

wget http://prdownloads.sourceforge.net/ta-lib/ta-lib-0.4.0-src.tar.gz
tar -xvf ta-lib-0.4.0-src.tar.gz
cd ta-lib
./configure --prefix=/usr
make

#can use make install here if you want
sudo checkinstall

sudo easy_install TA-Lib

Once you have TA-Lib installed this is your strategy
Code:
"""
_talib.py

This example strategy will produce some HighStock.js compatible JSON files
"""

import numpy as np
import talib
import datetime
import json
import strategy
import goxapi
from goxapi import OHLCV

class OHLCV_Encoder(json.JSONEncoder):
    """JSONEncoder for class OHLCV()"""
    def default(self, obj):
        if isinstance(obj, OHLCV):
            return [obj.tim, obj.opn, obj.hig, obj.low, obj.cls, obj.vol]
        else:
            return json.JSONEncoder.default(self, obj)

class Strategy(strategy.Strategy):
    """a protfolio rebalancing bot"""
    def __init__(self, gox):       
        strategy.Strategy.__init__(self, gox)

    def slot_history_changed(self, history, _dummy):
        """History has changed so recalculate EMAs"""
        candles = []

        # read them all - don't wory about the history parameter
        for c in reversed(self.gox.history.candles):
            candles.append(
                OHLCV(
                    # fix time for HighStock.js JSON import
                    c.tim * 1000,
                    # adjust values to be human readable
                    goxapi.int2float(c.opn, self.gox.currency),
                    goxapi.int2float(c.hig, self.gox.currency),
                    goxapi.int2float(c.low, self.gox.currency),
                    goxapi.int2float(c.cls, self.gox.currency),
                    goxapi.int2float(c.vol, "BTC")
                )
            ) 

        self.debug("New EMAs from history with %d candles" % len(candles))               

        rng = range(len(candles))
        iterable = (candles[i].opn for i in rng)
        a_opn = np.fromiter(iterable, np.float)

        iterable = (candles[i].hig for i in rng)
        a_hig = np.fromiter(iterable, np.float)
       
        iterable = (candles[i].low for i in rng)
        a_low = np.fromiter(iterable, np.float)
       
        iterable = (candles[i].cls for i in rng)
        a_cls = np.fromiter(iterable, np.float)

        iterable = (candles[i].vol for i in rng)
        a_vol = np.fromiter(iterable, np.float)

        iterable = (candles[i].tim for i in rng)
        a_tim = np.fromiter(iterable, np.int64)


        a_sma = talib.SMA(a_cls, 10)
        a_wma = talib.WMA(a_cls, 25)
        a_chaikin = talib.AD(a_hig, a_low, a_cls, a_vol)
        a_cdlCounterAttack = talib.CDLCOUNTERATTACK(a_opn, a_hig, a_low, a_cls)

        # create json compatible with HighStock.js
        with open("talib_ohlcv.json", 'w') as outfile:                   
            json.dump(candles, outfile, cls = OHLCV_Encoder)
        with open("talib_sma.json", 'w') as outfile:
            # first 10 elements contain Nan
            json.dump(np.dstack((a_tim[10:], a_sma[10:])).tolist()[0], outfile)
        with open("talib_wma.json", 'w') as outfile:
            # first 25 elements contain Nan
            json.dump(np.dstack((a_tim[25:], a_wma[25:])).tolist()[0], outfile)
        with open("talib_chaikin.json", 'w') as outfile:
            json.dump(np.dstack((a_tim, a_chaikin)).tolist()[0], outfile)
        with open("talib_cdlCounterAttack.json", 'w') as outfile:
            json.dump(np.dstack((a_tim, a_cdlCounterAttack)).tolist()[0], outfile)

        # Current price in relation to EMA
        self.debug("SMA = %f" % a_sma[-1])
        self.debug("WMA = %f" % a_wma[-1])
        self.debug("CLS = %f" % a_cls[-1])

Run goxtool like so

Code:
python ./goxtool.py --protocol=websocket --strategy=_talib.py

Create index.html like this

Code:




BTC Highstock - Goxtool/TA-Lib










And run a simple web server like this
Code:
python -m SimpleHTTPServer 8080

Then go to http://localhost:8080/index.html and see sunshine

You could have the index.html auto reload with simple script or refresh with your mouse Smiley


EDIT: Also, can anyone give me a hint on how to create a separate debug file for my bot? The regular debug goxtool file gets too big and its very cumbersome to check.

To reduce debug clutter for writing strategies I use a crafty tail.  I did have a cut on the end but it lagged
Code:
tail -f goxtool.log| grep -E 'Strat|Trace|^[^0-9]'



And how are the results?.

I just loaded the chart along with current data from bitcoin charts and they are pretty similar. 
http://bitcoincharts.com/charts/chart.json?m=mtgoxUSD&r=1&i=15-min
I don't know what weighted moving average function they are using but the data is close enough for my use though I probably should check it out on a few more days before I invest the yacht Cheesy
If you use this data you will need to alter the timestamp for HighStock.js  add on 000 (three zeros)

EDIT: I haven't checked for the 1 hour EMAs but will do some more testing and experimenting later.  Using the TA-Lib you can specify the timeperiod and then ignore those first timeperiod output values as the results settle down.  I've used the default values from bitcoincharts.  Simple at 10 and Weighted at 25.
hero member
Activity: 560
Merit: 500
I am the one who knocks
MtQuid...
Would you be willing to share a basic demo of the Ta-lib integration?  I do not live in python so it would be very helpful for me, and I am sure others. 
legendary
Activity: 1148
Merit: 1001
Radix-The Decentralized Finance Protocol
I only wanted the graph output to make sure the ta-lib functions were returning the correct values so I could use them to control my bot.

And how are the results?.

I manually created a ema function, basically following the formula you see everywhere in the Internet (f.e. http://stockcharts.com/school/doku.php?id=chart_school:technical_indicators:moving_averages) but Im getting results quite different from bitcoincharts.com, specially for the hourly ema's but almost identical for the 1minute ema's. Because of this my guess was: the formula calculates the sma with the first periods and then starts adding the data as a exponential. I assumed this means that over time with enough data it would converge and that is why the 1minute ema's were almost identical while the hourly ema's were not, since Im getting data of two days and there is only 48 hours while there is 48 * 60 minutes. Does ta-lib get better results with data from one day only?

Quote
I don't think this functionality needs to be part of the main goxtool branch but a good strategy plugin using ta-lib would be handy.  Like providing values and firing signals that can be caught.

I actually would love to have the the web part in goxtool, maybe just as a plugin or something.

EDIT: Also, can anyone give me a hint on how to create a separate debug file for my bot? The regular debug goxtool file gets too big and its very cumbersome to check.
newbie
Activity: 24
Merit: 0
Yep they do look good.  I robbed some of the colour styling straight from MtGox.com

I only wanted the graph output to make sure the ta-lib functions were returning the correct values so I could use them to control my bot.  I don't think this functionality needs to be part of the main goxtool branch but a good strategy plugin using ta-lib would be handy.  Like providing values and firing signals that can be caught.

And because the data sets are small (1 days worth of trade) you could write the whole index.html file to include the data and then there would be no need for the .json files or a web server.  Just pops out a stand alone html page every now and again if that's what you require.

I've also moved my function so it automatically gets called when new history arrives.

Code:
    def slot_history_changed(self, history, _dummy):
        """History has changed so recalculate EMAs"""

Now to try and find the best usage for the ta-lib functions and make some coins..  Grin
member
Activity: 105
Merit: 10
I had some problems to see HTML file
so I contacted MtQuid using PM
I post here for other members

in fact you have to run a webserver (and not directly open HTML file with a browser)

Quote
python -m SimpleHTTPServer 8080

Open index.htm in your web browser using this url http://127.0.0.1:8080/index.hml


You will get a beautiful chart



It will be nice to also make orderbook depth chart with a JS lib

Maybe adding this feature to goxgui (goxtool GUI) will be fine
https://bitcointalk.org/index.php?topic=176489.80

Thnaks MtQuid for your code
newbie
Activity: 24
Merit: 0
@MtQuid

How far back does that 'historical' day pull from?  Since GoxTool started?

It's using the same history candles that prof7bit uses for the trade graph.  So it goes back a day in 15 minute blocks.
Though you can change the history_timeframe value in goxtool.ini to change the block size.
hero member
Activity: 560
Merit: 500
I am the one who knocks
@MtQuid

How far back does that 'historical' day pull from?  Since GoxTool started?
newbie
Activity: 24
Merit: 0
I'm not very interested by you "speaker" strategy ... for now... but maybe one day it could be useful...

I tried to play with a Python wrapper for TA-Lib
http://ta-lib.org/
http://mrjbq7.github.io/ta-lib/
this wrapper is total...
you just need to install cython, TA-Lib, and this wrapper


I just tried this out and wanted to plot the results so I dumped the data to some files and used Highstock.js to render the view.

Code:
import goxapi
from goxapi import OHLCV
import datetime
import numpy as np
import talib
import json

class OHLCV_Encoder(json.JSONEncoder):
    """JSONEncoder for class OHLCV()"""
    def default(self, obj):
        if isinstance(obj, OHLCV):
            return [obj.tim, obj.opn, obj.hig, obj.low, obj.cls, obj.vol]
        else:
            return json.JSONEncoder.default(self, obj)

class Strategy(goxapi.BaseObject):
    ........
    ........

    def slot_keypress(self, gox, (key)):
        if key == ord("t"):
            """testing talib"""       
            candles = []

            for c in reversed(self.gox.history.candles):
                candles.append(
                    OHLCV(
                        # fix time for HighStock.js JSON import
                        c.tim * 1000,
                        # adjust values to be human readable
                        goxapi.int2float(c.opn, self.gox.currency),
                        goxapi.int2float(c.hig, self.gox.currency),
                        goxapi.int2float(c.low, self.gox.currency),
                        goxapi.int2float(c.cls, self.gox.currency),
                        goxapi.int2float(c.vol, "BTC")
                    )
                ) 

            self.debug("Test history with %d candles" % len(candles))               

            rng = range(len(candles))
            iterable = (candles[i].cls for i in rng)
            a_cls = np.fromiter(iterable, np.float)
                                 
            iterable = (candles[i].tim for i in rng)
            a_tim = np.fromiter(iterable, np.int64)


            a_sma = talib.SMA(a_cls, 10)
            a_wma = talib.WMA(a_cls, 25)

            # create json compatible with HighStock.js
            with open("talib_ohlcv.json", 'w') as outfile:                   
                json.dump(candles, outfile, cls = OHLCV_Encoder)
            with open("talib_sma.json", 'w') as outfile:
                # first 10 elements contain Nan
                json.dump(np.dstack((a_tim[10:], a_sma[10:])).tolist()[0], outfile)
            with open("talib_wma.json", 'w') as outfile:
                # first 25 elements contain Nan
                json.dump(np.dstack((a_tim[25:], a_wma[25:])).tolist()[0], outfile)

with

Code:




BTC Highstock - Goxtool/TA-Lib










Serve index.html up with
Code:
python -m SimpleHTTPServer 8080

EDIT:  Using close instead of open for EMAs and added python server example
hero member
Activity: 560
Merit: 500
I am the one who knocks
They are your own orders. If you zoom out the orderbook (- key) you will see them there too
Oh, awesome!  Thanks!
hero member
Activity: 938
Merit: 500
https://youengine.io/
Thanks prof7bit!

Question:  what do the yellow Os indicate?


They are your own orders. If you zoom out the orderbook (- key) you will see them there too
hero member
Activity: 560
Merit: 500
I am the one who knocks
Thanks prof7bit!

Question:  what do the yellow Os indicate?

Pages:
Jump to: