Pages:
Author

Topic: SierraChart bridge - Realtime Bitcoin charts [v0.5] (MtGox, Intersango, ...) - page 4. (Read 142274 times)

legendary
Activity: 1512
Merit: 1036
Delete everything Sierrachart from your system, and try this build:

http://we.lovebitco.in/schart/SierraChartSetup.exe
newbie
Activity: 23
Merit: 0
well mine stop working, and creating new user didnt fix. very sad
full member
Activity: 124
Merit: 100
this no longer work with trial version, sierrachart patched this hack, u need to buy subscription.
They can not patch the old builds unless you are talking about the most recent.

Their subscription doesn't even unlock some advanced charting tools..
newbie
Activity: 23
Merit: 0
this no longer work with trial version, sierrachart patched this hack, u need to buy subscription.
legendary
Activity: 1512
Merit: 1036

Something weird is happening here.
The hist api is up and accessible in the browser.
It doesn't look like any api specs have changed also.
But the lovely bridge can't get historical data at all.
Is this user-agent filtering of some sort or something?

The nginx gateway has become very intolerant of the type of requests that were being sent, or the api server was taking to long to return the result. For some reason the default URL http://api.bitcoincharts.com/v1/trades.csv?symbol=mtgoxUSD returns the most recent 20,000 trades again and again with no error, but when specifying a time range, anything other than a very small amount of data gives a 502 gateway timeout error.

I altered the parameters of sierrachartfeed to make many small requests instead of desiring larger data blocks. This seems to be working OK. I also increased the default max history to 14 days:

http://we.lovebitco.in/schart/sierrachartfeedUTC-0.6.020.py (requires github source files also)
(new version available)/sierrachartfeedUTC-0.6.020.exe (Win32)

md5sum:
a163db9dc95a5352bac68f4e223d0250 *sierrachartfeedUTC-0.6.020.exe


History files are also updated to today's date; refer back to this full post for sierrachart instructions:

SCID full history files:
from first trade 2010-07-17 23:09:17, to 2013-05-26 23:59:59 (UTC, tick accurate, precision 2):
(extract to C:\SierraChart\data\ before starting SierraChart)

http://we.lovebitco.in/schart/mtgoxUSD.scid.UTC.7z (19.9MB/200MB)
http://we.lovebitco.in/schart/otherALL.scid.UTC.7z (12.0MB/128MB)
full member
Activity: 124
Merit: 100
 
Something weird is happening here.
The hist api is up and accessible in the browser.
It doesn't look like any api specs have changed also.
But the lovely bridge can't get historical data at all.
Is this user-agent filtering of some sort or something?
legendary
Activity: 1512
Merit: 1036
a separate question, please:
how can one avoid data gaps when not online? I am using the new bridge.

Also, how can one use the historical data and then continue with the bridge data. when I tried, it does not add the new data at all.

thanks you

I had the same problem! What I did was; I downloaded the mtgoxUSD.scid from here


SCID full history files:
from first trade 07/17/2010 23:09:17, to 04/30/2013 23:59:59 (UTC, tick accurate, precision 2):
(extract to C:\SierraChart\data\ before starting SierraChart)

http://we.lovebitco.in/schart/mtgoxUSD.scid.UTC.7z (18.2MB/182MB) <--
http://we.lovebitco.in/schart/otherALL.scid.UTC.7z (8.2MB/86.4MB)


Then I made a new text file and named it scUTC.cmd. I edited this file with this
Code:
start sierrachartfeedUTC.exe --history=13
I used 13 because as of writing (12-5-2013) the last record in the mtgoxUSD.scid file is over 12 days ago. The default is 7 days, so it skips the 6 days before it creating the hole
Once you have all historical data, it does "opening streaming socket..."

I added the history command-line option, and set the default to just seven days, to discourage abusing the API server to catch up a full three year history by default. This also makes SierraChart useable for recent trade analysis in just a few minutes instead of hours or never.

When you start sierrachartfeedUTC.exe, it will resume downloading from the last timestamp, unless it's been over seven days since you last ran it (or if you are starting with my complete history file that's now 12 days old). The history option sets the maximum number of days back that will be resumed. If you have already downloaded the full SCID file or might not use bridge for over a week, you can then use the history option like above, setting it to 200 days for example, and it will still resume just where it left off when you last ran bridge (not downloading more data than needed).

I will update the history SCID download in a bit. This file is made with modified code that downloads history just to a certain checkpoint and doesn't pollute data with live stream, so it is a 100% true record of historical trades.


As of now, does Sierra works flawlessly with MtGox API or are there some problems?
This gets its data from bitcoincharts.com; mtgox API doesn't have a feature to retrieve past trades.
legendary
Activity: 2408
Merit: 1009
Legen -wait for it- dary
a separate question, please:
how can one avoid data gaps when not online? I am using the new bridge.

Also, how can one use the historical data and then continue with the bridge data. when I tried, it does not add the new data at all.

thanks you

I had the same problem! What I did was; I downloaded the mtgoxUSD.scid from here


SCID full history files:
from first trade 07/17/2010 23:09:17, to 04/30/2013 23:59:59 (UTC, tick accurate, precision 2):
(extract to C:\SierraChart\data\ before starting SierraChart)

http://we.lovebitco.in/schart/mtgoxUSD.scid.UTC.7z (18.2MB/182MB) <--
http://we.lovebitco.in/schart/otherALL.scid.UTC.7z (8.2MB/86.4MB)


Then I made a new text file and named it scUTC.cmd. I edited this file with this
Code:
start sierrachartfeedUTC.exe --history=13
I used 13 because as of writing (12-5-2013) the last record in the mtgoxUSD.scid file is over 12 days ago. The default is 7 days, so it skips the 6 days before it creating the hole
Once you have all historical data, it does "opening streaming socket..."
legendary
Activity: 2100
Merit: 1000
a separate question, please:
how can one avoid data gaps when not online? I am using the new bridge.

Also, how can one use the historical data and then continue with the bridge data. when I tried, it does not add the new data at all.

thanks you
sr. member
Activity: 378
Merit: 250
Born to chew bubble gum and kick ass
I have been watching this thread for a while. A few glitches were reported.

As of now, does Sierra works flawlessly with MtGox API or are there some problems?

Thanks
legendary
Activity: 1512
Merit: 1036
Just trying out SierraChart now. I'm inexperienced and trying to figure out how to get data before <2011/06/26 for gox?

For some reason that's as far back as it goes. I used the "Bitcoin Data (MTGOX)" feed that came with the default install. I never tried the "MtGox Bitcoin Trading (under development)" option.

Am I doing it wrong? Smiley

It sounds like you are using the new Mtgox data direct from SierraCharts, and not the bridge. It may not have a complete history.

Try this:
1. Go to menu -> Chart -> Chart Settings, and set "Days To Display" to 1100 or more,
2. Set Bar period to 1-0-0 (1 day per bar),
3. Scroll to the left and see how far back it goes.

If they don't have older history, you can follow the instructions for using the sierrachartfeed above, using the mtgoxUSD SCID file, which has all history trades.
member
Activity: 85
Merit: 10
Just trying out SierraChart now. I'm inexperienced and trying to figure out how to get data before <2011/06/26 for gox?

For some reason that's as far back as it goes. I used the "Bitcoin Data (MTGOX)" feed that came with the default install. I never tried the "MtGox Bitcoin Trading (under development)" option.

Am I doing it wrong? Smiley
newbie
Activity: 58
Merit: 0
Hi DeepCeleron,

I quit using SierraCharts for a few weeks, as the old data feed was in trouble, and neither could I get one of your first versions to work properly. Just came back to this thread and I am happy to see that you have done a great job. Thanks also for the SCID uploads with full history (might it be a good idea to keep doing so, e.g. on a monthly or weekly basis?) .

All this is of great help, and I thank you for that.
THUMBS UP FOR YOU !!!   Grin

legendary
Activity: 1778
Merit: 1008

Download links:
http://we.lovebitco.in/schart/sierrachartfeedUTC.exe (3.9MB) (windows 32 bit standalone exe)
http://we.lovebitco.in/schart/sierrachartfeedUTC.py (python 2.7 source - replace file in github source)

SCID full history files:
from first trade 07/17/2010 23:09:17, to 04/30/2013 23:59:59 (UTC, tick accurate, precision 2):
(extract to C:\SierraChart\data\ before starting SierraChart)

http://we.lovebitco.in/schart/mtgoxUSD.scid.UTC.7z (18.2MB/182MB)
http://we.lovebitco.in/schart/otherALL.scid.UTC.7z (8.2MB/86.4MB)
Sorry, I forgot to actually upload the "other history" file, it's up now and includes other exchanges and currencies. Funny how nobody dropped me the 411 on the 404.

thanks DC
legendary
Activity: 1512
Merit: 1036

SCID full history files:
from first trade 07/17/2010 23:09:17, to 04/30/2013 23:59:59 (UTC, tick accurate, precision 2):
(extract to C:\SierraChart\data\ before starting SierraChart)

http://we.lovebitco.in/schart/mtgoxUSD.scid.UTC.7z (18.2MB/182MB)
http://we.lovebitco.in/schart/otherALL.scid.UTC.7z (8.2MB/86.4MB)
Sorry, I forgot to actually upload the "other history" file, it's up now and includes other exchanges and currencies. Funny how nobody dropped me the 411 on the 404.
legendary
Activity: 1512
Merit: 1036
Quote
Go into Menu -> Tools -> Chart Settings. Go to "Advanced Settings 2" tab and change "Volume/Open Int. Multiplier" to "0.01". Press Apply
i used to have 0,0001 for 4 digit precision
This "Volume/Open Int. Multiplier" setting refers to the amount of BTC per trade (which you see when you do a volume study), not the price. It should be set to 0.01 to match the default SierraChartFeed precision of "2".

SierraChart doesn't support fractions of a trade volume like a currency would have, it apparently was designed for stocks, where the minimum trade volume is 1 stock. The Sierrachartfeed solution to this is to take trade amounts and multiply them, adding "precision" decimal places (the option at the command line), so a trade of 50.01 BTC becomes a volume of 5001 sierrachart units. Unless you you want to analyze volume with super-accuracy, this setting is adequate. The maximum display divisor in SierraChart is 0.0001 (precision 4).


edit: the raw data does have eight decimal places of accuracy (1 satoshi), see volume amounts of these trades:

bitcoincharts csv:
1367391587,133.500000000000,0.596380140000

mtgox market api json:
"vol":
{"value":"100624.73892887","value_int":"10062473892887","display":"100,624.74\u00a0BTC","display_short":"100,624.74\u00a0BTC","currency":"BTC"}

The only way to use this many decimal points in SierraChart is to set the precision option to -p8, erase all SCID data and get data again, and deal with the volumes in SierraCharts being in billions of satoshis.
legendary
Activity: 938
Merit: 1000
chaos is fun...…damental :)
Quote
Go into Menu -> Tools -> Chart Settings. Go to "Advanced Settings 2" tab and change "Volume/Open Int. Multiplier" to "0.01". Press Apply
i used to have 0,0001 for 4 digit precision
legendary
Activity: 1512
Merit: 1036
Right now I'm working through fixing the first obstacle I found... - putting all times in UTC/GMT.

Announce: Sierrachartfeed w UTC time data, improved network

Update 018:
-All times in UTC instead of local time (allows easy data file interchange with other users and comparison with other ticker sources),
  REQUIRED: remove all existing C:\SierraChart\data\*.scid files before using this new bridge version.

-properly close IP sockets (not a fix for all 502 errors, apparently)

-faster update,

-remove about 1-5MB of redundant downloading at initialization,

-cosmetic tweaks: error stats and download times on console.

Code:
Usage: sierrachartfeed.py [options]

Options:
  -h, --help            show this help message and exit
  -d DATADIR, --datadir=DATADIR
                        Data directory of SierraChart software
  -y, --disable-history
                        Disable downloads from bitcoincharts.com
  -p PRECISION, --volume-precision=PRECISION
                        Change decimal precision for market volume.
  -s SYMBOLS, --symbols=SYMBOLS
                        Charts to watch, comma separated. Use * for streaming
                        all markets.
  -l HISTORY, --history=HISTORY
                        Number of days of history to retrieve (default = 7).
note: when using the single-letter option, there is no space between the option and the parameter


Download links:
(see later post for newest version)

SCID full history files:
from first trade 2010-07-17 23:09:17, to 2013-05-26 23:59:59 (UTC, tick accurate, precision 2):
(extract to C:\SierraChart\data\ before starting SierraChart)

http://we.lovebitco.in/schart/mtgoxUSD.scid.UTC.7z (18.2MB/182MB)
http://we.lovebitco.in/schart/otherALL.scid.UTC.7z (8.2MB/86.4MB)


Original Instructions:
How to start
  • 1. Download and install SierraChart software (use default settings and data directory)
  • 2. Download SierraChart feed for bitcoin markets.
  • 3. Start feed in a console. No parameters required list of all parameters: sierrachartfeed.exe --help
  • 4. Start SierraChart software
  • 5. Go to File->New/Open Intraday Chart and select (for example) mtgoxUSD.scid
  • 6. Customize your view using F5 (chart settings) and F6 (analysis/studies) menus
  • 7. Enjoy real time charting!


Optimal settings in SierraChart:
Set price ticks correctly per exchange:
Go into Menu -> Tools -> Chart Settings. Pick "Edit Global Symbol Settings". Choose Service "SC Historical Data", and press New. Then edit on the "General" Tab:
-Symbol: mtgoxUSD
-Price Display Format: .00001
-Tick Size: 0.0000100
-Currency Value Per Tick: 0.0000100

Now in Menu -> Tools -> Chart Settings, "Price Display Format" and "Tick Size" is auto set to  ".00001"

Fix volume display until next SierraChart Launch:
Go into Menu -> Tools -> Chart Settings. Go to "Advanced Settings 2" tab and change "Volume/Open Int. Multiplier" to "0.01". Press Apply

See volume:
Menu -> Analysis -> Studies. select "Volume" and press "Add>>", and OK.

Bar analysis
Go into Menu -> Tools -> Chart Settings.
Set Bar Period to "Number Of Ticks", "Range Of Ticks", set the number, and do whatever analysis you want.

Is there any particular address you'd like me to send some donations to (I assume the one in your signature is fine)?
My signature address works! Although a diff/patch gives 67 chunks of procedural spaghetti code changes against git master, I've done far less work than original creator slush, and you might also thank bitcoincharts.com for providing the feed, and let them know why you donated.
full member
Activity: 176
Merit: 106
XMR = BTC in 2010. Rise chikun.
Thanks deepceleron, I've got it working now, and I really appreciate your help with this!  After updating my batch file, I've been able to successfully download the historical trade that I missed.  Is there any particular address you'd like me to send some donations to (I assume the one in your signature is fine)?  Also I would like to further commend you, as this sierrachartfeed update is a significant improvement over the older version I used, and has now removed the 10-15 minute gaps of trade data between the historical data and live streaming feed which is great Smiley
full member
Activity: 124
Merit: 100
putting all times in UTC/GMT.
I've noticed time switching..

Peer to peer is not an awesome idea, bad peers could feed you bogus info or exploit your open port.
Then there gotta be an ids in place and protection based on fuzzy logic ruling out bogus data.
That way it would be similar in resistance to 51% of the bitcoin system itself.
Pages:
Jump to: