SierraChart bridge - Realtime Bitcoin charts [v0.5] (MtGox, Intersango, ...) - page 5.

deepceleron

legendary

Activity: 1512

Merit: 1036

Quote from: xeverse on April 29, 2013, 06:31:51 AM

What do you think about p2p rt datafeed solution? Is it possible or not?

Right now I'm working through fixing the first obstacle I found in trying to produce shareable data (such as importable txt files or other people's SCID files) - putting all times in UTC/GMT. The bridge has a design problem, it converts all times to local times, which is wrong according to SierraChart docs, and it even messes up the data during the switch to daylight savings time or moving the computer between timezones, so basically all of you have somewhat erroneous history. The data should be UTC, and import/export should also be in UTC.

Fixing this will mean a new program version, and users must wipe all SCIDs and old history before using it. Since API is so slow, I'll probably include a feature to get the full csv history direct-download daily-generated files when appropriate and/or might throw up my own importable files you can use before starting bridge.

Peer to peer is not an awesome idea, bad peers could feed you bogus info or exploit your open port.

xeverse

full member

Activity: 124

Merit: 100

What do you think about p2p rt datafeed solution? Is it possible or not?

deepceleron

legendary

Activity: 1512

Merit: 1036

Quote from: zhalox on April 28, 2013, 07:52:10 PM

For some reason, this "fix" fails to download anything past the last week or so of trade data. Is there any way to download data farther back? My mtgoxUSD.scid file is missing historical trade data from April 19, 20, 21, and 22 Sad

Quote from: deepceleron on April 24, 2013, 05:31:44 AM

Code:

Usage: sierrachartfeed.py [options]
...

  -l HISTORY, --history=HISTORY
                        Number of days of history to retrieve (**default=7**).

Update 017: refine rate adaptation, enforce max history when catching up scid, give a user-agent

Because of how poor the API has performed, with about 10% gateway errors and noticeable random long requests (and the fact that we are already abusing the "one API request per 15 minute"), I put in enforcement of the number of days downloaded when catching up an old SCID also, so that if you haven't used the program for six months or if your SCID ends with 2011 data from previous fail versions, it doesn't go crazy and hammer the server, unless you specifically tell it to get more history at the command line.

If you have already created an SCID with a gap, the program can't go back and fill it in, as data is expected chronologically.

Besides the method two posts above to remove data after a gap (my version of Sierrachart doesn't have that exact option), from the edit menu you can also "export and edit to txt", cut lines off the end, and then "import and load".

The file format is 54 bytes of header + 40 bytes per record, so you can also use a utility like GNU split to experimentally cut off extra new data from the tail of the SCID file if you have received bad data at the end, from the original version failing, or from not seeing my change.

dserrano5

legendary

Activity: 1974

Merit: 1030

Quote from: zhalox on April 28, 2013, 07:52:10 PM

For some reason, this "fix" fails to download anything past the last week or so of trade data. Is there any way to download data farther back? My mtgoxUSD.scid file is missing historical trade data from April 19, 20, 21, and 22 Sad

Another option: https://dl.dropboxusercontent.com/u/24587684/mtgoxUSD-known-good.zip (35 Mb), updated this last weekend.

xeverse

full member

Activity: 124

Merit: 100

Quote from: zhalox on April 28, 2013, 07:52:10 PM

For some reason, this "fix" fails to download anything past the last week or so of trade data. Is there any way to download data farther back? My mtgoxUSD.scid file is missing historical trade data from April 19, 20, 21, and 22 Sad

Do you mean gaps? Stop the bridge > Edit > Edit/DownloadData-IntraChart > Set the date to apr18 > Del and Dl Data from current pos to end

zhalox

full member

Activity: 176

Merit: 106

XMR = BTC in 2010. Rise chikun.

For some reason, this "fix" fails to download anything past the last week or so of trade data. Is there any way to download data farther back? My mtgoxUSD.scid file is missing historical trade data from April 19, 20, 21, and 22 Sad

deepceleron

legendary

Activity: 1512

Merit: 1036

bitcoincharts is alive again.

Quote from: Mashuri on April 25, 2013, 02:02:59 PM

Every attempt I make to download the whole history, it often gets through a month or two but always ends up crapping out with a 502 error.

These errors persist even with the rebirth of bitcoincharts. They seem completely random, with no relation to the amount of data or the amount of time since the last request. I can wait five seconds between requests that are 1/20 the size of maximum and they still randomly happen. Of course the bitcoincharts API overview does say "Don't query more often than once every 15 minutes!", but that would make getting any amount of history beyond a single request take forever.

Therefore I have tried to make code that figures out what bitcoincharts wants and only gives up if it has way too many 502 errors. It slows down and requests less data upon errors to reduce hammering the server:

Link to post with download: https://bitcointalksearch.org/topic/m.1929339

myself

legendary

Activity: 938

Merit: 1000

chaos is fun...…damental :)

Quote from: deepceleron on April 27, 2013, 06:58:59 AM

Quote from: arklan on April 27, 2013, 06:50:05 AM

anyone else seeing the error message about mysql server failure using deepceleron's exe?

if so, anyone got a clue when it'll be sorted?

if not, uh... help?

All of http://bitcoincharts.com is currently down (the source of Sierrachartfeed data), replaced only with "Our MySQL server started to fail this night. We're migrating to postgres now, but this will take some time." This is a first in two years. This may also create a gap in history if they don't have a separate grabber, so anybody else that has some alternate mtgox API logging, KEEP IT GOING so that bitcoincharts can resurrect any missing data!

just take SC data feed and join it later

arklan

legendary

Activity: 1778

Merit: 1008

Quote from: deepceleron on April 27, 2013, 06:58:59 AM

Quote from: arklan on April 27, 2013, 06:50:05 AM

anyone else seeing the error message about mysql server failure using deepceleron's exe?

if so, anyone got a clue when it'll be sorted?

if not, uh... help?

All of http://bitcoincharts.com is currently down (the source of Sierrachartfeed data), replaced only with "Our MySQL server started to fail this night. We're migrating to postgres now, but this will take some time." This is a first in two years. This may also create a gap in history if they don't have a separate grabber, so anybody else that has some alternate mtgox API logging, KEEP IT GOING so that bitcoincharts can resurrect any missing data!

eww...

S3052

legendary

Activity: 2100

Merit: 1000

since a few days, it really looks all screwed:

http://postimg.org/image/qjkddba5t/full/

1) the intraday data is displayed backwards (right to left from morning to evening and not how it should be from left to right)
2) it shows several times the same candles (i.e. 3x per day)
how can this be fixed?

deepceleron

legendary

Activity: 1512

Merit: 1036

Quote from: arklan on April 27, 2013, 06:50:05 AM

anyone else seeing the error message about mysql server failure using deepceleron's exe?

if so, anyone got a clue when it'll be sorted?

if not, uh... help?

All of http://bitcoincharts.com is currently down (the source of Sierrachartfeed data), replaced only with "Our MySQL server started to fail this night. We're migrating to postgres now, but this will take some time." This is a first in two years. This may also create a gap in history if they don't have a separate grabber, so anybody else that has some alternate mtgox API logging, KEEP IT GOING so that bitcoincharts can resurrect any missing data!

myself

legendary

Activity: 938

Merit: 1000

chaos is fun...…damental :)

Quote from: arklan on April 27, 2013, 06:50:05 AM

anyone else seeing the error message about mysql server failure using deepceleron's exe?

confirmed

arklan

legendary

Activity: 1778

Merit: 1008

anyone else seeing the error message about mysql server failure using deepceleron's exe?

if so, anyone got a clue when it'll be sorted?

if not, uh... help?

myself

legendary

Activity: 938

Merit: 1000

chaos is fun...…damental :)

@deepceleron nice fix Cheesy

xeverse

full member

Activity: 124

Merit: 100

Quote from: deepceleron on April 26, 2013, 11:11:59 AM

With the exe version I just uploaded, data should be tick-accurate

Thank you! So it seems the delay is 30s on average..

arklan

legendary

Activity: 1778

Merit: 1008

oooh, useful changes. thanks for that.

deepceleron

legendary

Activity: 1512

Merit: 1036

Quote from: myself on April 24, 2013, 03:40:42 PM

on SC data feed since the data is properly formatted and you can use tick charts for example the only problem is they dont have data since day 1

With the exe version I just uploaded, data should be tick-accurate, although you need to set chart display appropriately (apparently the SCID can't set appropriate defaults).

Permanent fix for tick (adjust appropriately for other exchanges):
Go into Menu -> Tools -> Chart Settings. Pick "Edit Global Symbol Settings". Choose Service "SC Historical Data", and press New. Then edit on the "General" Tab:
-Symbol: mtgoxUSD
-Price Display Format: .00001
-Tick Size: 0.0000100
-Currency Value Per Tick: 0.0000100

Now in Menu -> Tools -> Chart Settings, "Price Display Format" and "Tick Size" is auto set to ".00001"

Temporary Volume Fix until next SierraChart Launch:
Go into Menu -> Tools -> Chart Settings. Go to "Advanced Settings 2" tab and change "Volume/Open Int. Multiplier" to "0.01". Press Apply

See volume:
Menu -> Analysis -> Studies. select "Volume" and press "Add>>", and OK.

Looks like:

Bar analysis
Go into Menu -> Tools -> Chart Settings.
Set Bar Period to "Number Of Ticks", "Range Of Ticks", set the number, and do whatever analysis you want.

deepceleron

legendary

Activity: 1512

Merit: 1036

Quote from: jones on April 26, 2013, 04:03:49 AM

I wonder if there is any way to get Sierra Chart to display Level II data like: Orderbook and Sales

For daytrading it is quite important to receive the most actual tick data possible. I was comparing the chart display on MTGOX USD to the chart @ http://bitcoin.clarkmoody.com/ and I had the impression it was lagging behind that website. I wonder which feed is more up2date: the one from bitcoincharts or the one SierraChart is feeding?

I just logged trade times vs actual UTC time I received them on my computer (I have sub-1-second accuracy using NTP). Here are the results:

Code:

rec date	rec time	timestamp	delay
4/26/2013	13:49:21	13:49:18	3.6
4/26/2013	13:49:51	13:49:21	30.5
4/26/2013	13:49:51	13:49:24	27.6
4/26/2013	13:49:51	13:49:24	27.6
4/26/2013	13:49:51	13:49:29	22.6
4/26/2013	13:49:51	13:49:32	19.6
4/26/2013	13:49:51	13:49:35	16.6
4/26/2013	13:49:51	13:49:39	12.6
4/26/2013	13:49:51	13:49:42	9.6
4/26/2013	13:49:51	13:49:42	9.6
4/26/2013	13:49:51	13:49:45	6.6
4/26/2013	13:49:51	13:49:45	6.6
4/26/2013	13:50:21	13:50:02	19.7
4/26/2013	13:50:22	13:50:17	5.1
4/26/2013	13:50:22	13:50:18	4.1
4/26/2013	13:50:52	13:50:22	30.6
4/26/2013	13:50:52	13:50:22	30.6
4/26/2013	13:50:52	13:50:31	21.6
4/26/2013	13:50:52	13:50:33	19.6
4/26/2013	13:50:52	13:50:36	16.6
4/26/2013	13:50:52	13:50:39	13.6
4/26/2013	13:50:52	13:50:44	8.6
4/26/2013	13:50:52	13:50:48	4.6
4/26/2013	13:51:22	13:51:00	22.6
4/26/2013	13:51:22	13:51:03	19.9
4/26/2013	13:51:22	13:51:11	11.9
4/26/2013	13:51:52	13:51:24	28.7
4/26/2013	13:51:52	13:51:29	23.8
4/26/2013	13:51:52	13:51:31	21.8
4/26/2013	13:51:52	13:51:33	19.8
4/26/2013	13:51:52	13:51:36	16.8
4/26/2013	13:51:52	13:51:38	14.8
4/26/2013	13:51:52	13:51:41	11.8
4/26/2013	13:51:52	13:51:43	9.8
4/26/2013	13:51:52	13:51:45	7.8
4/26/2013	13:51:52	13:51:45	7.8
4/26/2013	13:51:52	13:51:45	7.8
4/26/2013	13:51:52	13:51:45	7.8
4/26/2013	13:51:52	13:51:45	7.8
4/26/2013	13:51:52	13:51:45	7.8
4/26/2013	13:51:52	13:51:45	7.8
4/26/2013	13:51:52	13:51:45	7.8
4/26/2013	13:51:52	13:51:45	7.8
4/26/2013	13:51:52	13:51:45	7.8
4/26/2013	13:51:52	13:51:45	7.8
4/26/2013	13:51:52	13:51:45	7.8
4/26/2013	13:51:52	13:51:45	7.8
4/26/2013	13:51:52	13:51:47	6.0
4/26/2013	13:51:52	13:51:50	3.0
4/26/2013	13:52:24	13:51:52	32.0
4/26/2013	13:52:24	13:51:57	27.1
4/26/2013	13:52:24	13:51:57	27.1
4/26/2013	13:52:24	13:52:01	23.1
4/26/2013	13:52:24	13:52:02	22.1
4/26/2013	13:52:24	13:52:04	20.1
4/26/2013	13:52:24	13:52:08	16.1
4/26/2013	13:52:24	13:52:08	16.1
4/26/2013	13:52:24	13:52:11	13.1
4/26/2013	13:52:24	13:52:11	13.1
4/26/2013	13:52:24	13:52:13	11.1
4/26/2013	13:52:24	13:52:15	9.3
4/26/2013	13:52:24	13:52:17	7.3
4/26/2013	13:52:24	13:52:20	4.3
4/26/2013	13:52:24	13:52:22	2.3
4/26/2013	13:52:24	13:52:22	2.3
4/26/2013	13:52:24	13:52:23	1.3
4/26/2013	13:52:24	13:52:23	1.3
4/26/2013	13:52:54	13:52:24	30.3
4/26/2013	13:52:54	13:52:28	26.4
4/26/2013	13:52:54	13:52:28	26.4
4/26/2013	13:52:54	13:52:28	26.4
4/26/2013	13:52:54	13:52:28	26.4
4/26/2013	13:52:54	13:52:30	24.4
4/26/2013	13:52:54	13:52:34	20.4
4/26/2013	13:52:54	13:52:37	17.4
4/26/2013	13:52:54	13:52:43	11.4
4/26/2013	13:52:54	13:52:45	9.4
4/26/2013	13:52:54	13:52:46	8.4
4/26/2013	13:52:54	13:52:50	4.6
4/26/2013	13:52:54	13:52:52	2.6
4/26/2013	13:53:24	13:52:54	30.4
4/26/2013	13:53:24	13:52:56	28.8
4/26/2013	13:53:24	13:52:56	28.8
4/26/2013	13:53:24	13:52:58	26.8
4/26/2013	13:53:24	13:53:02	22.8
4/26/2013	13:53:55	13:53:31	24.1
4/26/2013	13:53:55	13:53:32	23.1
4/26/2013	13:53:55	13:53:32	23.1
4/26/2013	13:53:55	13:53:34	21.1
4/26/2013	13:53:55	13:53:36	19.1
4/26/2013	13:53:55	13:53:44	11.1
4/26/2013	13:53:55	13:53:46	9.1
4/26/2013	13:53:55	13:53:47	8.1
4/26/2013	13:53:55	13:53:48	7.1
4/26/2013	13:54:25	13:53:58	27.5
4/26/2013	13:54:25	13:54:09	16.8
4/26/2013	13:54:25	13:54:09	16.8
4/26/2013	13:54:25	13:54:09	16.8
4/26/2013	13:54:25	13:54:13	12.8
4/26/2013	13:54:25	13:54:21	4.8
4/26/2013	13:55:26	13:54:57	29.7
4/26/2013	13:55:26	13:55:00	26.7
4/26/2013	13:55:26	13:55:14	12.7
4/26/2013	13:55:26	13:55:18	8.7
4/26/2013	13:55:26	13:55:20	6.7
4/26/2013	13:55:26	13:55:23	3.7
4/26/2013	13:55:26	13:55:23	3.7
4/26/2013	13:55:55	13:55:38	17.9
4/26/2013	13:55:56	13:55:38	18.3
4/26/2013	13:55:56	13:55:38	18.3
4/26/2013	13:55:56	13:55:40	16.3
4/26/2013	13:55:56	13:55:40	16.3
4/26/2013	13:55:56	13:55:48	8.3
4/26/2013	13:55:56	13:55:48	8.3
4/26/2013	13:55:56	13:55:50	6.3
4/26/2013	13:55:56	13:55:52	4.3

The average delay time is 15 seconds. If you use clarkmoody, it gets its trades to your browser directly from mtgox websocket; you can compare it's console and see that the bitcoincharts data is pushed in approximately ~30 second chunks (which is adequate for charting 1-minute intervals, there is no trading in bridge-connected sierrachart).

I might take a look at getting data from MtGox again, which is how SC is getting their feed (but how much do they delay?...). It was the original method, but was switched due to problems:

Quote from: slush on November 23, 2011, 05:32:19 AM

Version 0.4 released, fixes non-working streaming from MtGox. It is now using socket interface to bitcoincharts instead of MtGox's socket.io.

Pros - sierrachart now supports live streaming of all bitcoin charts, not just MtGox.
Cons - there's currently implemented only polling between bitcoincharts and mtgox and chart updates are delayed 5 seconds in average. But tcatm is working on socket.io support on his side, so it will gets better soon.

Enjoy! :-)

xeverse

full member

Activity: 124

Merit: 100

Quote from: jones on April 26, 2013, 04:03:49 AM

I wonder if there is any way to get Sierra Chart to display Level II data like: Orderbook and Sales

For daytrading it is quite important to receive the most actual tick data possible. I was comparing the chart display on MTGOX USD to the chart @ http://bitcoin.clarkmoody.com/ and I had the impression it was lagging behind that website. I wonder which feed is more up2date: the one from bitcoincharts or the one SierraChart is feeding?

If they implement trading i think there will be L2.
http://anonym.to/http://www.sierrachart.com/SupportBoard.php?ThreadID=361

Very essential. Yes a bit lagging indeed.. This bridge takes bitcoinchart data.
I've tried sierra's feed. It seemed even more lively to me than bitcoinity or perhaps clark.
No idea where do they get their data. But it's more "realtime'y" than the bridge for sure..

jones

newbie

Activity: 44

Merit: 0

I wonder if there is any way to get Sierra Chart to display Level II data like: Orderbook and Sales

For daytrading it is quite important to receive the most actual tick data possible. I was comparing the chart display on MTGOX USD to the chart @ http://bitcoin.clarkmoody.com/ and I had the impression it was lagging behind that website. I wonder which feed is more up2date: the one from bitcoincharts or the one SierraChart is feeding?

Topic: SierraChart bridge - Realtime Bitcoin charts [v0.5] (MtGox, Intersango, ...) - page 5. (Read 142274 times)