SierraChart feed/bridge reborn - Realtime Bitcoin charting - page 3.

movellan

full member

Activity: 347

Merit: 100

Quote from: deepceleron on February 18, 2014, 06:31:45 AM

Quote from: arklan on February 18, 2014, 02:10:04 AM

so i've just gottne back to my charting pc, and triedthe new exe. on launch it throws "the program can't start because python27.dll is missing from your computer."

i'm assuming i could just install python to fix that, but if i can avoid doing so, i prefer to keep a tidy ship.

Uploaded a new exe made the same way as before with dlls included.

Now that the mtgox feed no longer exists, will you be adding any other exchanges to the feed? TIA.

arklan

legendary

Activity: 1778

Merit: 1008

assuming you mean the download link in the first post - i tried it, no difference.

deepceleron

legendary

Activity: 1512

Merit: 1032

Quote from: arklan on February 18, 2014, 02:10:04 AM

so i've just gottne back to my charting pc, and triedthe new exe. on launch it throws "the program can't start because python27.dll is missing from your computer."

i'm assuming i could just install python to fix that, but if i can avoid doing so, i prefer to keep a tidy ship.

Uploaded a new exe made the same way as before with dlls included.

arklan

legendary

Activity: 1778

Merit: 1008

so i've just gottne back to my charting pc, and triedthe new exe. on launch it throws "the program can't start because python27.dll is missing from your computer."

i'm assuming i could just install python to fix that, but if i can avoid doing so, i prefer to keep a tidy ship.

deepceleron

legendary

Activity: 1512

Merit: 1032

Quote from: xeverse on February 15, 2014, 05:59:22 AM

*** Loading data file /xe/_4mkt/2data/bitstampUSD-p4.scid
importing /xe/_4mkt/2data/bitstampUSD.csv
- No CSV history file found for bitstampUSD, skipping
- (Download manually from http://api.bitcoincharts.com/v1/csv/)
- (unzip to /xe/_4mkt/2data/bitstampUSD.csv)

Request bitstampUSD from 2014-02-15 09:47:20
Got 2014-02-15 09:47:20Z to 2014-02-15 09:47:20Z: 1 trades (43 bytes in 0.208534 sec)

--------------------------------------------------------------------------------------

There is a bunch of these cvs.imported files instead in the dir.

Why they aren't recognized then?

If the csv files have been renamed by sierrachartfeed to ".imported", then all the data in them is already in the Sierrachart SCID file; the ".imported" files and ".csv.gz" files should be deleted after successful import (unless you have a specific reason you might use them again soon; they are obsolete after four days). You can ignore the "no CSV history file found" message - it is not an error in this case, just a normal procedural message.

You will get the big *danger* error message (like four posts above) when there is an actual problem that requires you to re-download a new CSV (because the bitcoincharts API only has five-day-old and newer data).

The csv files inside the gzips actually start with a period, like ".bitstampUSD.csv". I've done a few tricks to look for the csv either with or without a period at the start.

xeverse

full member

Activity: 124

Merit: 100

Quote from: arklan on February 15, 2014, 12:40:35 PM

That looks like the message DC was just taking about four csv files that are more then five days out of date.

Yes, but only these renamed files are like 10min out of sync max.
You don't experience csv.renamed files?
http://i.imgur.com/OClQvXs.png

arklan

legendary

Activity: 1778

Merit: 1008

Quote from: xeverse on February 15, 2014, 05:59:22 AM

*** Loading data file /xe/_4mkt/2data/bitstampUSD-p4.scid
importing /xe/_4mkt/2data/bitstampUSD.csv
- No CSV history file found for bitstampUSD, skipping
- (Download manually from http://api.bitcoincharts.com/v1/csv/)
- (unzip to /xe/_4mkt/2data/bitstampUSD.csv)

Request bitstampUSD from 2014-02-15 09:47:20
Got 2014-02-15 09:47:20Z to 2014-02-15 09:47:20Z: 1 trades (43 bytes in 0.208534 sec)

--------------------------------------------------------------------------------------

There is a bunch of these
cvs.imported files instead in the dir.

Why they aren't recognized then?

That looks like the message DC was just taking about four csv files that are more then five days out of date.

xeverse

full member

Activity: 124

Merit: 100

*** Loading data file /xe/_4mkt/2data/bitstampUSD-p4.scid
importing /xe/_4mkt/2data/bitstampUSD.csv
- No CSV history file found for bitstampUSD, skipping
- (Download manually from http://api.bitcoincharts.com/v1/csv/)
- (unzip to /xe/_4mkt/2data/bitstampUSD.csv)

Request bitstampUSD from 2014-02-15 09:47:20
Got 2014-02-15 09:47:20Z to 2014-02-15 09:47:20Z: 1 trades (43 bytes in 0.208534 sec)

--------------------------------------------------------------------------------------

There is a bunch of these
cvs.imported files instead in the dir.

Why they aren't recognized then?

deepceleron

legendary

Activity: 1512

Merit: 1032

Quote from: arklan on February 13, 2014, 02:12:38 PM

How often are the csvs updated? I'm away from my charting / mining rig.

nightly

arklan

legendary

Activity: 1778

Merit: 1008

How often are the csvs updated? I'm away from my charting / mining rig.

xeverse

full member

Activity: 124

Merit: 100

Thank You!

deepceleron

legendary

Activity: 1512

Merit: 1032

Okay, it's fixed again, discarding about 100 lines of new code I had written to auto-resume using CSV data. First post updated.

You can now manually download and extract the complete zipped history CSVs to the SierraChart data directory, but won't be prompted to do so if it is a new install or if you specify a new ticker symbol you previously hadn't downloaded. It will simply give you five days of history. You also won't have to get CSVs as long as you run sierrachartfeed and have been keeping your local ticker data updated.

If you have already run sierrachartfeed to download ticker data and are more than five days behind, the software can't automatically catch up. You will get a dire warning from sierrachartfeed that you need to re-download these CSVs manually:

> scfeed.exe -s mtgoxUSD *** Loading data file C:/SierraChart/Data/mtgoxUSD-p4.scid importing C:/SierraChart/Data/mtgoxUSD.csv - No CSV history file found for mtgoxUSD, skipping - (Download manually from http://api.bitcoincharts.com/v1/csv/) - (unzip to C:/SierraChart/Data/mtgoxUSD.csv) Request mtgoxUSD from 2014-01-10 11:49:29 *** Danger! *** Your saved mtgoxUSD history is 640 hours older than that available from history website *** Continuing download will create an unrecoverable gap in trade data *** Press [c] and [enter] if you want to continue: - For trades older than 5 days, manually download a new CSV file from http://api.bitcoincharts.com/v1/csv/

You can proceed by typing the letter "C" and pressing enter; in my example above, I just pressed enter, so the program quit and gave info.

Happy charting.

xeverse

full member

Activity: 124

Merit: 100

Is it possible to disable history and just have the live socket stream for now?

arklan

legendary

Activity: 1778

Merit: 1008

I was wondering what was up. Sigh. Good luck DC. If I could help in anyway I would...

deepceleron

legendary

Activity: 1512

Merit: 1032

Once again major unannounced API changes at bitcoincharts have broken software.

Sierrachartfeed gets CSV data from this URL (this is the only source for trades older than five days):

http://api.bitcoincharts.com/v1/csv/

As documented here, it was individual nightly CSV files containing all historical trades for each exchange, about 300MB files for the top exchanges. Sierrachartfeed maintains a local copy of the CSV and "resumes", adding new data to your existing CSV file.

Now the remote data files are zipped. That means that any previous CSV data that was downloaded is useless. The old file names have changed, the files are now, for example, "mtgoxUSD.csv.gz". This file cannot be resumed, as it is compressed data wrapped in a header and footer. I would need to write gzip-aware software that can strip out the gzipped stream from the current file and rebuild it with added data, and of course unzip to import the new trades. And that's if bitcoincharts doesn't screw up the data contents again - one bit in one early trade changing will change the entire gzipped archive.

I don't know if the proper response is to say fuck it and not use data from that site any more, or to show that regardless of their contempt for users and software developers, we can make it work.

K3n

newbie

Activity: 1

Merit: 0

I did as described in the top posts: downloaded program and the sierrachartfeed using the second link under subpoint sierrachartfeed.
The next step would be to start the sierrachartfeed. Here I am getting confused as there are 3 different files, when I exract the downoaded .zip file. If I start the application scfeed.exe the console opens up and closes immediatly to fast for me to read the text in there. then Microsoft Excel document pops u in the same folder, with the name mtgoxUSD but seems to be emtpy.

Can't do step 5 as the Data directory is empty.

And what am I to do with the posted script?

Any help is appreciated.

xeverse

full member

Activity: 124

Merit: 100

syncing /xe/sierra/data/mtgoxUSD.csv
URLError = Not Found
Traceback (most recent call last):
File "/xe/sierra/scfeed.py", line 739, in
csvupdater(csv_filename, s)
File "/xe/sierra/scfeed.py", line 385, in csvupdater
remotecsv = CSVrange(csvsymbol)
File "/xe/sierra/scfeed.py", line 303, in __init__
self.getsize(self.symbol)
File "/xe/sierra/scfeed.py", line 327, in getsize
raise Exception('URLError = ' + str(err.reason))
Exception: URLError = Not Found

arklan

legendary

Activity: 1778

Merit: 1008

right... so, installing ot a custom location is a bad idea... reinstalled to default, everything is fine.

arklan

legendary

Activity: 1778

Merit: 1008

Yea, I'm running it from my d drove instead of c, so I don't have to remember to re-download or save the data when I reinstall windows or such... I'll sorry it later. Not home now.

Thanks.

deepceleron

legendary

Activity: 1512

Merit: 1032

I'll guess the error looks like this:

>scfeed.exe -d d:/scfeed/data -slocalbtcUSD syncing d:/scfeed/data\localbtcUSD.csv Traceback (most recent call last): File "scfeed.py", line 739, in csvupdater(csv_filename, s) File "scfeed.py", line 380, in csvupdater csvfile = open(csvfilename, "wb") # write, binary IOError: [Errno 2] No such file or directory: 'd:/scfeed/data\\localbtcUSD.csv'

Sierrachartfeed doesn't attempt to make a directory, it will throw this error if the directory you specify or the default directory doesn't exist. You just need to create the directory. SierraChart should have already made the data directory, it is where the SCID files go; I haven't tried even running SierraChart with a non-default location (which is C:\SierraChart\Data).

If you use a directory that already exists, it works fine:

>scfeed.exe -d D:\Bitcoin\scfeed -slocalbtcUSD syncing D:\Bitcoin\scfeed\localbtcUSD.csv Updating CSV: bytes 0 to 1048575 of 3433481 Updating CSV: bytes 1048576 to 2097151 of 3433481 Updating CSV: bytes 2097152 to 3145727 of 3433481 Updating CSV: bytes 3145728 to 3433480 of 3433481 *** Loading data file D:\Bitcoin\scfeed\localbtcUSD-p4.scid Downloading historical data syncing D:\Bitcoin\scfeed\localbtcUSD.csv local CSV already up to date importing D:\Bitcoin\scfeed\localbtcUSD.csv adding D:\Bitcoin\scfeed\localbtcUSD.csv to SCID Request localbtcUSD from 2014-02-01 10:46:07 Got 2014-02-01 10:46:07Z to 2014-02-02 21:29:19Z: 650 trades (28057 bytes in 0.806000 sec) Adding 2014-02-01 11:21:36Z to 2014-02-02 21:29:19Z: 649 trades to SCID Opening live trade socket, wait........ 2014-02-02 21:45:02: 21:41:26 localbtcUSD 1000.0 0.48 2014-02-02 21:45:02: 21:37:16 localbtcUSD 894.52 0.5702 2014-02-02 21:45:02: 21:41:05 localbtcUSD 983.08 0.0997 2014-02-02 21:45:02: 21:32:21 localbtcUSD 759.81 0.1998 2014-02-02 21:45:02: 21:38:58 localbtcUSD 922.0 0.2223

Topic: SierraChart feed/bridge reborn - Realtime Bitcoin charting - page 3. (Read 10151 times)