so it updated the histories fine, but then when it should be converting the csv's to scid's, it error's out with "error 2: d:/sierrachart/data/bitstampusd.csv no csv exists" i'm paraphrasing, i couldn't copy and paste it for some reason, but the slashes WERE those ones. that's wrong isn't it?
any idea what's up? i miss this data feed...
oh, and per your suggestion i'm leaving btce off. just using stamp and gox.
Topic: SierraChart feed/bridge reborn - Realtime Bitcoin charting - page 4. (Read 10182 times)
btc-e seems to sometimes be available on bitcoincharts, and then the ticker symbol completely disappears from existence. Since I put this as one of the default exchanges, you should manually specify an exchange or a list of exchanges with the command-line option instead. If http://bitcoincharts.com/charts/btceUSD says "page not found", that means its still not fixed.
i just realized i hadn't followed through and updated - i don't see a way to disable the history grab, only the reverse.
i'm running the feed now to see if it will update the history and begin the live stream. it's got a ways to go though.
i'm running the feed now to see if it will update the history and begin the live stream. it's got a ways to go though.
Haven't had a chance to try it. Busy week. I'll check it when I ghetto home and let you know.
Any progress? My feed is not streaming the live trades, after doing a fresh download of the history.
Do I need to run with -h right now to get any data?
Do I need to run with -h right now to get any data?
I'm getting mtgox live socket trades right now, so that's working; this part, strangely, has never been a problem, even though it is documented nowhere except by use in sierrachartfeed and one other 3 year old trade software. Is it magically working again for you?
It looks like bitcoincharts hasn't re-scrambled existing CSV data since Dec 28, so I will create a new 7zipped CSV share of my own since I can make it 10% the size. I was quite frustrated by a lot of programming and testing being flushed by another change at bitcoincharts.
The code is intolerant of web site errors - this is by design so I could see exception errors when I rewrote this. I need to make downloading more error-tolerant, as I'm now getting timeouts and errors I previously didn't, like "Exception: URLError = [Errno 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond", even for some very small requests.
Any progress? My feed is not streaming the live trades, after doing a fresh download of the history.
Do I need to run with -h right now to get any data?
Do I need to run with -h right now to get any data?
There is no guarantee on the order of the data so you'd need to sort it in some cases. Sorting can be done in O(n log n) so it shouldn't be a huge problem. I'll consider pre-sorting on the server in the future but please don't rely on it!
The reason for the unsorted output is likely that I've switched to a database backend that can run things in parallel now or maybe it just went unnoticed until now for some other reason.
The reason for the unsorted output is likely that I've switched to a database backend that can run things in parallel now or maybe it just went unnoticed until now for some other reason.
I have identified the problem. The downloadable CSVs recently generated by bitcoincharts are corrupted.
On the left is the CSVs as they were up to Dec 24, on the right, download Dec 28, out of order trades have been written into the file. There's lots more of this in the file.
In the new CSV, the very first historic trade is missing, making the whole file different. Then things start going really wrong about 90% in, where at Wed, 11 Dec 2013 16:14:58 GMT, we start having chunks of trades from 1388094380- (Thu, 26 Dec 2013 21:46:20 GMT and on) randomly inserted into the file. As seen above at 11 Dec 2013 08:53:45, we have some 26 Dec 2013 16:42:24 trades inserted.
Now what? Will I need to anticipate new trades being written to the beginning of the file?? If you are 6 days behind, I have to get a year of data just to look for trades in the wrong spot?
On the left is the CSVs as they were up to Dec 24, on the right, download Dec 28, out of order trades have been written into the file. There's lots more of this in the file.
In the new CSV, the very first historic trade is missing, making the whole file different. Then things start going really wrong about 90% in, where at Wed, 11 Dec 2013 16:14:58 GMT, we start having chunks of trades from 1388094380- (Thu, 26 Dec 2013 21:46:20 GMT and on) randomly inserted into the file. As seen above at 11 Dec 2013 08:53:45, we have some 26 Dec 2013 16:42:24 trades inserted.
Now what? Will I need to anticipate new trades being written to the beginning of the file?? If you are 6 days behind, I have to get a year of data just to look for trades in the wrong spot?
Sorry for the delay, I was letting a day elapse to examine the state of CSV and discover what they are doing at bitcoincharts. I downloaded Dec-27 CSVs, and then now that newer nightly CSVs are published, attempted to synchronize again. Again the files have changed in a way where the previous history is not byte identical with either of the previous CSV downloads. I'll do a diff/compare to see what's being altered, but it's out of my control anyway.
To clarify, there are three sources of data to be used that are combined:
-Nightly CSVs: data since the start of time to about 2-26 hours ago,
-History API: maximum of five days of history available up to now,
-Socket stream: live trades
I will have to resort to the strategy I initially conceived but thought impractical, which is to not have a local copy, but to "explore" the current remote CSV by downloading ranges of bytes until I find a point not too far before the last SCID timestamp. If they keep changing the CSV, even indexing it locally won't work precisely. If your SCID is less than five days old, we don't need to hit the CSV data, but bytes ranges of CSV is faster than API calls, and should cause less server load.
To clarify, there are three sources of data to be used that are combined:
-Nightly CSVs: data since the start of time to about 2-26 hours ago,
-History API: maximum of five days of history available up to now,
-Socket stream: live trades
I will have to resort to the strategy I initially conceived but thought impractical, which is to not have a local copy, but to "explore" the current remote CSV by downloading ranges of bytes until I find a point not too far before the last SCID timestamp. If they keep changing the CSV, even indexing it locally won't work precisely. If your SCID is less than five days old, we don't need to hit the CSV data, but bytes ranges of CSV is faster than API calls, and should cause less server load.
Got an exception using the historical data provided for MTGox, was working fine yesterday:
Code:
syncing C:/SierraChart/Data/mtgoxUSD.csv
Updating CSV: bytes 312690830 to 313739405 of 316504899
312690798: ,670.000000000000,3.727747830000 ::: 00
1387445072,640.549000000000,0
Traceback (most recent call last):
File "scfeed.py", line 739, in
csvupdater(csv_filename, s)
File "scfeed.py", line 406, in csvupdater
raise Exception('Local and remote CSVs do not match')
Exception: Local and remote CSVs do not match
Yep, I got it also.
The updating method is based on the assumption that in the complete nightly CSV files, older trade data wouldn't be changing, there would just be new trades added to them. I, of course, don't just blindly add new data to your local CSV file, I get an overlapping range and make sure that we are appending to identical data. My checks caught the alteration.
the last 32 bytes of your local file ::: the bytes that are now being returned when re-checking that range
312690798: ,670.000000000000,3.727747830000 ::: 001387445072,640.549000000000,0
The problem here seems to be that bitcoincharts made CSVs with different old history than was previously downloaded (after several days of the nightly updates being the same and working correctly.) The files appear to have changed for other tickers too.
Sierrachartfeed will work again by deleting and letting it re-download the C:/SierraChart/Data/mtgoxUSD.csv file. On bitcoincharts, this is the only file and method to get trades that are older than five days, so if the CSV data continues to change in this way that the file download can't be "resumed", I'll need to think of different ways to synchronize this data and resume from the last time scfeed was run.
Got an exception using the historical data provided for MTGox, was working fine yesterday:
Code:
syncing C:/SierraChart/Data/mtgoxUSD.csv
Updating CSV: bytes 312690830 to 313739405 of 316504899
312690798: ,670.000000000000,3.727747830000 ::: 00
1387445072,640.549000000000,0
Traceback (most recent call last):
File "scfeed.py", line 739, in
csvupdater(csv_filename, s)
File "scfeed.py", line 406, in csvupdater
raise Exception('Local and remote CSVs do not match')
Exception: Local and remote CSVs do not match
wonderful. thanks!
SierraChart setup tips:
Set price ticks correctly per exchange:
Go into Menu -> Tools -> Chart Settings. Pick "Edit Global Symbol Settings". Choose Service "SC Historical Data", and press New. Then edit on the "General" Tab:
-Symbol: mtgoxUSD
-Price Display Format: .00001
-Tick Size: 0.0000100
-Currency Value Per Tick: 0.0000100
Now in Menu -> Tools -> Chart Settings, "Price Display Format" and "Tick Size" is auto set to ".00001"
Fix volume display until next SierraChart Launch:
Go into Menu -> Tools -> Chart Settings. Go to "Advanced Settings 2" tab and change "Volume/Open Int. Multiplier" to "0.0001". Press Apply
See volume:
Menu -> Analysis -> Studies. select "Volume" and press "Add>>", set volume format to .0001, and OK.
Bar analysis
Go into Menu -> Tools -> Chart Settings.
Set Bar Period to "Number Of Ticks", "Range Of Ticks", set the number, and do whatever analysis you want.
Set price ticks correctly per exchange:
Go into Menu -> Tools -> Chart Settings. Pick "Edit Global Symbol Settings". Choose Service "SC Historical Data", and press New. Then edit on the "General" Tab:
-Symbol: mtgoxUSD
-Price Display Format: .00001
-Tick Size: 0.0000100
-Currency Value Per Tick: 0.0000100
Now in Menu -> Tools -> Chart Settings, "Price Display Format" and "Tick Size" is auto set to ".00001"
Fix volume display until next SierraChart Launch:
Go into Menu -> Tools -> Chart Settings. Go to "Advanced Settings 2" tab and change "Volume/Open Int. Multiplier" to "0.0001". Press Apply
See volume:
Menu -> Analysis -> Studies. select "Volume" and press "Add>>", set volume format to .0001, and OK.
Bar analysis
Go into Menu -> Tools -> Chart Settings.
Set Bar Period to "Number Of Ticks", "Range Of Ticks", set the number, and do whatever analysis you want.
ANN: New version of Sierrachartfeed and direct-download history files, working again (and fixed again Feb 10 2014)
Original post with details of what this software is, read first:
Sierrachartfeed gets data from bitcoincharts.com for multiple exchanges. Each time they change some API "feature", it seems to break other people's software. This is thefourth fifth "rebirth" of working software I've written since slush gave up maintenance of sierrachartfeed.
note: when using the single-letter option, there is no space between the option and the parameter
Download links:
For history older than five days, before first running sierrachartfeed
The normal Bitcoincharts API only has five days of data available. If you have never run sierrachartfeed for a particular exchange ticker before, it will only get the last five days. However, if you want all trade data from the beginning of time, you can download some big files first:
You will also need to re-download a new CSV if it has been over five days since you last ran sierrachartfeed.
The SCID filenames are different from previous versions, to avoid using or overwriting obsolete data. New filenames are of the format mtgoxUSD-p4.scid (where p4 indicates precision 4).
Changelog
(0.7.1 - 2014-02-10)
(0.7.0 - 2013-12-19)
Original post with details of what this software is, read first:
SierraChart is desktop application providing a professional Trading, Real-time and Historical Charting, and Technical Analysis platform for the financial markets. It's not so intuitive as other platforms, but once you learn how it works (and remember few shortcuts), it is very strong tool for professional technical analysis. Because of lacking real trading tools for Bitcoin economy, I decided to write simple Bitcoin markets->Sierrachart bridge, which enable real time Bitcoin charting in Sierrachart.
How to start
How to start
- 1. Download and install SierraChart software (use default settings and data directory)
- 2. Download SierraChartfeed, extract zip to any program directory.
- 3. Start feed in a console. No parameters required for mtgox (list of all parameters: sierrachartfeed.exe --help)
- 4. Start SierraChart software
- 5. Go to File->New/Open Intraday Chart and select (for example) mtgoxUSD.scid
- 6. Customize your view using F5 (chart settings) and F6 (analysis/studies) menus
- 7. Enjoy real time charting!
Sierrachartfeed gets data from bitcoincharts.com for multiple exchanges. Each time they change some API "feature", it seems to break other people's software. This is the
Code:
Usage: scfeed.py [options]
Options:
-h, --help show this help message and exit
-t, --tickers List known/current ticker symbols and exit
-z, --disable-live Disable live socket, update history only
-d DATADIR, --datadir=DATADIR
SierraChart data directory (default =
C:/SierraChart/Data/)
-s SYMBOLS, --symbols=SYMBOLS
Ticker symbols to download, comma separated with no
spaces; 'a' for all (default =
-smtgoxUSD,btceUSD,bitstampUSD)
-p PRECISION, --volume-precision=PRECISION
Multiply volume by this many decimal places before
import (default = 4)
Download links:
- SierraChart software:
- sierrachartfeed:
- http://we.lovebitco.in/schart/scfeed.py (Python 2.7 standalone source) or
- http://we.lovebitco.in/schart/scfeed.exe (2.41MiB) (Windows 32 bit exe, py2exe)
md5sum: eeaabc7edb427e1c6e419c042cad6848 *scfeed.exe
For history older than five days, before first running sierrachartfeed
The normal Bitcoincharts API only has five days of data available. If you have never run sierrachartfeed for a particular exchange ticker before, it will only get the last five days. However, if you want all trade data from the beginning of time, you can download some big files first:
- Go to this web page: http://api.bitcoincharts.com/v1/csv/
- Download the individual CSV files for exchange tickers you wish to track
- Extract the gzipped files to the SierraChart data directory (C:\SierraChart\Data)
(a utility like 7-zip can help you with this) - When you first run sierrachartfeed and specify a ticker with CSV, data will be imported from this file first
- The CSV file will be renamed after all data from it is imported
You will also need to re-download a new CSV if it has been over five days since you last ran sierrachartfeed.
The SCID filenames are different from previous versions, to avoid using or overwriting obsolete data. New filenames are of the format mtgoxUSD-p4.scid (where p4 indicates precision 4).
Changelog
(0.7.1 - 2014-02-10)
- CSV file synchronization impossible due to the remote files now being gzipped and unstable, abandoned,
- Import user-supplied local CSV method added,
- warn of a trade gap if last local trades are older than the 5 days available through API.
(0.7.0 - 2013-12-19)
- A local CSV is mantained and updated with trade data, resumed from nightly updates
- Fixed the only-5-days-available API
- Merged libraries into single file, fixed much library fustercluck
- Higher default precision, fixed volume rounding
- About 1/2 of .py is new code.
Jump to: