Topic: Bitcoin market price monitor (Read 9224 times)

Here you go:

http://martindevans.appspot.com/file/Bitcoin_Market_Monitor.rar?agxtYXJ0aW5kZXZhbnNyEAsSCEJsb2JGaWxlGPW_JAw

A quick overview:
django
Contains django templates used for pages, currently just the one template for the market history graph (which is broken, and I don't intend to fix it anytime soon)

javascript
Contains the scripts needed for the history graph

Stylesheets
CSS files

__init.py__
Indicates that this folder is a python module which can be imported

cron.yaml
Defines to appengine the cron jobs I want to run

index.yaml
Describes to appengine the indices to build from my datastore

main.py
Nothing important, I should have deleted this really. It's just a dummy handler (literally "hello world") I used when setting everything up.

market_data.py
This is the interesting one, it contains the cron job which collects data, the handler which returns data when you access /market_data and the handler which returns the history graph which is broken at market-history.

trading.py
I initially hoped to build some functionality to automatically trade coins between markets as the prices moved. I haven't really got around to that yet, so this just contains a few data models, and a handler which just prints "foo" when trading should occur.

If you have any specific questions feel free to ask

please release - I'd just like to see how you did it. right now I really only use perl, but trying to migrate to python.

Appengine only offers Python, java and go. So I'm using python, I can release the source if you really like, it's nothing special though - I'm no python expert.

This new change has exposed something of an issue in the new system: in the last 5 days I have consumed 50% of my datastore, where I consumed 25% in the previous 6 months - the depth table is a *lot* more information than the ticker. I'm going to have to store the data more compactly, and every time the datastore starts getting full pack it up into a prerendered json document and store it somewhere else.

nb. I'll try to get around to writing a data dump cron job tomorrow, but if I don't get around to it there may be a day or two break in the data (which of course no one would know since I haven't got around to writing an API yet, I'll try to get round to that tomorrow too)

Sick dude.

Any chance I could get a look at the source code? What language are you using to pull in the data from mtgox?

Hey Ben, sorry for the slow response. It's been a busy month!

"fetch_all" isn't an option the site understands, instead you should use fetch=N to select a certain amount of values. N can be arbitrarily large, but if it's too big you'll get an error instead of useful data (because appengine has a hard limit on the time a page can take to load).

To load all data I suggest you fetch 1000 or so points, and fetch data in batches. Use end=Y-M-D, set that to the last value you got in the last batch, and fetch another 1000 and so on. Like this you can walk over the complete data set.

Announcing a change to data collected

previously, the site has collected the mtgox (and other sites) ticker data, this is nice but not terribly detailed. I recently switched the service over to gathering the complete mtgox depth table (currently no other exchanges). Currently this data can't be downloaded, I'll be working on a way to retrieve a history of the depth table sometime soon, it's a bit of a headache trying to find a reasonable way to publish such a massive dataset. If anyone is particularly interested in this data throw me a PM and I'll see what I can do about hurrying up development a little.

The current dataset is likely to stay up indefinitely, in its current form, but no more data will be added to it. Once an API is online for the depth data, there will be an overlap of the two sets, so there will be no gap in the data.

Hey Martin,

I tried accessing http://nosoperor-internal.appspot.com/market_data?fetch_all=true, but I only got data from 17 April. Do you have earlier data available?

Thanks!

That's odd, what error do you get for this link:

http://nosoperor-internal.appspot.com/market_data?start=2011-1-1

It works perfectly for me, and I get a screen of JSON spam.

As for the other link:

http://nosoperor-internal.appspot.com/market_data?fetch=60&market=mtgox

I had forgotten to deploy the correct kind of index, sorry. This link should now work too.

I get an error on the last two of those links. 

I've added some new query capabilities to the JSON data side of this:

Raw data:
http://nosoperor-internal.appspot.com/market_data

Fetch N datapoints. This fetches the first N datapoints ordered by date descending
fetch=N

Set start date:
start=Y-M-D

Set end date:
end=Y-M-D

filter by market ("mtgox", "bitcoin market" or "britcoin", I can add more if people wish)
market=M

eg: latest hour of mtgox data:
http://nosoperor-internal.appspot.com/market_data?fetch=60&market=mtgox

eg: all data this year:
http://nosoperor-internal.appspot.com/market_data?start=2011-1-1

As ever, requests welcome. And donations even more so

Uh, yeah, I should probably have mentioned that I haven't put any kind of limitson the data displayed on the graph. It tends to kill things if you try to select a large data set. That's next on my list to do!

@boombox
The graph you see is generated by Raphael js already, it's just not very pretty because it's a little overcrowded at the moment. I intend to use different data sets for different time periods, with different densities of points.

As for the raw JSON, I guess I could return the last day of data points, offset by a count for how many days into the past you wish to fetch the data.

I had to kill my page.

Hehehe, is the graph Javascript generating bitcoins for you? It stalled my browser totally.

Ok, some feedback on the graph: the timestamps would be better to have in a readable format.
I haven't yet seen a nice graph on any site of the bitcoin price, if I were you I would use:
http://www.highcharts.com/
or http://raphaeljs.com/
or http://code.google.com/p/flot/
And I would probably have different data sets for different timeperiods: For a 24 hour graph I would have perhaps 15 min between points and for a week graph I would have 2 hours between the points in order to make the graph quick loading.

And the JSON: I don't think you should return that many data points as default, or even if a request asks for it.
Multiple requests for getting that much data would be better.

Good work.

I threw this little market monitor together a while ago, mostly for fun and as an exercise in learning a bit more about google appengine. I guess it might come in handy to someone. This application has been monitoring prices on mtgox and bitcoin market for the last few months, at 1 hour intervals. I recently updated it to check every minute, it has all of this data stored and available to download, you can view a graph of the last 10000 data points here (ignore the weird web address, it was just an address i happened to have spare on appengine from an old project):

http://nosoperor-internal.appspot.com/market-history

If anyone wants to download the raw data, you can pull the market data in JSON format here:

http://nosoperor-internal.appspot.com/market_data

By default, this will return only the last 10000 data points, if you set the option fetchall=true then it will fetch every single data point ever collected.

So, it's got some basic functionality at the moment. It would be cool to add some more stuff, feel free to make suggestions of things which would be useful and I'll probably implement them