MtGox trades history download R script

MusX

full member

Activity: 175

Merit: 100

Quote from: trepper on November 22, 2013, 11:50:35 AM

Your cool script works great on my desktop (win7x64+R3.0.2) and I find it very useful.

Thank you very much for it!!!!

I have just a Rnewbie question: what is the origin of the $date timeline?
In fact when I set the parameter "origin" to "1970-01-01" and apply as.date(mtgoxtrade$date) in order to have a more readable time column , I get dates like "4870-07-23". Undecided

simple way: pass as.numeric(substr(date,1,10)) to as.POSIXct. You will lose 0.1s precision if available
most correct way: (as.numeric(date)/x) - x should be kind of 1000, 1000000, to have a result with 10 numbers on left left from decimal sign. Than it will keep the microseconds after conversion to posix.

check this one: https://bitcointalksearch.org/topic/rbitcoin-r-bitcoin-integration-bitstamp-btce-kraken-343504

trepper

member

Activity: 89

Merit: 12

Your cool script works great on my desktop (win7x64+R3.0.2) and I find it very useful.

Thank you very much for it!!!!

I have just a Rnewbie question: what is the origin of the $date timeline?
In fact when I set the parameter "origin" to "1970-01-01" and apply as.date(mtgoxtrade$date) in order to have a more readable time column , I get dates like "4870-07-23". Undecided

MusX

full member

Activity: 175

Merit: 100

Quote from: chenchunyu88 on October 29, 2013, 12:48:19 PM

I am wondering if you can add code to convert the date and time to GMT time. I am trying to do some analysis on BTC price and its correlation and some more analysis with stock market index and some other index. Once we can figure out those stuffs, maybe we can publish a R package for BTC analysis.

The dates field in data from script should be in GMT already. To have the same TZ during the processing I'm using

Code:

Sys.setenv(TZ = 'GMT')

adjusted globally for whole project in RStudio.
It would be great to have a R package related to bitcoin, I have already a lot of code for that (which include mtgox/bitstamp private api calls with authorization (also v2 for mtgox) and it is possible to add new markets).
The blocker currently for me is a lack of documentation for that code. If anybody with experience of publishing R package would be interested to create one I would be happy to contribute. I would need to simplify some codes, drop some dependencies (example. RpostgreSQL), but I wanted to create package from my codes so it is on my todo list someday.

chenchunyu88

full member

Activity: 126

Merit: 100

I am wondering if you can add code to convert the date and time to GMT time. I am trying to do some analysis on BTC price and its correlation and some more analysis with stock market index and some other index. Once we can figure out those stuffs, maybe we can publish a R package for BTC analysis.

MusX

full member

Activity: 175

Merit: 100

good to see any feedback

I put your comment in the instruction.

razorfishsl

sr. member

Activity: 399

Merit: 250

Quote

3. if using first time after installation run commented "install.packages" function call, else skip this function call

Actually it is a bit more complicated on some platforms

Better you should go to "package installer" in 'R' and ensure the following are loaded and updated.

data.table
RCurl
RJSONIO
RSQLite

From CRAN(binaries)

Ensure that you also check the box for "lnstall dependancies"

Then load the script…. and off it goes.

chenchunyu88

full member

Activity: 126

Merit: 100

This is super cool.

MusX

full member

Activity: 175

Merit: 100

Quote from: arvin8 on September 12, 2013, 01:39:24 PM

http://pastebin.com/DWhZfGTy
this link seems to be offline?

now it's working fine, was the whole pastebin offline or only this one link?

arvin8

newbie

Activity: 9

Merit: 0

http://pastebin.com/DWhZfGTy
this link seems to be offline?

Adriano

staff

Activity: 1286

Merit: 1085

Quote from: MusX on September 06, 2013, 01:54:03 PM

anybody used it? any experience, feedback?

I saved your topic in order to test it when I have the time, but it wont happened before next monday at least... If no one else comment then I'll post my thoughts when I use it.

Thanks

Adriano

MusX

full member

Activity: 175

Merit: 100

anybody used it? any experience, feedback?

MusX

full member

Activity: 175

Merit: 100

There is some issues with fetching all the history from MtGox api.
I've prepared a script which joins trades history from bitcoincharts csv and mtgox recent trades to full set.
Output format supported are csv and sqlite db, configurable.
CSV separator and decimal mark configurable.
Support other currencies.
SSL support.
Open source.

environment: http://www.r-project.org/ (http://www.rstudio.com/)
script (v1.3): http://pastebin.com/DWhZfGTy

how to use:
1. open R
2. open script
3. if using first time after installation run commented "install.packages" function call, else skip this function call

Quote from: razorfishsl on October 27, 2013, 11:18:15 PM

Actually it is a bit more complicated on some platforms
Better you should go to "package installer" in 'R' and ensure the following are loaded and updated.

4. adjust params if needed: mtgox currency, output types (csv,sqlite), csv config
5. run the full script
6. wait until it will download and dump to files
7. look for .csv and/or .db files in R working directory (default on windows /Documents)

timing:
- downloading mtgoxUSD history from bitcoincharts: ~10min
- downloading mtgoxUSD recent trades from mtgox: ~5sec
- transform and merge: ~1min
- save to csv: ~1.5min (~296mb)
- save to sqlite: ~1min (~225mb, not compressed)

please report any bugs you will find

Topic: MtGox trades history download R script (Read 5143 times)