Jupyter Notebook is also bloaty software.
Along with what ETF and Loyce said. Use the proper tool for the job, there are better ways to get at the data. I'm assuming (yes I know don't assume) you are a student and have to use Jupyter for class so in the end it has to wind up in there. But, you can't just do A->B you need to get it to a size and format that it can handle.
-Dave
Topic: Hardware requirements for analyzing ouput of rusty-blockparser (Read 76 times)
This produces four .csv files which the biggest one is nearly 100 GB of data. Due to the size, I can't load it on a python jupyter notebook.
You can read the data into some database that does what you need, or use things like grep and split to get only the lines you need.
Hello,
I am trying to do some data analytics on the bitcoin blockchain. I am running a node and then parsing the .dat files using the rusty-blockparser.
This produces four .csv files which the biggest one is nearly 100 GB of data. Due to the size, I can't load it on a python jupyter notebook. I am wondering what common programs and strategies that people use to process this data.
Thanks for the help,
Best,
I am trying to do some data analytics on the bitcoin blockchain. I am running a node and then parsing the .dat files using the rusty-blockparser.
This produces four .csv files which the biggest one is nearly 100 GB of data. Due to the size, I can't load it on a python jupyter notebook. I am wondering what common programs and strategies that people use to process this data.
Thanks for the help,
Best,
Jump to: