Topic: Bitcoin block data available in CSV format (Read 2118 times)

First bump of the year!

I have been using this forum site for just over a year now but only just registered. Originally I found a BTC header dump online with about 589,000 * 80 byte headers in binary format. When I found this site, I came back from time to time to update my header database.

However just last week I decided to verify each block starting from the genesis block. Since there is no previous hash for the genesis block, the previous hash field is all zeroes. So all that is needed are 5 fields and we can generate all the output hashes to the current block.

The TimeStamp fields below are corrections needed to verify all blocks up to today:

729192,2022-03-27 01:07:19
729193,2022-03-27 01:34:44
736702,2022-05-17 00:24:12
736703,2022-05-17 00:26:48
736704,2022-05-17 00:27:55
736705,2022-05-17 00:33:43
736706,2022-05-17 00:45:21
736707,2022-05-17 00:51:46
736708,2022-05-17 01:13:24
736709,2022-05-17 01:27:32
736710,2022-05-17 01:29:01
736711,2022-05-17 01:36:42
736712,2022-05-17 01:40:09
736713,2022-05-17 01:52:02
736714,2022-05-17 02:04:50
736715,2022-05-17 02:11:46
736716,2022-05-17 02:17:59
736717,2022-05-17 02:20:44
736718,2022-05-17 03:22:42
736719,2022-05-17 03:25:02
736720,2022-05-17 03:50:52
736721,2022-05-17 03:51:57
736722,2022-05-17 04:08:49
736723,2022-05-17 04:09:59
736724,2022-05-17 04:11:00
736725,2022-05-17 04:11:29
736726,2022-05-17 04:32:19

This whole discussion would fit more in List of all Bitcoin addresses with a balance than in this topic. That's where I posted this:

Thanks for suggestions. It is working, and I am also downloading it 3 times a day only if the file is newer (-N, i hope it works).


Loyce, I found an error in your file:


24175084+6562107+2332583 = 33069774

But your "total" is 33069791.

I checked a few days, all of them have a total of 17 more than the sum of all 1,3 and bc1q.

I am  ignoring the total column for now...

Another possibility is that there are other address types with balance that do not start with neither 1, 3 or bc1q? That's new to me, but it could be. Maybe some people mistakenly generated address format?

You can see the first version of this job here:

By now it actually has a full week worth of data.

If it doesn't work: add the full path for wget. You can run it a bit more often, to catch updates quicker. For instance by replacing the second zero ("hours") by 7,8,10,14 (Amsterdam time).
You may also want to add "-q" so you don't receive an email every time it runs.

I just saw this now.

I will add a last week tab in my dashboard. Thanks


Let me try to add this cronojob. My first attempt.

I used this command:


Let's see if it works.

Can I just upload total_number_of_funded_addresses.json to your server after each update? All I need is an FTP-account (which isn't encrypted by default but I don't really worry about this data being compromised) or SSH-access so I can scp or rsync the file.
Alternatively, you can just wget it yourself from a cronjob, but the exact timing for my update varies.

I'll help you, but as to not hijack the thread, send me a PM with your OS, browser and what you are trying to do.

Hello. I am new to this, I might have made a mistake.
Do you know how to work around this problem?

With all due respect Mr. President, this is not true. 

LoyceV, I have tried many things. I made all types of requests I know and all of them failed.

I made a brief research about this topic, and it is not my server which is blocking: it is the browser. All modern browsers will not allow mixed content of HTTPS and HTTP

https://web.dev/what-is-mixed-content/

Since it's your own server blocking you: is there another way to work around this?

I now use 3 different (cheap) servers for 3 different subdomains (addresses.loyce.club, alladdresses.loyce.club and work in progress blockdata.loyce.club), and I've already replaced several older (also cheap) servers. I'm a bit hesitant to enable https on all of them, as it will make it more complicated to migrate to a different server again.
So I can enable it, but I'd rather not

Thanks and Lol Only for the first 3 days, I used this:
It's fixed.

Hi Loyce,

I am getting an error when trying to access your data:


But when I try https://addresses.loyce.club/total_number_of_funded_addresses.json]addresses.loyce.club/total_number_of_funded_addresses.json keeps failing, and my browser do not let me get http://.....

Can you enable https for your address.loyce.club?

Thank you.


Edit:
I also found an error in your file. You are adding new values to January 2020 

Html is certainly the best way to deliver data to the user.
But you cannot work with data in html.

For example, how will you group LoyceV blockdata in html format by miners? Html cannot do that. Or how will you calculate an average of all transaction size?

You need to convert the file to some better format for data analysis.

To make data analysis in the web you will need to use javascript.  Javascript has a very limited way to work with csv. So you need to convert it to json , work data in javascript,  then deliver it to html.

Csv is good to work in python/pandas, not js

Edit: open the console log in my daily dashboard (just click inspect to open devtools and click console). Write csvtoJSON and you will see LoyceV blockdata in json format. Then write miners, you will that json grouped by guessed_miners instead of blocked. You cannot do that with html.

HTML is the best format for the web and human consumption of data - you are reading it now.  

JSON is better suited to data transfer that doesn't need to (but can) be read by humans.

CSV (what LoyceV is doing) is best for import/export of data.

Updated: addresses.loyce.club/total_number_of_funded_addresses.json

CORS should work for you now:

I'm just using a standard Linux server.
Running this once a day works:
Result: a file named blockdata_12-23-2020.txt.
I could store this myself somewhere, but I kinda don't want to keep adding cronjobs. I've added it, see loyce.club/bitmover/. Starting today, it will keep daily snapshots. File blockdata.lastweek.txt will have data for the last 7 days.
Note that it currently only holds data for 1 day, adding one day per day.
Also note that this is untested, we'll know tomorrow if it works.

I am sorry. I am still learning how to deal with Json, but it is the best format for web.

This a valid json structure.

You can validate your json here

https://jsonformatter.curiousconcept.com/


Yes, please. I will do it after Christmas.
Sure. this will be a challenge to me. I will study how to do it . What platform do you use?can you point me some directions? I just use cpanel, I am a complete newbie with this.

I made addresses.loyce.club/total_number_of_funded_addresses.json, but I'm confused with the commas and accolades:
Can you tell me exactly where to put them?

Note that this server doesn't have CORS enabled yet. Do you need that again?

Because of the way I create the files, this wouldn't be easier than you keeping daily snapshots of blockdata.lastday.txt.gz by yourself.

I am not sure about it. But I think you are right. I will change it to total blocks-1.