Topic: Bitcoin block data available in CSV format (Read 2005 times)
Bump!
Bump
4 month bump
I have been using this forum site for just over a year now but only just registered. Originally I found a BTC header dump online with about 589,000 * 80 byte headers in binary format. When I found this site, I came back from time to time to update my header database.
However just last week I decided to verify each block starting from the genesis block. Since there is no previous hash for the genesis block, the previous hash field is all zeroes. So all that is needed are 5 fields and we can generate all the output hashes to the current block.
The TimeStamp fields below are corrections needed to verify all blocks up to today:
729192,2022-03-27 01:07:19
729193,2022-03-27 01:34:44
736702,2022-05-17 00:24:12
736703,2022-05-17 00:26:48
736704,2022-05-17 00:27:55
736705,2022-05-17 00:33:43
736706,2022-05-17 00:45:21
736707,2022-05-17 00:51:46
736708,2022-05-17 01:13:24
736709,2022-05-17 01:27:32
736710,2022-05-17 01:29:01
736711,2022-05-17 01:36:42
736712,2022-05-17 01:40:09
736713,2022-05-17 01:52:02
736714,2022-05-17 02:04:50
736715,2022-05-17 02:11:46
736716,2022-05-17 02:17:59
736717,2022-05-17 02:20:44
736718,2022-05-17 03:22:42
736719,2022-05-17 03:25:02
736720,2022-05-17 03:50:52
736721,2022-05-17 03:51:57
736722,2022-05-17 04:08:49
736723,2022-05-17 04:09:59
736724,2022-05-17 04:11:00
736725,2022-05-17 04:11:29
736726,2022-05-17 04:32:19
Loyce, I found an error in your file:
24175084+6562107+2332583 = 33069774
But your "total" is 33069791.
I checked a few days, all of them have a total of 17 more than the sum of all 1,3 and bc1q.
This whole discussion would fit more in List of all Bitcoin addresses with a balance than in this topic. That's where I posted this:Code:
"06/01/2021":[
{"total": 33069791,
"1": 24175084,
"3": 6562107,
"bc1q": 2332583}]
}
{"total": 33069791,
"1": 24175084,
"3": 6562107,
"bc1q": 2332583}]
}
24175084+6562107+2332583 = 33069774
But your "total" is 33069791.
I checked a few days, all of them have a total of 17 more than the sum of all 1,3 and bc1q.
If you add up the addresses starting with 1, 3 and bc1q, you'll notice 16 addresses are missing. Those are:
Code:
bc1p23jk6urvv96x2gp3yqszqgpqyqszqgqa6qtuj
bc1p8qsysgrgypgjqufqtgs85gpcyqjzqsqfrw0l9
bc1p8ysyjgrfypfzqu3q9usrqgpeyqnzqfgexpv74
bc1pmfr3p9j00pfxjh0zmgp99y8zftmd3s5pmedqhyptwy6lm87hf5ss52r5n8
bc1pq2kqvpm76ewe20lcacq740p054at9sv7vxs0jn2u0r90af0k633322m7s8v
bc1pqyqszqgpqyqszqgpqyqszqgpqyqszqgpqyqszqgpqyqszqgpqyqs3wf0qm
bc1pv22mcnt30gwvk8g72szz700n4tkkx2qur2adj6pt8hl37hcf9dascxyf42
bc1px5sy2gr9yp8zqm3q2us8wgp4yq4jq0guggdp8
bc1pxcsyvgrxyp8jqmeqtqs8sgpkyq7zq0snaecz5
bc1pxgsyygrzyp9jq6eq2ss8ggpjyq5zq2gqvjed5
bc1pxqsrzgpjyqejqdpqx5srvgphyquzqwgdd7yg9
bc1pxssyggryypxjqmfq2cs8vgp5yqsjq0c760r6g
bc1pxusywgr8ypgzqupqtys8jgphyq4zqgcwqe32u
bc1pxvsyxgrrypxzqmpq25s82gpnypajqlgtqkfun
bc1pxysyzgrpyp9zq63q2vs8xgp3ypdjqhguvkagn
bc1zqyqsywvzqe
I don't know the story behind them, someone has been creating non-standard outputs. See txid 8bb2ce18914cfcb68e21686362b879396c2c27b51f1ec4be25c064f48f848f2d for most of them.bc1p8qsysgrgypgjqufqtgs85gpcyqjzqsqfrw0l9
bc1p8ysyjgrfypfzqu3q9usrqgpeyqnzqfgexpv74
bc1pmfr3p9j00pfxjh0zmgp99y8zftmd3s5pmedqhyptwy6lm87hf5ss52r5n8
bc1pq2kqvpm76ewe20lcacq740p054at9sv7vxs0jn2u0r90af0k633322m7s8v
bc1pqyqszqgpqyqszqgpqyqszqgpqyqszqgpqyqszqgpqyqszqgpqyqs3wf0qm
bc1pv22mcnt30gwvk8g72szz700n4tkkx2qur2adj6pt8hl37hcf9dascxyf42
bc1px5sy2gr9yp8zqm3q2us8wgp4yq4jq0guggdp8
bc1pxcsyvgrxyp8jqmeqtqs8sgpkyq7zq0snaecz5
bc1pxgsyygrzyp9jq6eq2ss8ggpjyq5zq2gqvjed5
bc1pxqsrzgpjyqejqdpqx5srvgphyquzqwgdd7yg9
bc1pxssyggryypxjqmfq2cs8vgp5yqsjq0c760r6g
bc1pxusywgr8ypgzqupqtys8jgphyq4zqgcwqe32u
bc1pxvsyxgrrypxzqmpq25s82gpnypajqlgtqkfun
bc1pxysyzgrpyp9zq63q2vs8xgp3ypdjqhguvkagn
bc1zqyqsywvzqe
I just saw this now.
By now it actually has a full week worth of data.Quote
Code:
0 0 * * * wget http://addresses.loyce.club/total_number_of_funded_addresses.json -O /home2/bitmover/public_html/json/total_number_of_funded_addresses.json
You may also want to add "-q" so you don't receive an email every time it runs.
Thanks for suggestions. It is working, and I am also downloading it 3 times a day only if the file is newer (-N, i hope it works).
Code:
0 0,8,16 * * * wget -N -q http://addresses.loyce.club/total_number_of_funded_addresses.json -O /home2/bitmover/public_html/json/total_number_of_funded_addresses.json
Loyce, I found an error in your file:
Code:
"06/01/2021":[
{"total": 33069791,
"1": 24175084,
"3": 6562107,
"bc1q": 2332583}]
}
{"total": 33069791,
"1": 24175084,
"3": 6562107,
"bc1q": 2332583}]
}
24175084+6562107+2332583 = 33069774
But your "total" is 33069791.
I checked a few days, all of them have a total of 17 more than the sum of all 1,3 and bc1q.
I am ignoring the total column for now...
Another possibility is that there are other address types with balance that do not start with neither 1, 3 or bc1q? That's new to me, but it could be. Maybe some people mistakenly generated address format?
You can see the first version of this job here:
I just saw this now.
By now it actually has a full week worth of data.Quote
Code:
0 0 * * * wget http://addresses.loyce.club/total_number_of_funded_addresses.json -O /home2/bitmover/public_html/json/total_number_of_funded_addresses.json
You may also want to add "-q" so you don't receive an email every time it runs.
I'm just using a standard Linux server.
Running this once a day works:
I could store this myself somewhere, but I kinda don't want to keep adding cronjobs. I've added it, see loyce.club/bitmover/. Starting today, it will keep daily snapshots. File blockdata.lastweek.txt will have data for the last 7 days.
Note that it currently only holds data for 1 day, adding one day per day.
Also note that this is untested, we'll know tomorrow if it works.
Running this once a day works:
Code:
wget https://loyce.club/blockdata/blockdata.lastday.txt.gz
gunzip blockdata.lastday.txt.gz
mv blockdata.lastday.txt $(date -r blockdata.lastday.txt "+blockdata_%m-%d-%Y.txt")
Result: a file named blockdata_12-23-2020.txt.gunzip blockdata.lastday.txt.gz
mv blockdata.lastday.txt $(date -r blockdata.lastday.txt "+blockdata_%m-%d-%Y.txt")
Note that it currently only holds data for 1 day, adding one day per day.
Also note that this is untested, we'll know tomorrow if it works.
I just saw this now.
I will add a last week tab in my dashboard. Thanks
Can I just upload total_number_of_funded_addresses.json to your server after each update? All I need is an FTP-account (which isn't encrypted by default but I don't really worry about this data being compromised) or SSH-access so I can scp or rsync the file.
Alternatively, you can just wget it yourself from a cronjob, but the exact timing for my update varies.
Alternatively, you can just wget it yourself from a cronjob, but the exact timing for my update varies.
Let me try to add this cronojob. My first attempt.
I used this command:
Code:
0 0 * * * wget http://addresses.loyce.club/total_number_of_funded_addresses.json -O /home2/bitmover/public_html/json/total_number_of_funded_addresses.json
Let's see if it works.
LoyceV, I have tried many things. I made all types of requests I know and all of them failed.
Can I just upload total_number_of_funded_addresses.json to your server after each update? All I need is an FTP-account (which isn't encrypted by default but I don't really worry about this data being compromised) or SSH-access so I can scp or rsync the file.Alternatively, you can just wget it yourself from a cronjob, but the exact timing for my update varies.
All modern browsers will not allow mixed content of HTTPS and HTTP
With all due respect Mr. President, this is not true.
Hello. I am new to this, I might have made a mistake.
Do you know how to work around this problem?
I'll help you, but as to not hijack the thread, send me a PM with your OS, browser and what you are trying to do.
All modern browsers will not allow mixed content of HTTPS and HTTP
With all due respect Mr. President, this is not true.
Hello. I am new to this, I might have made a mistake.
Do you know how to work around this problem?
All modern browsers will not allow mixed content of HTTPS and HTTP
With all due respect Mr. President, this is not true.
Since it's your own server blocking you: is there another way to work around this?
So I can enable it, but I'd rather not
Quote
Can you enable https for your address.loyce.club?
I now use 3 different (cheap) servers for 3 different subdomains (addresses.loyce.club, alladdresses.loyce.club and work in progress blockdata.loyce.club), and I've already replaced several older (also cheap) servers. I'm a bit hesitant to enable https on all of them, as it will make it more complicated to migrate to a different server again.So I can enable it, but I'd rather not
LoyceV, I have tried many things. I made all types of requests I know and all of them failed.
I made a brief research about this topic, and it is not my server which is blocking: it is the browser. All modern browsers will not allow mixed content of HTTPS and HTTP
Quote
What is mixed content?
Mixed content occurs when initial HTML is loaded over a secure HTTPS connection, but other resources (such as images, videos, stylesheets, scripts) are loaded over an insecure HTTP connection. This is called mixed content because both HTTP and HTTPS content are being loaded to display the same page, and the initial request was secure over HTTPS.
Requesting subresources using the insecure HTTP protocol weakens the security of the entire page, as these requests are vulnerable to on-path attacks, where an attacker eavesdrops on a network connection and views or modifies the communication between two parties. Using these resources, attackers can track users and replace content on a website, and in the case of active mixed content, take complete control over the page, not just the insecure resources.
Although many browsers report mixed content warnings to the user, by the time this happens, it is too late: the insecure requests have already been performed and the security of the page is compromised.
This is why browsers are increasingly blocking mixed content. If you have mixed content on your site, then fixing it will ensure the content continues to load as browsers become more strict.
https://web.dev/what-is-mixed-content/ Mixed content occurs when initial HTML is loaded over a secure HTTPS connection, but other resources (such as images, videos, stylesheets, scripts) are loaded over an insecure HTTP connection. This is called mixed content because both HTTP and HTTPS content are being loaded to display the same page, and the initial request was secure over HTTPS.
Requesting subresources using the insecure HTTP protocol weakens the security of the entire page, as these requests are vulnerable to on-path attacks, where an attacker eavesdrops on a network connection and views or modifies the communication between two parties. Using these resources, attackers can track users and replace content on a website, and in the case of active mixed content, take complete control over the page, not just the insecure resources.
Although many browsers report mixed content warnings to the user, by the time this happens, it is too late: the insecure requests have already been performed and the security of the page is compromised.
This is why browsers are increasingly blocking mixed content. If you have mixed content on your site, then fixing it will ensure the content continues to load as browsers become more strict.
I am getting an error when trying to access your data:
Since it's your own server blocking you: is there another way to work around this?Quote
This request has been blocked; the content must be served over HTTPS.
Quote
Can you enable https for your address.loyce.club?
I now use 3 different (cheap) servers for 3 different subdomains (addresses.loyce.club, alladdresses.loyce.club and work in progress blockdata.loyce.club), and I've already replaced several older (also cheap) servers. I'm a bit hesitant to enable https on all of them, as it will make it more complicated to migrate to a different server again.So I can enable it, but I'd rather not
Quote
I also found an error in your file. You are adding new values to January 2020 
Thanks and Lol Only for the first 3 days, I used this:Code:
last two digits of year of ISO week number (see %G)
This a valid json structure.
Updated: addresses.loyce.club/total_number_of_funded_addresses.jsonCORS should work for you now:
Code:
Header set Access-Control-Allow-Origin "https://bitcoindata.science"
Hi Loyce,
I am getting an error when trying to access your data:
Quote
Mixed Content: The page at 'https://bitcoindata.science/bitcoin-funded-addresses.html' was loaded over HTTPS, but requested an insecure XMLHttpRequest endpoint 'http://addresses.loyce.club/total_number_of_funded_addresses.json'. This request has been blocked; the content must be served over HTTPS.
But when I try https://addresses.loyce.club/total_number_of_funded_addresses.json]addresses.loyce.club/total_number_of_funded_addresses.json keeps failing, and my browser do not let me get http://.....
Can you enable https for your address.loyce.club?
Thank you.
Edit:
I also found an error in your file. You are adding new values to January 2020
Quote
"30/12/20":[
{"total": 33052369,
"1": 24279797,
"3": 6530683,
"bc1q": 2241872}],
"31/12/20":[
{"total": 33061152,
"1": 24245572,
"3": 6565930,
"bc1q": 2249633}],
"01/01/20":[
{"total": 33047455,
"1": 24204869,
"3": 6576644,
"bc1q": 2265925}],
"02/01/20":[
{"total": 32980241,
"1": 24189224,
"3": 6515391,
"bc1q": 2275609}],
"03/01/20":[
{"total": 32973335,
"1": 24204386,
"3": 6480663,
"bc1q": 2288269}]
}
{"total": 33052369,
"1": 24279797,
"3": 6530683,
"bc1q": 2241872}],
"31/12/20":[
{"total": 33061152,
"1": 24245572,
"3": 6565930,
"bc1q": 2249633}],
"01/01/20":[
{"total": 33047455,
"1": 24204869,
"3": 6576644,
"bc1q": 2265925}],
"02/01/20":[
{"total": 32980241,
"1": 24189224,
"3": 6515391,
"bc1q": 2275609}],
"03/01/20":[
{"total": 32973335,
"1": 24204386,
"3": 6480663,
"bc1q": 2288269}]
}
HTML is the best format for the web and human consumption of data - you are reading it now.
JSON is better suited to data transfer that doesn't need to (but can) be read by humans.
CSV (what LoyceV is doing) is best for import/export of data.
Html is certainly the best way to deliver data to the user.
But you cannot work with data in html.
For example, how will you group LoyceV blockdata in html format by miners? Html cannot do that. Or how will you calculate an average of all transaction size?
You need to convert the file to some better format for data analysis.
To make data analysis in the web you will need to use javascript. Javascript has a very limited way to work with csv. So you need to convert it to json , work data in javascript, then deliver it to html.
Csv is good to work in python/pandas, not js
Edit: open the console log in my daily dashboard (just click inspect to open devtools and click console). Write csvtoJSON and you will see LoyceV blockdata in json format. Then write miners, you will that json grouped by guessed_miners instead of blocked. You cannot do that with html.
I am sorry. I am still learning how to deal with Json, but it is the best format for web.
HTML is the best format for the web and human consumption of data - you are reading it now.
JSON is better suited to data transfer that doesn't need to (but can) be read by humans.
CSV (what LoyceV is doing) is best for import/export of data.
This a valid json structure.
Updated: addresses.loyce.club/total_number_of_funded_addresses.jsonCORS should work for you now:
Code:
Header set Access-Control-Allow-Origin "https://bitcoindata.science"
Quote
Sure. this will be a challenge to me. I will study how to do it . What platform do you use?can you point me some directions? I just use cpanel, I am a complete newbie with this.
I'm just using a standard Linux server.Running this once a day works:
Code:
wget https://loyce.club/blockdata/blockdata.lastday.txt.gz
gunzip blockdata.lastday.txt.gz
mv blockdata.lastday.txt $(date -r blockdata.lastday.txt "+blockdata_%m-%d-%Y.txt")
Note that it currently only holds data for 1 day, adding one day per day.
Also note that this is untested, we'll know tomorrow if it works.
Can you tell me exactly where to put them?
I am sorry. I am still learning how to deal with Json, but it is the best format for web.This a valid json structure.
Code:
{
"10/09/20":[
{"total": 31115394,
"1": 23536754,
"3": 6025854,
"bc1q": 1552770}],
"12/11/20":[
{"total": 32467107,
"1": 24223336,
"3": 6402910,
"bc1q": 1840844}],
"13/11/20":[
{"total": 32478246,
"1": 24230032,
"3": 6400301,
"bc1q": 1847896}],
"01/12/20":
[{"total": 32690359,
"1": 24255508,
"3": 6445534,
"bc1q": 1989300}]
}
https://jsonformatter.curiousconcept.com/
Quote
Note that this server doesn't have CORS enabled yet. Do you need that again?
Yes, please. I will do it after Christmas.
Quote
Quote
Can you make a blockdata.lastweek.txt file?
Because of the way I create the files, this wouldn't be easier than you keeping daily snapshots of blockdata.lastday.txt.gz by yourself.Can you make a CSV or JSON for total_number_of_funded_addresses.txt? It will make my work much easier.
A JSON would be amazing, like this. But it can be a csv as well
I made addresses.loyce.club/total_number_of_funded_addresses.json, but I'm confused with the commas and accolades:A JSON would be amazing, like this. But it can be a csv as well
Code:
{12/12/20:
{1: 200000,
3:300000,
bc1:30000},
13/12/20:{....
}
Code:
{10/09/20:
{total: 31115394,
1: 23536754,
3: 6025854,
bc1q: 1552770}
}
{12/11/20:
{total: 32467107,
1: 24223336,
3: 6402910,
bc1q: 1840844}
}
{13/11/20:
{total: 32478246,
1: 24230032,
3: 6400301,
bc1q: 1847896}
}
{01/12/20:
{total: 32690359,
1: 24255508,
3: 6445534,
bc1q: 1989300}
}Note that this server doesn't have CORS enabled yet. Do you need that again?
Quote
Can you make a blockdata.lastweek.txt file?
Because of the way I create the files, this wouldn't be easier than you keeping daily snapshots of blockdata.lastday.txt.gz by yourself.I think your average block time is too high: you have to divide the total time interval by 135, not 136, to get the average time in between blocks.
I am not sure about it. But I think you are right. I will change it to total blocks-1.
That's nice. I will have to learn a lot to work on that data.
Jump to: