Topic: Bitcoin block data available in CSV format (Read 2118 times)
First bump of the year!
4 month bump
I have been using this forum site for just over a year now but only just registered. Originally I found a BTC header dump online with about 589,000 * 80 byte headers in binary format. When I found this site, I came back from time to time to update my header database.
However just last week I decided to verify each block starting from the genesis block. Since there is no previous hash for the genesis block, the previous hash field is all zeroes. So all that is needed are 5 fields and we can generate all the output hashes to the current block.
The TimeStamp fields below are corrections needed to verify all blocks up to today:
729192,2022-03-27 01:07:19
729193,2022-03-27 01:34:44
736702,2022-05-17 00:24:12
736703,2022-05-17 00:26:48
736704,2022-05-17 00:27:55
736705,2022-05-17 00:33:43
736706,2022-05-17 00:45:21
736707,2022-05-17 00:51:46
736708,2022-05-17 01:13:24
736709,2022-05-17 01:27:32
736710,2022-05-17 01:29:01
736711,2022-05-17 01:36:42
736712,2022-05-17 01:40:09
736713,2022-05-17 01:52:02
736714,2022-05-17 02:04:50
736715,2022-05-17 02:11:46
736716,2022-05-17 02:17:59
736717,2022-05-17 02:20:44
736718,2022-05-17 03:22:42
736719,2022-05-17 03:25:02
736720,2022-05-17 03:50:52
736721,2022-05-17 03:51:57
736722,2022-05-17 04:08:49
736723,2022-05-17 04:09:59
736724,2022-05-17 04:11:00
736725,2022-05-17 04:11:29
736726,2022-05-17 04:32:19
Loyce, I found an error in your file:
24175084+6562107+2332583 = 33069774
But your "total" is 33069791.
I checked a few days, all of them have a total of 17 more than the sum of all 1,3 and bc1q.
This whole discussion would fit more in List of all Bitcoin addresses with a balance than in this topic. That's where I posted this:Code:
"06/01/2021":[
{"total": 33069791,
"1": 24175084,
"3": 6562107,
"bc1q": 2332583}]
}
{"total": 33069791,
"1": 24175084,
"3": 6562107,
"bc1q": 2332583}]
}
24175084+6562107+2332583 = 33069774
But your "total" is 33069791.
I checked a few days, all of them have a total of 17 more than the sum of all 1,3 and bc1q.
If you add up the addresses starting with 1, 3 and bc1q, you'll notice 16 addresses are missing. Those are:
Code:
bc1p23jk6urvv96x2gp3yqszqgpqyqszqgqa6qtuj
bc1p8qsysgrgypgjqufqtgs85gpcyqjzqsqfrw0l9
bc1p8ysyjgrfypfzqu3q9usrqgpeyqnzqfgexpv74
bc1pmfr3p9j00pfxjh0zmgp99y8zftmd3s5pmedqhyptwy6lm87hf5ss52r5n8
bc1pq2kqvpm76ewe20lcacq740p054at9sv7vxs0jn2u0r90af0k633322m7s8v
bc1pqyqszqgpqyqszqgpqyqszqgpqyqszqgpqyqszqgpqyqszqgpqyqs3wf0qm
bc1pv22mcnt30gwvk8g72szz700n4tkkx2qur2adj6pt8hl37hcf9dascxyf42
bc1px5sy2gr9yp8zqm3q2us8wgp4yq4jq0guggdp8
bc1pxcsyvgrxyp8jqmeqtqs8sgpkyq7zq0snaecz5
bc1pxgsyygrzyp9jq6eq2ss8ggpjyq5zq2gqvjed5
bc1pxqsrzgpjyqejqdpqx5srvgphyquzqwgdd7yg9
bc1pxssyggryypxjqmfq2cs8vgp5yqsjq0c760r6g
bc1pxusywgr8ypgzqupqtys8jgphyq4zqgcwqe32u
bc1pxvsyxgrrypxzqmpq25s82gpnypajqlgtqkfun
bc1pxysyzgrpyp9zq63q2vs8xgp3ypdjqhguvkagn
bc1zqyqsywvzqe
I don't know the story behind them, someone has been creating non-standard outputs. See txid 8bb2ce18914cfcb68e21686362b879396c2c27b51f1ec4be25c064f48f848f2d for most of them.bc1p8qsysgrgypgjqufqtgs85gpcyqjzqsqfrw0l9
bc1p8ysyjgrfypfzqu3q9usrqgpeyqnzqfgexpv74
bc1pmfr3p9j00pfxjh0zmgp99y8zftmd3s5pmedqhyptwy6lm87hf5ss52r5n8
bc1pq2kqvpm76ewe20lcacq740p054at9sv7vxs0jn2u0r90af0k633322m7s8v
bc1pqyqszqgpqyqszqgpqyqszqgpqyqszqgpqyqszqgpqyqszqgpqyqs3wf0qm
bc1pv22mcnt30gwvk8g72szz700n4tkkx2qur2adj6pt8hl37hcf9dascxyf42
bc1px5sy2gr9yp8zqm3q2us8wgp4yq4jq0guggdp8
bc1pxcsyvgrxyp8jqmeqtqs8sgpkyq7zq0snaecz5
bc1pxgsyygrzyp9jq6eq2ss8ggpjyq5zq2gqvjed5
bc1pxqsrzgpjyqejqdpqx5srvgphyquzqwgdd7yg9
bc1pxssyggryypxjqmfq2cs8vgp5yqsjq0c760r6g
bc1pxusywgr8ypgzqupqtys8jgphyq4zqgcwqe32u
bc1pxvsyxgrrypxzqmpq25s82gpnypajqlgtqkfun
bc1pxysyzgrpyp9zq63q2vs8xgp3ypdjqhguvkagn
bc1zqyqsywvzqe
I just saw this now.
By now it actually has a full week worth of data.Quote
Code:
0 0 * * * wget http://addresses.loyce.club/total_number_of_funded_addresses.json -O /home2/bitmover/public_html/json/total_number_of_funded_addresses.json
You may also want to add "-q" so you don't receive an email every time it runs.
Thanks for suggestions. It is working, and I am also downloading it 3 times a day only if the file is newer (-N, i hope it works).
Code:
0 0,8,16 * * * wget -N -q http://addresses.loyce.club/total_number_of_funded_addresses.json -O /home2/bitmover/public_html/json/total_number_of_funded_addresses.json
Loyce, I found an error in your file:
Code:
"06/01/2021":[
{"total": 33069791,
"1": 24175084,
"3": 6562107,
"bc1q": 2332583}]
}
{"total": 33069791,
"1": 24175084,
"3": 6562107,
"bc1q": 2332583}]
}
24175084+6562107+2332583 = 33069774
But your "total" is 33069791.
I checked a few days, all of them have a total of 17 more than the sum of all 1,3 and bc1q.
I am ignoring the total column for now...
Another possibility is that there are other address types with balance that do not start with neither 1, 3 or bc1q? That's new to me, but it could be. Maybe some people mistakenly generated address format?
You can see the first version of this job here:
I just saw this now.
By now it actually has a full week worth of data.Quote
Code:
0 0 * * * wget http://addresses.loyce.club/total_number_of_funded_addresses.json -O /home2/bitmover/public_html/json/total_number_of_funded_addresses.json
You may also want to add "-q" so you don't receive an email every time it runs.
I'm just using a standard Linux server.
Running this once a day works:
I could store this myself somewhere, but I kinda don't want to keep adding cronjobs. I've added it, see loyce.club/bitmover/. Starting today, it will keep daily snapshots. File blockdata.lastweek.txt will have data for the last 7 days.
Note that it currently only holds data for 1 day, adding one day per day.
Also note that this is untested, we'll know tomorrow if it works.
Running this once a day works:
Code:
wget https://loyce.club/blockdata/blockdata.lastday.txt.gz
gunzip blockdata.lastday.txt.gz
mv blockdata.lastday.txt $(date -r blockdata.lastday.txt "+blockdata_%m-%d-%Y.txt")
Result: a file named blockdata_12-23-2020.txt.gunzip blockdata.lastday.txt.gz
mv blockdata.lastday.txt $(date -r blockdata.lastday.txt "+blockdata_%m-%d-%Y.txt")
Note that it currently only holds data for 1 day, adding one day per day.
Also note that this is untested, we'll know tomorrow if it works.
I just saw this now.
I will add a last week tab in my dashboard. Thanks
Can I just upload total_number_of_funded_addresses.json to your server after each update? All I need is an FTP-account (which isn't encrypted by default but I don't really worry about this data being compromised) or SSH-access so I can scp or rsync the file.
Alternatively, you can just wget it yourself from a cronjob, but the exact timing for my update varies.
Alternatively, you can just wget it yourself from a cronjob, but the exact timing for my update varies.
Let me try to add this cronojob. My first attempt.
I used this command:
Code:
0 0 * * * wget http://addresses.loyce.club/total_number_of_funded_addresses.json -O /home2/bitmover/public_html/json/total_number_of_funded_addresses.json
Let's see if it works.
LoyceV, I have tried many things. I made all types of requests I know and all of them failed.
Can I just upload total_number_of_funded_addresses.json to your server after each update? All I need is an FTP-account (which isn't encrypted by default but I don't really worry about this data being compromised) or SSH-access so I can scp or rsync the file.Alternatively, you can just wget it yourself from a cronjob, but the exact timing for my update varies.
All modern browsers will not allow mixed content of HTTPS and HTTP
With all due respect Mr. President, this is not true.
Hello. I am new to this, I might have made a mistake.
Do you know how to work around this problem?
I'll help you, but as to not hijack the thread, send me a PM with your OS, browser and what you are trying to do.
All modern browsers will not allow mixed content of HTTPS and HTTP
With all due respect Mr. President, this is not true.
Hello. I am new to this, I might have made a mistake.
Do you know how to work around this problem?
All modern browsers will not allow mixed content of HTTPS and HTTP
With all due respect Mr. President, this is not true.
Since it's your own server blocking you: is there another way to work around this?
So I can enable it, but I'd rather not
Quote
Can you enable https for your address.loyce.club?
I now use 3 different (cheap) servers for 3 different subdomains (addresses.loyce.club, alladdresses.loyce.club and work in progress blockdata.loyce.club), and I've already replaced several older (also cheap) servers. I'm a bit hesitant to enable https on all of them, as it will make it more complicated to migrate to a different server again.So I can enable it, but I'd rather not
LoyceV, I have tried many things. I made all types of requests I know and all of them failed.
I made a brief research about this topic, and it is not my server which is blocking: it is the browser. All modern browsers will not allow mixed content of HTTPS and HTTP
Quote
What is mixed content?
Mixed content occurs when initial HTML is loaded over a secure HTTPS connection, but other resources (such as images, videos, stylesheets, scripts) are loaded over an insecure HTTP connection. This is called mixed content because both HTTP and HTTPS content are being loaded to display the same page, and the initial request was secure over HTTPS.
Requesting subresources using the insecure HTTP protocol weakens the security of the entire page, as these requests are vulnerable to on-path attacks, where an attacker eavesdrops on a network connection and views or modifies the communication between two parties. Using these resources, attackers can track users and replace content on a website, and in the case of active mixed content, take complete control over the page, not just the insecure resources.
Although many browsers report mixed content warnings to the user, by the time this happens, it is too late: the insecure requests have already been performed and the security of the page is compromised.
This is why browsers are increasingly blocking mixed content. If you have mixed content on your site, then fixing it will ensure the content continues to load as browsers become more strict.
https://web.dev/what-is-mixed-content/ Mixed content occurs when initial HTML is loaded over a secure HTTPS connection, but other resources (such as images, videos, stylesheets, scripts) are loaded over an insecure HTTP connection. This is called mixed content because both HTTP and HTTPS content are being loaded to display the same page, and the initial request was secure over HTTPS.
Requesting subresources using the insecure HTTP protocol weakens the security of the entire page, as these requests are vulnerable to on-path attacks, where an attacker eavesdrops on a network connection and views or modifies the communication between two parties. Using these resources, attackers can track users and replace content on a website, and in the case of active mixed content, take complete control over the page, not just the insecure resources.
Although many browsers report mixed content warnings to the user, by the time this happens, it is too late: the insecure requests have already been performed and the security of the page is compromised.
This is why browsers are increasingly blocking mixed content. If you have mixed content on your site, then fixing it will ensure the content continues to load as browsers become more strict.
I am getting an error when trying to access your data:
Since it's your own server blocking you: is there another way to work around this?Quote
This request has been blocked; the content must be served over HTTPS.
Quote
Can you enable https for your address.loyce.club?
I now use 3 different (cheap) servers for 3 different subdomains (addresses.loyce.club, alladdresses.loyce.club and work in progress blockdata.loyce.club), and I've already replaced several older (also cheap) servers. I'm a bit hesitant to enable https on all of them, as it will make it more complicated to migrate to a different server again.So I can enable it, but I'd rather not
Quote
I also found an error in your file. You are adding new values to January 2020 
Thanks and Lol Only for the first 3 days, I used this:Code:
last two digits of year of ISO week number (see %G)
It's fixed. This a valid json structure.
Updated: addresses.loyce.club/total_number_of_funded_addresses.jsonCORS should work for you now:
Code:
Header set Access-Control-Allow-Origin "https://bitcoindata.science"
Hi Loyce,
I am getting an error when trying to access your data:
Quote
Mixed Content: The page at 'https://bitcoindata.science/bitcoin-funded-addresses.html' was loaded over HTTPS, but requested an insecure XMLHttpRequest endpoint 'http://addresses.loyce.club/total_number_of_funded_addresses.json'. This request has been blocked; the content must be served over HTTPS.
But when I try https://addresses.loyce.club/total_number_of_funded_addresses.json]addresses.loyce.club/total_number_of_funded_addresses.json keeps failing, and my browser do not let me get http://.....
Can you enable https for your address.loyce.club?
Thank you.
Edit:
I also found an error in your file. You are adding new values to January 2020
Quote
"30/12/20":[
{"total": 33052369,
"1": 24279797,
"3": 6530683,
"bc1q": 2241872}],
"31/12/20":[
{"total": 33061152,
"1": 24245572,
"3": 6565930,
"bc1q": 2249633}],
"01/01/20":[
{"total": 33047455,
"1": 24204869,
"3": 6576644,
"bc1q": 2265925}],
"02/01/20":[
{"total": 32980241,
"1": 24189224,
"3": 6515391,
"bc1q": 2275609}],
"03/01/20":[
{"total": 32973335,
"1": 24204386,
"3": 6480663,
"bc1q": 2288269}]
}
{"total": 33052369,
"1": 24279797,
"3": 6530683,
"bc1q": 2241872}],
"31/12/20":[
{"total": 33061152,
"1": 24245572,
"3": 6565930,
"bc1q": 2249633}],
"01/01/20":[
{"total": 33047455,
"1": 24204869,
"3": 6576644,
"bc1q": 2265925}],
"02/01/20":[
{"total": 32980241,
"1": 24189224,
"3": 6515391,
"bc1q": 2275609}],
"03/01/20":[
{"total": 32973335,
"1": 24204386,
"3": 6480663,
"bc1q": 2288269}]
}
HTML is the best format for the web and human consumption of data - you are reading it now.
JSON is better suited to data transfer that doesn't need to (but can) be read by humans.
CSV (what LoyceV is doing) is best for import/export of data.
Html is certainly the best way to deliver data to the user.
But you cannot work with data in html.
For example, how will you group LoyceV blockdata in html format by miners? Html cannot do that. Or how will you calculate an average of all transaction size?
You need to convert the file to some better format for data analysis.
To make data analysis in the web you will need to use javascript. Javascript has a very limited way to work with csv. So you need to convert it to json , work data in javascript, then deliver it to html.
Csv is good to work in python/pandas, not js
Edit: open the console log in my daily dashboard (just click inspect to open devtools and click console). Write csvtoJSON and you will see LoyceV blockdata in json format. Then write miners, you will that json grouped by guessed_miners instead of blocked. You cannot do that with html.
I am sorry. I am still learning how to deal with Json, but it is the best format for web.
HTML is the best format for the web and human consumption of data - you are reading it now.
JSON is better suited to data transfer that doesn't need to (but can) be read by humans.
CSV (what LoyceV is doing) is best for import/export of data.
This a valid json structure.
Updated: addresses.loyce.club/total_number_of_funded_addresses.jsonCORS should work for you now:
Code:
Header set Access-Control-Allow-Origin "https://bitcoindata.science"
Quote
Sure. this will be a challenge to me. I will study how to do it . What platform do you use?can you point me some directions? I just use cpanel, I am a complete newbie with this.
I'm just using a standard Linux server.Running this once a day works:
Code:
wget https://loyce.club/blockdata/blockdata.lastday.txt.gz
gunzip blockdata.lastday.txt.gz
mv blockdata.lastday.txt $(date -r blockdata.lastday.txt "+blockdata_%m-%d-%Y.txt")
Result: a file named blockdata_12-23-2020.txt.gunzip blockdata.lastday.txt.gz
mv blockdata.lastday.txt $(date -r blockdata.lastday.txt "+blockdata_%m-%d-%Y.txt")
Note that it currently only holds data for 1 day, adding one day per day.
Also note that this is untested, we'll know tomorrow if it works.
Can you tell me exactly where to put them?
I am sorry. I am still learning how to deal with Json, but it is the best format for web.This a valid json structure.
Code:
{
"10/09/20":[
{"total": 31115394,
"1": 23536754,
"3": 6025854,
"bc1q": 1552770}],
"12/11/20":[
{"total": 32467107,
"1": 24223336,
"3": 6402910,
"bc1q": 1840844}],
"13/11/20":[
{"total": 32478246,
"1": 24230032,
"3": 6400301,
"bc1q": 1847896}],
"01/12/20":
[{"total": 32690359,
"1": 24255508,
"3": 6445534,
"bc1q": 1989300}]
}
You can validate your json here"10/09/20":[
{"total": 31115394,
"1": 23536754,
"3": 6025854,
"bc1q": 1552770}],
"12/11/20":[
{"total": 32467107,
"1": 24223336,
"3": 6402910,
"bc1q": 1840844}],
"13/11/20":[
{"total": 32478246,
"1": 24230032,
"3": 6400301,
"bc1q": 1847896}],
"01/12/20":
[{"total": 32690359,
"1": 24255508,
"3": 6445534,
"bc1q": 1989300}]
}
https://jsonformatter.curiousconcept.com/
Quote
Note that this server doesn't have CORS enabled yet. Do you need that again?
Yes, please. I will do it after Christmas.
Quote
Quote
Can you make a blockdata.lastweek.txt file?
Because of the way I create the files, this wouldn't be easier than you keeping daily snapshots of blockdata.lastday.txt.gz by yourself.Can you make a CSV or JSON for total_number_of_funded_addresses.txt? It will make my work much easier.
A JSON would be amazing, like this. But it can be a csv as well
I made addresses.loyce.club/total_number_of_funded_addresses.json, but I'm confused with the commas and accolades:A JSON would be amazing, like this. But it can be a csv as well
Code:
{12/12/20:
{1: 200000,
3:300000,
bc1:30000},
13/12/20:{....
}
{1: 200000,
3:300000,
bc1:30000},
13/12/20:{....
}
Code:
{10/09/20:
{total: 31115394,
1: 23536754,
3: 6025854,
bc1q: 1552770}
}
{12/11/20:
{total: 32467107,
1: 24223336,
3: 6402910,
bc1q: 1840844}
}
{13/11/20:
{total: 32478246,
1: 24230032,
3: 6400301,
bc1q: 1847896}
}
{01/12/20:
{total: 32690359,
1: 24255508,
3: 6445534,
bc1q: 1989300}
}
Can you tell me exactly where to put them?{total: 31115394,
1: 23536754,
3: 6025854,
bc1q: 1552770}
}
{12/11/20:
{total: 32467107,
1: 24223336,
3: 6402910,
bc1q: 1840844}
}
{13/11/20:
{total: 32478246,
1: 24230032,
3: 6400301,
bc1q: 1847896}
}
{01/12/20:
{total: 32690359,
1: 24255508,
3: 6445534,
bc1q: 1989300}
}
Note that this server doesn't have CORS enabled yet. Do you need that again?
Quote
Can you make a blockdata.lastweek.txt file?
Because of the way I create the files, this wouldn't be easier than you keeping daily snapshots of blockdata.lastday.txt.gz by yourself.I think your average block time is too high: you have to divide the total time interval by 135, not 136, to get the average time in between blocks.
I am not sure about it. But I think you are right. I will change it to total blocks-1.
That's nice. I will have to learn a lot to work on that data.
I think I did something nice. In time I will add more data to it, maybe week, month... ?
I think your average block time is too high: you have to divide the total time interval by 135, not 136, to get the average time in between blocks.Code:
Header set Access-Control-Allow-Origin "*"
Code:
# Added 21-8-2020 based on https://bitcointalk.org/index.php?topic=5167469.msg55037886#msg55037886
Header set Access-Control-Allow-Origin "https://bitcointalk.org"
# Added 18-12-2020 based on https://bitcointalk.org/index.php?topic=5246271.msg55854208#msg55854208
Header set Access-Control-Allow-Origin "https://bitcoindata.science"
Let me know if it still doesn't work.Header set Access-Control-Allow-Origin "https://bitcointalk.org"
# Added 18-12-2020 based on https://bitcointalk.org/index.php?topic=5246271.msg55854208#msg55854208
Header set Access-Control-Allow-Origin "https://bitcoindata.science"
It is working.
THank you.
I think I did something nice. In time I will add more data to it, maybe week, month... ?
I just need to finish building some nice statistics and charts, then I can move on to more data. I will slowly scale it.
Code:
Header set Access-Control-Allow-Origin "*"
Code:
# Added 21-8-2020 based on https://bitcointalk.org/index.php?topic=5167469.msg55037886#msg55037886
Header set Access-Control-Allow-Origin "https://bitcointalk.org"
# Added 18-12-2020 based on https://bitcointalk.org/index.php?topic=5246271.msg55854208#msg55854208
Header set Access-Control-Allow-Origin "https://bitcoindata.science"
Let me know if it still doesn't work. Header set Access-Control-Allow-Origin "https://bitcointalk.org"
# Added 18-12-2020 based on https://bitcointalk.org/index.php?topic=5246271.msg55854208#msg55854208
Header set Access-Control-Allow-Origin "https://bitcoindata.science"
Hello LoyceV,
I have been working with your CSV and noticied that your server is blocking cross-origin requests (CORS). I don't know if that is your intention or not. My server was blocking CORS as well by default.
I cannot make simple requests like this one, that they will be blocked:
You can make tests here. This is a public library for CSV parsing. They have a demo, and if you try to load your CSV it will show this
"Access to XMLHttpRequest at 'https://loyce.club/blockdata/blockdata.lastday.txt' from origin 'null' has been blocked by CORS policy: The 'Access-Control-Allow-Origin' header has a value 'https://bitcointalk.org' that is not equal to the supplied origin."
I solved that error in my server just by adding this line to my .htaccess file.
You can read more here:
https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS/Errors/CORSAllowOriginNotMatchingOrigin
https://stackoverflow.com/questions/13421463/htaccess-access-control-allow-origin
I have been working with your CSV and noticied that your server is blocking cross-origin requests (CORS). I don't know if that is your intention or not. My server was blocking CORS as well by default.
I cannot make simple requests like this one, that they will be blocked:
Code:
$.ajax({
url: 'https://loyce.club/blockdata/blockdata.lastday.txt',
dataType: 'text',
})
url: 'https://loyce.club/blockdata/blockdata.lastday.txt',
dataType: 'text',
})
You can make tests here. This is a public library for CSV parsing. They have a demo, and if you try to load your CSV it will show this
"Access to XMLHttpRequest at 'https://loyce.club/blockdata/blockdata.lastday.txt' from origin 'null' has been blocked by CORS policy: The 'Access-Control-Allow-Origin' header has a value 'https://bitcointalk.org' that is not equal to the supplied origin."
I solved that error in my server just by adding this line to my .htaccess file.
Code:
Header set Access-Control-Allow-Origin "*"
You can read more here:
https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS/Errors/CORSAllowOriginNotMatchingOrigin
https://stackoverflow.com/questions/13421463/htaccess-access-control-allow-origin
Knock yourself out
I can't imagine much interesting graphs from only one day of data though.The lastday.txt file is meant to add to the full file if you download an update once a day. But it wouldn't make sense to download more than once a day.
I think I can make something interesting with daily data.
Additionally, as it is a small file it can be easily downloaded.
How often do you update this file?
https://loyce.club/blockdata/blockdata.lastday.txt
Daily.https://loyce.club/blockdata/blockdata.lastday.txt
Quote
Can I use this in my website? I plan to make some charts with it. Everyone who enter the url would make one request. I dont have many visitors.. yet (I hope lol)
Knock yourself out I can't imagine much interesting graphs from only one day of data though.The lastday.txt file is meant to add to the full file if you download an update once a day. But it wouldn't make sense to download more than once a day.
Hello LoyceV
How often do you update this file?
https://loyce.club/blockdata/blockdata.lastday.txt
Can I use this in my website? I plan to make some charts with it. Everyone who enter the url would make one request. I dont have many visitors.. yet (I hope lol)
How often do you update this file?
https://loyce.club/blockdata/blockdata.lastday.txt
Can I use this in my website? I plan to make some charts with it. Everyone who enter the url would make one request. I dont have many visitors.. yet (I hope lol)
Do you have any plans of capturing the transactions or outputs in a similar manner as the blocks?
I'd love to, but I'd need much more server space for this.I have Blockchair.com's "outputs" downloaded. It takes 115 GB, and Blockchair's "inputs" (which I don't have yet) is ~3.5 times larger.
"Outputs" has files like this:
Code:
block_id transaction_hash index time value value_usd recipient type script_hex is_from_coinbase is_spendable
653356 dd9ffd382bf83248711ea47abe5b8ac21d9b9a34c6c7abdda6a0b9b1e99adc60 0 2020-10-19 00:02:17 688376206 79156.1016 1KFHE7w8BhaENAswwryaoccDb6qcT6DbYY pubkeyhash 76a914c825a1ecf2a6830c4401620c3a16f1995057c2ab88ac 1 -1
653356 dd9ffd382bf83248711ea47abe5b8ac21d9b9a34c6c7abdda6a0b9b1e99adc60 1 2020-10-19 00:02:17 0 0 d-0ca2834cdc0b2ca644a7aa6c6c6b571e nulldata 6a24aa21a9ed79186d8dd65899b51c686badefe90276e5dc2879c2ca48f05d25830989285e3e080000000000000000 1 0
653356 dd9ffd382bf83248711ea47abe5b8ac21d9b9a34c6c7abdda6a0b9b1e99adc60 2 2020-10-19 00:02:17 0 0 d-7ac1238b74a3ecbfab24678cb7010a8d nulldata 6a4c2952534b424c4f434b3a03b3765c82064ea9fb16ab2211f12c3a6e7b13586cb2e474e7372531002a982c 1 0
653356 dd9ffd382bf83248711ea47abe5b8ac21d9b9a34c6c7abdda6a0b9b1e99adc60 3 2020-10-19 00:02:17 0 0 d-00eb8b6bf9a16a7d66d3739eadcf9c09 nulldata 6a24b9e11b6d01b8daa2d578f9b3cc56d35b0a71ab02e7878e20773ac2d34bf50dce8f9cd4b8 1 0
653356 81f82b69058e7e7d766928183ec05f6dfac5a71b950ac388f5301ba9be2af74e 0 2020-10-19 00:02:17 7890000 907.2679 1GRh1yKNhxjfKmPATYdYAC7xwUHvv6doVN pubkeyhash 76a914a935462d90eb370ed808de551b57df5eb9016bf588ac 0 -1
653356 1f7c023b5fa5c9389db2977578f6bba48c9aaa3627be2bb87686666949d9a6c7 0 2020-10-19 00:02:17 872314 100.307 1G4Y3mFLamLffj4p248iNWYiWt7A5v2MTS pubkeyhash 76a914a5350d4988f7607cc8001c7436e2750720dd395c88ac 0 -1
653356 1f7c023b5fa5c9389db2977578f6bba48c9aaa3627be2bb87686666949d9a6c7 1 2020-10-19 00:02:17 97406 11.2007 1G2AUdDQDFBDKgx8zyJMVJ2mh65KFs4Qs5 pubkeyhash 76a914a4c2368f121f9eabf345b3ec7af3c851da7ebaf488ac 0 -1
653356 1f7c023b5fa5c9389db2977578f6bba48c9aaa3627be2bb87686666949d9a6c7 1 2020-10-19 00:02:17 97406 11.2007 1G2AUdDQDFBDKgx8zyJMVJ2mh65KFs4Qs5 pubkeyhash 76a914a4c2368f121f9eabf345b3ec7af3c851da7ebaf488ac 0 -1
653356 1f7c023b5fa5c9389db2977578f6bba48c9aaa3627be2bb87686666949d9a6c7 1 2020-10-19 00:02:17 97406 11.2007 1G2AUdDQDFBDKgx8zyJMVJ2mh65KFs4Qs5 pubkeyhash 76a914a4c2368f121f9eabf345b3ec7af3c851da7ebaf488ac 0 -1
I'm not sure how useful it would be for any purpose to download CSV-files with a billion rows and many gigabytes large.653356 dd9ffd382bf83248711ea47abe5b8ac21d9b9a34c6c7abdda6a0b9b1e99adc60 0 2020-10-19 00:02:17 688376206 79156.1016 1KFHE7w8BhaENAswwryaoccDb6qcT6DbYY pubkeyhash 76a914c825a1ecf2a6830c4401620c3a16f1995057c2ab88ac 1 -1
653356 dd9ffd382bf83248711ea47abe5b8ac21d9b9a34c6c7abdda6a0b9b1e99adc60 1 2020-10-19 00:02:17 0 0 d-0ca2834cdc0b2ca644a7aa6c6c6b571e nulldata 6a24aa21a9ed79186d8dd65899b51c686badefe90276e5dc2879c2ca48f05d25830989285e3e080000000000000000 1 0
653356 dd9ffd382bf83248711ea47abe5b8ac21d9b9a34c6c7abdda6a0b9b1e99adc60 2 2020-10-19 00:02:17 0 0 d-7ac1238b74a3ecbfab24678cb7010a8d nulldata 6a4c2952534b424c4f434b3a03b3765c82064ea9fb16ab2211f12c3a6e7b13586cb2e474e7372531002a982c 1 0
653356 dd9ffd382bf83248711ea47abe5b8ac21d9b9a34c6c7abdda6a0b9b1e99adc60 3 2020-10-19 00:02:17 0 0 d-00eb8b6bf9a16a7d66d3739eadcf9c09 nulldata 6a24b9e11b6d01b8daa2d578f9b3cc56d35b0a71ab02e7878e20773ac2d34bf50dce8f9cd4b8 1 0
653356 81f82b69058e7e7d766928183ec05f6dfac5a71b950ac388f5301ba9be2af74e 0 2020-10-19 00:02:17 7890000 907.2679 1GRh1yKNhxjfKmPATYdYAC7xwUHvv6doVN pubkeyhash 76a914a935462d90eb370ed808de551b57df5eb9016bf588ac 0 -1
653356 1f7c023b5fa5c9389db2977578f6bba48c9aaa3627be2bb87686666949d9a6c7 0 2020-10-19 00:02:17 872314 100.307 1G4Y3mFLamLffj4p248iNWYiWt7A5v2MTS pubkeyhash 76a914a5350d4988f7607cc8001c7436e2750720dd395c88ac 0 -1
653356 1f7c023b5fa5c9389db2977578f6bba48c9aaa3627be2bb87686666949d9a6c7 1 2020-10-19 00:02:17 97406 11.2007 1G2AUdDQDFBDKgx8zyJMVJ2mh65KFs4Qs5 pubkeyhash 76a914a4c2368f121f9eabf345b3ec7af3c851da7ebaf488ac 0 -1
653356 1f7c023b5fa5c9389db2977578f6bba48c9aaa3627be2bb87686666949d9a6c7 1 2020-10-19 00:02:17 97406 11.2007 1G2AUdDQDFBDKgx8zyJMVJ2mh65KFs4Qs5 pubkeyhash 76a914a4c2368f121f9eabf345b3ec7af3c851da7ebaf488ac 0 -1
653356 1f7c023b5fa5c9389db2977578f6bba48c9aaa3627be2bb87686666949d9a6c7 1 2020-10-19 00:02:17 97406 11.2007 1G2AUdDQDFBDKgx8zyJMVJ2mh65KFs4Qs5 pubkeyhash 76a914a4c2368f121f9eabf345b3ec7af3c851da7ebaf488ac 0 -1
Someone requested similar data last week, so I made one large CSV-file with this data. See all_Bitcoin_txids.txt.bz2 (19 GB) (this hosting expires January 23, 2021). The header is not included:
Code:
block_id,transaction_hash
The file itself starts with this:Code:
0,4a5e1e4baab89f3a32518a88c31bc87f618f76673e2cc77ab2127b7afdeda33b
1,0e3e2357e806b6cdb1f70b54c3a3a17b6714ee1f0e68bebb44a74b1efd512098
2,9b0fc92260312ce44e74ef369f5c66bbb85848f2eddd5a7a1cde251e54ccfdd5
3,999e1c837c76a1b7fbb7e57baf87b309960f5ffefbf2a9b95dd890602272f644
4,df2b060fa2e5e9c8ed5eaf6a45c13753ec8c63282b2688322eba40cd98ea067a
5,63522845d294ee9b0188ae5cac91bf389a0c3723f084ca1025e7d9cdfe481ce1
6,20251a76e64e920e58291a30d4b212939aae976baca40e70818ceaa596fb9d37
7,8aa673bc752f2851fd645d6a0a92917e967083007d9c1684f9423b100540673f
8,a6f7f1c0dad0f2eb6b13c4f33de664b1b0e9f22efad5994a6d5b6086d85e85e3
9,0437cd7f8525ceed2324359c2d0ba26006d92d856a9c20fa0241106ee5a597c9
This data is updated until blockchair_bitcoin_outputs_20201127.tsv.gz.1,0e3e2357e806b6cdb1f70b54c3a3a17b6714ee1f0e68bebb44a74b1efd512098
2,9b0fc92260312ce44e74ef369f5c66bbb85848f2eddd5a7a1cde251e54ccfdd5
3,999e1c837c76a1b7fbb7e57baf87b309960f5ffefbf2a9b95dd890602272f644
4,df2b060fa2e5e9c8ed5eaf6a45c13753ec8c63282b2688322eba40cd98ea067a
5,63522845d294ee9b0188ae5cac91bf389a0c3723f084ca1025e7d9cdfe481ce1
6,20251a76e64e920e58291a30d4b212939aae976baca40e70818ceaa596fb9d37
7,8aa673bc752f2851fd645d6a0a92917e967083007d9c1684f9423b100540673f
8,a6f7f1c0dad0f2eb6b13c4f33de664b1b0e9f22efad5994a6d5b6086d85e85e3
9,0437cd7f8525ceed2324359c2d0ba26006d92d856a9c20fa0241106ee5a597c9
I'd like to have Blockchair's "outputs" on my own VPS, and while I'm at it "inputs" too, so I have it available with high bandwidth when needed. Downloading "inputs" will take about 6 weeks.
But I haven't decided on a webhost yet. It'll instantly fill 0.5 TB and if I start this, I want it to be future proof (and not too expensive).
LoyceV,
Do you have any plans of capturing the transactions or outputs in a similar manner as the blocks?
Do you have any plans of capturing the transactions or outputs in a similar manner as the blocks?
Is there any way to sort blocks from the lowest hash value to the highest?
sort?Code:
wget https://loyce.club/blockdata/hash.txt.gz -O - | gunzip | grep -v "^id,hash$" | sort -t, -k2 | head
Quote
I want to see what is the lowest hash value human has generated on the proof of work procedure.
000000000000000000000003681c2df35533c9578fb6aace040b0dfe0d446413, block 634842: "Mined on 2020-06-15 09:23 (5 months ago)"More results:
Code:
id,hash
634842,000000000000000000000003681c2df35533c9578fb6aace040b0dfe0d446413
585774,000000000000000000000019b43763eb4519f4fe65eae9be90fe73117b89026d
625857,000000000000000000000030f8cf8e0a76db53525aff8d56dcfdf4c74fc7878c
622050,000000000000000000000031a10e42c80137b3c3ad3e15c5dfb4ea213c83e497
649141,000000000000000000000084dc46d0da1c920a9b1ce16d91159655cfe1055482
593568,00000000000000000000008b80d0529bc20ac2c580f85acd2aed0baf5b82bdf5
622125,0000000000000000000000a773ce38733b4f36cf2e52581e055e3a4216291256
515910,0000000000000000000000ab789f6d71d9642ae3f697975ccd00afcb98fe6bd2
500174,0000000000000000000000bb5b432a764ad6c7acf677dcd99161abfdf68e698e
643346,0000000000000000000000c87731a1d688f3671d44ef862fe0ee2e74e1a845df
556828,0000000000000000000000ce3e3d62bb5575c1dccfca1ab635412b2c7818b3b8
600225,0000000000000000000000db2867af8040293a4a38db547883a053e0852d85f3
616078,0000000000000000000000e485cca75b9e9a48a6b532d21582ebfb2ebd87470a
575171,0000000000000000000000eb502db6b128d84c1f65943d55b31dc17871528ff9
643496,0000000000000000000000f3c07c18949f337fd5dafc908d3b72bd55684b08d2
620586,0000000000000000000000f4785d6f53ad5adccb7e4cfe31f4b207cf3b5b6329
641181,0000000000000000000000f9794a504f24ef5a8376bd71999f5c1be0cb030178
458091,00000000000000000000011246f099d94f91628d71c9d75ad2f9a06e2beb7e92
599638,00000000000000000000011679cdea1ace135eb018393b0124c22e270a028210
626740,0000000000000000000001295d284da4495a126feb5c803adda37923dd491fa5
625273,00000000000000000000012c360cda3d75b7c06af83c5883422878e2b15c79c0
637882,000000000000000000000132b2092267bd8d9325414627c2d0dc3ff5b4e7b2b2
641109,0000000000000000000001333ed8158bdb219a195102db40b13925ec0ae8fc88
368527,00000000000000000000013712632da34788a7b4ae2cd78f7982c7036610126b
615608,00000000000000000000014bc0b06ecff006a771d9a7b3247dbeb323a9e12d4b
619919,00000000000000000000014e5dc331e8841eb0f0aed8fd22602bfaf93f7a7640
582757,000000000000000000000150ad70b6d369df271f83ca3d6ac859b800f137c9a4
638817,00000000000000000000015b80b2a0848ccefdf5fbf7ad5a24e90f157d751a5a
598482,0000000000000000000001602eb228828476fe030cdcdfb3114f2ee3fdb51d77
622241,000000000000000000000174d9b1478a84a04e978a6250ce18e1799b39ca2d87
652660,0000000000000000000001836835d7c46f837f06a149be9e8b0a24de40210624
532023,00000000000000000000018ba128090ad6dd5fbcd9d666541f060c791f86ad78
605463,0000000000000000000001906fd2657c2aa87d80a1dd31af693c7507cd31179f
572822,00000000000000000000019257b49b3f46d8f7c3d6f3d4153962f1386c19337a
649763,000000000000000000000193a0abf401558a9c125d7f94773d74b6a1329c6b41
577021,0000000000000000000001aa6b9676d6b988c0549f42047dc11ac4d7660b845e
607040,0000000000000000000001b26ca7b70da9a77187933009c7d56893ceba127501
595237,0000000000000000000001bc0f20af28f38472c0eb221137fce12b089c9cd82c
522896,0000000000000000000001d3a96927a8878a4c2b0651cd816594c80b77e29111
598034,0000000000000000000001d5b5ce68ef4f426909c6f6299a945a5a781cd1f18a
587409,0000000000000000000001f57b098911a90b164b9812304f4f7615cf9f91f66a
606267,0000000000000000000001fdb893eeb401358d00c29a9a6246054bd55a9dbf53
524513,00000000000000000000020093ea502dec7cd56a1f5e2e8dd0e3d669c47ebc0c
543672,000000000000000000000206254c56610d6bc9b64c141d7b1705ac9e8fa43b55
512588,00000000000000000000021c043e439b5f4b632389b0062306bf2d4e0b657c7c
613562,00000000000000000000021e32be09862eafb7a2fe78f2b648d7252a022dc132
620410,000000000000000000000227e4a7dbbf84953cdb4beaaabcea9a8309ed5c8e64
522090,00000000000000000000022e7a595abee58621a090c8076dac4a1c3a041079af
513032,000000000000000000000241d3121e74bbbc7a034666d6c6f4d46cac2dc2394a
491270,0000000000000000000002569beb610e7f806774fd0c650c415a78a087b9d979
........
........
........
24372,00000000ff98f141099bd4d415c14152129f0b90994ff4c954beee9f0f435647
20104,00000000ff99a9ecd6a37d2ed796caff46f5ad959d5b6682b3672916375d3090
14280,00000000ff9e75d76ba82637f82ec0ce3c0c24f7d0d7af9a65c5d370f5cdaf97
688,00000000ffa5e19e744552ad9cf9255de8ac6d4c03b9ed259fca020c9a071b1e
10997,00000000ffa6ec3c76e387d9efe6d8c0a0644042b610600ec5c211249e12f0d8
27991,00000000ffab0cc1d059461b509882d0806c1ff38d9617818246c5d884d8b5f4
22465,00000000ffabeba413276a288d957179c7a4a4088de6c22508c71698a6476cd8
25021,00000000ffac6dab77739c3fad6f6853968875c613fdfc886a6056806ccb4d69
2569,00000000ffb05d42c619788bb38636c1e59ec056ad854893cb9fcb6ceabe14dd
18143,00000000ffb062296c9d4eb5f87bbf905d30669d26eab6bced341bd3f1dba5fd
12508,00000000ffb0aaf87b2664c6fe25e112195b0d7836250f7d77493ca1afaead71
23246,00000000ffb0ddfef22ff4b3522c2593c066caa235da1ffa6668b62271258c63
12605,00000000ffb145ba5f8a00e205c1a25998c6e02e4bd3f84a8c152e74d0fff9a7
23896,00000000ffb23c30adbe040379f8a0782364a0afc0dfe3a94d99a650f4e5ea09
32169,00000000ffb62a7f74340a082d00f82dbfdcfbbdf1e0f2f5de0fbe9e91ac0a14
26790,00000000ffb6c4107cbfe90cd60e610de37bcba3ae63a4320d9df3dd904b0001
20725,00000000ffb805b552395141aec921db3877bb1220651f0f7a265e2754629562
23484,00000000ffb874ed66ee7a872921177ea3e96d281c78902e21e497b45aec9785
12798,00000000ffbc3b6502d7df969a2d5546d0fa8a6e72fa69e4e84689d43f093419
2507,00000000ffbc5bf0a1427c2542e9ae43ede6fe888582702f50c5719f67873ca3
4345,00000000ffbdef4f2a349b6ce934e80ac0965cf7f841e7ffc9c03ea78404e11a
1391,00000000ffbe97137d327beae4ec8b7d96b1eb961505d6ef1c079b587e136604
10046,00000000ffc8f2beb8c54d552bf7e0322dd3b70e7901085ed081b78b06a38567
14890,00000000ffca2b3b5df775b36415f5c19ef904dfd70a97c3dc2ea56654e87d73
8750,00000000ffcb592e42740b24ff659f3f2e5d938175fe918e7ef46efa6b86e36d
3340,00000000ffd27d3cd7323dfb9cfda5fb363cbfe283e7fb248a5bdab693055a60
29040,00000000ffd5fbc6cdb40029703159d1b1bbfcef040e76fed978cb4367dad0b0
6765,00000000ffd6cf9652a947452972f4cccfc65eed439f498abac9d250eb844ebe
12461,00000000ffd75a76e7fe0c04754f718ae5fddca404f42f60247ce454f025504e
16751,00000000ffd76c547245e25edb6ce120ffbde1254df21212c050cf81467869f5
1808,00000000ffd7ae8a525d0083ee8384e13a6cbe1f1218f9ae26a637fbc2093a05
27302,00000000ffd9ba907954169f7946abb04f507ee1f6170654ffdacd602e918f34
11561,00000000ffdace7df6579699cdd8ba7be094b5637dcecc7b381af8d83e6d60d1
1601,00000000ffdacee18e2c714d840888e87796855b446f4883c2834c6ad1b9a1c7
10376,00000000ffdb74d25c7238e3fc4d7d7244655357cc750d6be2955770a8fabdf9
17598,00000000ffdd1b6aa100d4abc38ed5618b900df7184e2f9a39d739f515d2375f
10946,00000000ffdf54f4df138ef2b35b1d6a46ff67512d09646a023b92e35ba52069
18333,00000000ffdfe5c62679db8724bd7f5c0b97636b0269a676a5149c0c2e148381
31737,00000000ffe17801335b03ff320346cdfa86609158fd1e39d8e91d77a5f2f519
31577,00000000ffe3cf2a3904cbfe6b3808936d85169a7f520ba413e3652dae432ae1
28614,00000000ffe5255e229781dd72b1626c4df90e77390d38f6337ce949bc3ff300
16250,00000000ffeb2708cfc0aa1b5db3a6eacccc9112eb10cc0de2ec3e98c6c11a7a
9396,00000000ffeb8e847d94f7085c6cc0359c77d8ebe52b9fb59b90e9ef95a23c44
31108,00000000ffeec57f29c3f3df6014774aa1a2f00114ab16bd26b0d270974cd706
1304,00000000fff0728c5da1548b4f15576aa272f0f4952031ae56a5be355310ba0a
30133,00000000fff31576873d6ec35ba6659e72193ddf621ea1edd7faa462336a2211
31664,00000000fff3d56591bc0eae8cee73a214d1d451d704f86126415ed540850df7
27948,00000000fff58158d6ce595a2976d547f16b0ec8aba64b7a9a68c78d469b54b5
5386,00000000fff91949181449d048aca5a1cc6d0e3e3f34e89c00ab2709696b8da0
32009,00000000fff9e01287736bee2fecdd88ae31b5602858c5273d43351cfadecd58
634842,000000000000000000000003681c2df35533c9578fb6aace040b0dfe0d446413
585774,000000000000000000000019b43763eb4519f4fe65eae9be90fe73117b89026d
625857,000000000000000000000030f8cf8e0a76db53525aff8d56dcfdf4c74fc7878c
622050,000000000000000000000031a10e42c80137b3c3ad3e15c5dfb4ea213c83e497
649141,000000000000000000000084dc46d0da1c920a9b1ce16d91159655cfe1055482
593568,00000000000000000000008b80d0529bc20ac2c580f85acd2aed0baf5b82bdf5
622125,0000000000000000000000a773ce38733b4f36cf2e52581e055e3a4216291256
515910,0000000000000000000000ab789f6d71d9642ae3f697975ccd00afcb98fe6bd2
500174,0000000000000000000000bb5b432a764ad6c7acf677dcd99161abfdf68e698e
643346,0000000000000000000000c87731a1d688f3671d44ef862fe0ee2e74e1a845df
556828,0000000000000000000000ce3e3d62bb5575c1dccfca1ab635412b2c7818b3b8
600225,0000000000000000000000db2867af8040293a4a38db547883a053e0852d85f3
616078,0000000000000000000000e485cca75b9e9a48a6b532d21582ebfb2ebd87470a
575171,0000000000000000000000eb502db6b128d84c1f65943d55b31dc17871528ff9
643496,0000000000000000000000f3c07c18949f337fd5dafc908d3b72bd55684b08d2
620586,0000000000000000000000f4785d6f53ad5adccb7e4cfe31f4b207cf3b5b6329
641181,0000000000000000000000f9794a504f24ef5a8376bd71999f5c1be0cb030178
458091,00000000000000000000011246f099d94f91628d71c9d75ad2f9a06e2beb7e92
599638,00000000000000000000011679cdea1ace135eb018393b0124c22e270a028210
626740,0000000000000000000001295d284da4495a126feb5c803adda37923dd491fa5
625273,00000000000000000000012c360cda3d75b7c06af83c5883422878e2b15c79c0
637882,000000000000000000000132b2092267bd8d9325414627c2d0dc3ff5b4e7b2b2
641109,0000000000000000000001333ed8158bdb219a195102db40b13925ec0ae8fc88
368527,00000000000000000000013712632da34788a7b4ae2cd78f7982c7036610126b
615608,00000000000000000000014bc0b06ecff006a771d9a7b3247dbeb323a9e12d4b
619919,00000000000000000000014e5dc331e8841eb0f0aed8fd22602bfaf93f7a7640
582757,000000000000000000000150ad70b6d369df271f83ca3d6ac859b800f137c9a4
638817,00000000000000000000015b80b2a0848ccefdf5fbf7ad5a24e90f157d751a5a
598482,0000000000000000000001602eb228828476fe030cdcdfb3114f2ee3fdb51d77
622241,000000000000000000000174d9b1478a84a04e978a6250ce18e1799b39ca2d87
652660,0000000000000000000001836835d7c46f837f06a149be9e8b0a24de40210624
532023,00000000000000000000018ba128090ad6dd5fbcd9d666541f060c791f86ad78
605463,0000000000000000000001906fd2657c2aa87d80a1dd31af693c7507cd31179f
572822,00000000000000000000019257b49b3f46d8f7c3d6f3d4153962f1386c19337a
649763,000000000000000000000193a0abf401558a9c125d7f94773d74b6a1329c6b41
577021,0000000000000000000001aa6b9676d6b988c0549f42047dc11ac4d7660b845e
607040,0000000000000000000001b26ca7b70da9a77187933009c7d56893ceba127501
595237,0000000000000000000001bc0f20af28f38472c0eb221137fce12b089c9cd82c
522896,0000000000000000000001d3a96927a8878a4c2b0651cd816594c80b77e29111
598034,0000000000000000000001d5b5ce68ef4f426909c6f6299a945a5a781cd1f18a
587409,0000000000000000000001f57b098911a90b164b9812304f4f7615cf9f91f66a
606267,0000000000000000000001fdb893eeb401358d00c29a9a6246054bd55a9dbf53
524513,00000000000000000000020093ea502dec7cd56a1f5e2e8dd0e3d669c47ebc0c
543672,000000000000000000000206254c56610d6bc9b64c141d7b1705ac9e8fa43b55
512588,00000000000000000000021c043e439b5f4b632389b0062306bf2d4e0b657c7c
613562,00000000000000000000021e32be09862eafb7a2fe78f2b648d7252a022dc132
620410,000000000000000000000227e4a7dbbf84953cdb4beaaabcea9a8309ed5c8e64
522090,00000000000000000000022e7a595abee58621a090c8076dac4a1c3a041079af
513032,000000000000000000000241d3121e74bbbc7a034666d6c6f4d46cac2dc2394a
491270,0000000000000000000002569beb610e7f806774fd0c650c415a78a087b9d979
........
........
........
24372,00000000ff98f141099bd4d415c14152129f0b90994ff4c954beee9f0f435647
20104,00000000ff99a9ecd6a37d2ed796caff46f5ad959d5b6682b3672916375d3090
14280,00000000ff9e75d76ba82637f82ec0ce3c0c24f7d0d7af9a65c5d370f5cdaf97
688,00000000ffa5e19e744552ad9cf9255de8ac6d4c03b9ed259fca020c9a071b1e
10997,00000000ffa6ec3c76e387d9efe6d8c0a0644042b610600ec5c211249e12f0d8
27991,00000000ffab0cc1d059461b509882d0806c1ff38d9617818246c5d884d8b5f4
22465,00000000ffabeba413276a288d957179c7a4a4088de6c22508c71698a6476cd8
25021,00000000ffac6dab77739c3fad6f6853968875c613fdfc886a6056806ccb4d69
2569,00000000ffb05d42c619788bb38636c1e59ec056ad854893cb9fcb6ceabe14dd
18143,00000000ffb062296c9d4eb5f87bbf905d30669d26eab6bced341bd3f1dba5fd
12508,00000000ffb0aaf87b2664c6fe25e112195b0d7836250f7d77493ca1afaead71
23246,00000000ffb0ddfef22ff4b3522c2593c066caa235da1ffa6668b62271258c63
12605,00000000ffb145ba5f8a00e205c1a25998c6e02e4bd3f84a8c152e74d0fff9a7
23896,00000000ffb23c30adbe040379f8a0782364a0afc0dfe3a94d99a650f4e5ea09
32169,00000000ffb62a7f74340a082d00f82dbfdcfbbdf1e0f2f5de0fbe9e91ac0a14
26790,00000000ffb6c4107cbfe90cd60e610de37bcba3ae63a4320d9df3dd904b0001
20725,00000000ffb805b552395141aec921db3877bb1220651f0f7a265e2754629562
23484,00000000ffb874ed66ee7a872921177ea3e96d281c78902e21e497b45aec9785
12798,00000000ffbc3b6502d7df969a2d5546d0fa8a6e72fa69e4e84689d43f093419
2507,00000000ffbc5bf0a1427c2542e9ae43ede6fe888582702f50c5719f67873ca3
4345,00000000ffbdef4f2a349b6ce934e80ac0965cf7f841e7ffc9c03ea78404e11a
1391,00000000ffbe97137d327beae4ec8b7d96b1eb961505d6ef1c079b587e136604
10046,00000000ffc8f2beb8c54d552bf7e0322dd3b70e7901085ed081b78b06a38567
14890,00000000ffca2b3b5df775b36415f5c19ef904dfd70a97c3dc2ea56654e87d73
8750,00000000ffcb592e42740b24ff659f3f2e5d938175fe918e7ef46efa6b86e36d
3340,00000000ffd27d3cd7323dfb9cfda5fb363cbfe283e7fb248a5bdab693055a60
29040,00000000ffd5fbc6cdb40029703159d1b1bbfcef040e76fed978cb4367dad0b0
6765,00000000ffd6cf9652a947452972f4cccfc65eed439f498abac9d250eb844ebe
12461,00000000ffd75a76e7fe0c04754f718ae5fddca404f42f60247ce454f025504e
16751,00000000ffd76c547245e25edb6ce120ffbde1254df21212c050cf81467869f5
1808,00000000ffd7ae8a525d0083ee8384e13a6cbe1f1218f9ae26a637fbc2093a05
27302,00000000ffd9ba907954169f7946abb04f507ee1f6170654ffdacd602e918f34
11561,00000000ffdace7df6579699cdd8ba7be094b5637dcecc7b381af8d83e6d60d1
1601,00000000ffdacee18e2c714d840888e87796855b446f4883c2834c6ad1b9a1c7
10376,00000000ffdb74d25c7238e3fc4d7d7244655357cc750d6be2955770a8fabdf9
17598,00000000ffdd1b6aa100d4abc38ed5618b900df7184e2f9a39d739f515d2375f
10946,00000000ffdf54f4df138ef2b35b1d6a46ff67512d09646a023b92e35ba52069
18333,00000000ffdfe5c62679db8724bd7f5c0b97636b0269a676a5149c0c2e148381
31737,00000000ffe17801335b03ff320346cdfa86609158fd1e39d8e91d77a5f2f519
31577,00000000ffe3cf2a3904cbfe6b3808936d85169a7f520ba413e3652dae432ae1
28614,00000000ffe5255e229781dd72b1626c4df90e77390d38f6337ce949bc3ff300
16250,00000000ffeb2708cfc0aa1b5db3a6eacccc9112eb10cc0de2ec3e98c6c11a7a
9396,00000000ffeb8e847d94f7085c6cc0359c77d8ebe52b9fb59b90e9ef95a23c44
31108,00000000ffeec57f29c3f3df6014774aa1a2f00114ab16bd26b0d270974cd706
1304,00000000fff0728c5da1548b4f15576aa272f0f4952031ae56a5be355310ba0a
30133,00000000fff31576873d6ec35ba6659e72193ddf621ea1edd7faa462336a2211
31664,00000000fff3d56591bc0eae8cee73a214d1d451d704f86126415ed540850df7
27948,00000000fff58158d6ce595a2976d547f16b0ec8aba64b7a9a68c78d469b54b5
5386,00000000fff91949181449d048aca5a1cc6d0e3e3f34e89c00ab2709696b8da0
32009,00000000fff9e01287736bee2fecdd88ae31b5602858c5273d43351cfadecd58
Is there any way to sort blocks from the lowest hash value to the highest? I want to see what is the lowest hash value human has generated on the proof of work procedure.
Ah OK. From your initial wording, I thought you did one dump, and then just the new data every day.
That is what I do I'm confused: what do you mean by a "full refresh"? As long as the block numbers are still consecutive, I didn't miss any data so there's no need to redo everything.Did you mean this:
Update
For the full blockdata.txt CSV, I've added a daily file with only the new lines:
Both files are updated daily. For the full blockdata.txt CSV, I've added a daily file with only the new lines:
►All-in-one file:
The file blockdata.lastday.txt will be overwritten daily.| | | | | Text file blockdata.txt blockdata.lastday.txt | | | | | Size (MB) 362 0.107 | | | | | GNU zip compressed file blockdata.txt.gz blockdata.lastday.txt.gz | | | | | Size (MB) 129 0.042 | | | | |
Any chance you can do a full refresh at the start of each month?
I do a full refresh each day.Ah OK. From your initial wording, I thought you did one dump, and then just the new data every day.
Any chance you can do a full refresh at the start of each month?
I do a full refresh each day.Quote
(you're late!)
My update starts at the same time, it just takes a while to process.Quote
And then each file after that I assume is incremental?
Are you looking to download only new additions once a month?
Any chance you can do a full refresh at the start of each month? (you're late!)
And then each file after that I assume is incremental?
And then each file after that I assume is incremental?
gz.blockchair.com now shows:
This may be the end of my updates.
Update: it works again, running an update now
I now notice the .gz-files get updated (with a new time stamp) even when fetching data failed. I won't add a check for this, so I'll leave it up to the user to check if column "id" (block number) is complete.
Quote
402 Payment Required
nginx/1.14.2
Database Dumps still shows it's available at low speed. I can't find any payment information. nginx/1.14.2
Update: it works again, running an update now
I now notice the .gz-files get updated (with a new time stamp) even when fetching data failed. I won't add a check for this, so I'll leave it up to the user to check if column "id" (block number) is complete.
For the next couple of days you can also get all funded addresses here (771 megabyte) at a fast download speed:
https://balances.crypto-nerdz.org/balances/balances-bitcoin-20200708-0000-MXSyuyTD.gz
Your quote is slightly off-topic here, but it made me realize I forgot to link List of all Bitcoin addresses with a balance. I have this data available with daily updates and it won't get locked any time soon.https://balances.crypto-nerdz.org/balances/balances-bitcoin-20200708-0000-MXSyuyTD.gz
It is fast indeed (104 MB/s)!
There's another very useful data dump on Blockchair that deserves attention: blockchair_bitcoin_addresses_latest.tsv.gz!
On many occasions I found myself searching for a complete snapshot of all funded addresses, for instance to search for a balance when you only have a partial Bitcoin address. This file takes almost a day to download at 10 kB/s.
For the next couple of days you can also get all funded addresses here (771 megabyte) at a fast download speed:On many occasions I found myself searching for a complete snapshot of all funded addresses, for instance to search for a balance when you only have a partial Bitcoin address. This file takes almost a day to download at 10 kB/s.
https://balances.crypto-nerdz.org/balances/balances-bitcoin-20200708-0000-MXSyuyTD.gz
Please note: it's not my site/data and link will probably be locked again. So if you need it I suggest you get it now.
What time will you update?
My update starts 4:20 AM Amsterdam time. My server time follows daylight saving time (twice a year it changes one hour).After I start my update, it takes 2 minutes to complete. However, in rare occasions of prolonged high server load, AWS throttles the CPU down to 20%. Combined with the high load, that can cause the update to take at least ten times longer. So keep some margin after the start of my cronjob
Full disclosure: during the daily update, I add lines to existing live TXT files, so if you download at "the wrong moment" right when I'm updating, you may miss out on the last lines. The compressed file doesn't have that risk.
The file blockdata.lastday.txt will be overwritten daily.
What time will you update?
Update
For the full blockdata.txt CSV, I've added a daily file with only the new lines:
For the full blockdata.txt CSV, I've added a daily file with only the new lines:
►All-in-one file:
The file blockdata.lastday.txt will be overwritten daily. | | | | | Text file blockdata.txt blockdata.lastday.txt | | | | | Size (MB) 362 0.107 | | | | | GNU zip compressed file blockdata.txt.gz blockdata.lastday.txt.gz | | | | | Size (MB) 129 0.042 | | | | |
can you get this data from Blockchair API (https://blockchair.com/api/docs) and put it into a CSV file?
I haven't used their API yet, and I don't really have the time to figure out how it works. At first glance, it looks like it only gives the current state. If you can figure out how I can get the data, I can probably collect it for you and convert it to CSV-format.
Hello LoyceV,
I will make one more data request if you don't mind
can you get this data from Blockchair API (https://blockchair.com/api/docs) and put it into a CSV file?
I will make one more data request if you don't mind
can you get this data from Blockchair API (https://blockchair.com/api/docs) and put it into a CSV file?
Code:
mempool_transactions — number of transactions in the mempool
mempool_size — mempool size in bytes
mempool_tps — number of transactions per second added to the mempool
mempool_total_fee_usd — sum of transaction fees in the mempool, in USD
What is this item, CDD?
I don't know, I just used Blockchair's names.I Just discovered
Quote
cdd_total float The number of destroyed coindays
https://blockchair.com/api/docs#link_001Cdd is a measure of bitcoin transaction volume.
Bitcoin days destroyed for any given transaction is calculated by taking the number of Bitcoins in a transaction and multiplying it by the number of days it has been since those coins were last spent.
https://en.bitcoin.it/wiki/Bitcoin_Days_Destroyed
What is this item, CDD?
I don't know, I just used Blockchair's names. Hello Loyce,
What is this item, CDD? I made a small research and couldn't discover what it is.
I will try to make some more graphics with this data.
Having this information in csv means that we can see what is the biggest fee (for example) or the biggest difficulty, right?
Right.
You've done a great job there, mate! Having this information in csv means that we can see what is the biggest fee (for example) or the biggest difficulty, right?
There's another very useful data dump on Blockchair that deserves attention: blockchair_bitcoin_addresses_latest.tsv.gz!
On many occasions I found myself searching for a complete snapshot of all funded addresses, for instance to search for a balance when you only have a partial Bitcoin address. This file takes almost a day to download at 10 kB/s.
Update: I made a mirror:
On many occasions I found myself searching for a complete snapshot of all funded addresses, for instance to search for a balance when you only have a partial Bitcoin address. This file takes almost a day to download at 10 kB/s.
Update: I made a mirror:
See addresses.loyce.club. I keep a few snapshots of Blockchair's daily data.
See List of all Bitcoin addresses with a balance. I would prefer to have a common and indicative variable for each data page. That is blockheight, or id depends on the way you name it.
Done! After importing, the first column can easily be deleted so this makes sense.That means the file sizes in the OP are already a few MB too small. I won't bother changing the table.
Important change: I've added a header to all files.
Quote
From OP, I think you name it as id.
I use the same column names as Blockchair uses in their data dumps.Update: cronjob activated. Daily updates start at 2:20 AM Amsterdam time. Please let me know if anything goes wrong.
I have a suggestion.
To make sure that people don't import data incorrectly from separated pages because a lack of at least one indicator, such as block height or blockid. Sometimes people might sort data descendingly or ascendingly, and by doing so they will lost original orders if data does not have a fix id variable.
I would prefer to have a common and indicative variable for each data page. That is blockheight, or id depends on the way you name it. From OP, I think you name it as id.
I meant it will be better if the data pages always include id (always fixed in the first column) besides others details in second column.
To make sure that people don't import data incorrectly from separated pages because a lack of at least one indicator, such as block height or blockid. Sometimes people might sort data descendingly or ascendingly, and by doing so they will lost original orders if data does not have a fix id variable.
I would prefer to have a common and indicative variable for each data page. That is blockheight, or id depends on the way you name it. From OP, I think you name it as id.
I meant it will be better if the data pages always include id (always fixed in the first column) besides others details in second column.
Without further ado:
The data
►Directory:
loyce.club/blockdata/
►All-in-one file:
This file is in CSV (Comma-Separated Values) format. It can be imported into a spreadsheet (this might take a while considering the size).
►Sample:
The file blockdata.txt has the following columns with data for each block, the first line is a header:
►Separate columns:
Instead of killing your spreadsheet, each column can be downloaded separately (the first line is a header). The column "id" (block number) is added to each file (except for id.txt) to detect errors. Make sure the rows are aligned when inserting the columns you need:
File sizes on May 6, 2020. I won't update the size as they grow each day.
Background
In Bitcoin's Empty Blocks Analaysis, mikeywith asked if I can help scrape data for a 10 year analysis. Tranthidung posted a link to Blockchair Database Dumps, which offers compressed block summaries with daily updates.
Updates
All files will be updated daily.
Credits
Blockchair Database Dumps has a staggering amount of data, easily accessible in daily files (at 10 kB/s). All data presented in this topic comes from Blockchair.
I've converted these files to make it easier to access all data at once.
No spam please.
Self-moderated against spam. Discussion and questions are welcome.
Related topics
Bitcoin block data available in CSV format
List of all Bitcoin addresses with a balance
List of all Bitcoin addresses ever used
The data
►Directory:
loyce.club/blockdata/
►All-in-one file:
| | | | | Text file blockdata.txt blockdata.lastday.txt | | | | | Size (MB) 362 0.107 | | | | | GNU zip compressed file blockdata.txt.gz blockdata.lastday.txt.gz | | | | | Size (MB) 129 0.042 | | | | |
This file is in CSV (Comma-Separated Values) format. It can be imported into a spreadsheet (this might take a while considering the size).
►Sample:
The file blockdata.txt has the following columns with data for each block, the first line is a header:
Code:
id,hash,time,median_time,size,stripped_size,weight,version,version_hex,version_bits,merkle_root,nonce,bits,difficulty,chainwork,coinbase_data_hex,transaction_count,witness_count,input_count,output_count,input_total,input_total_usd,output_total,output_total_usd,fee_total,fee_total_usd,fee_per_kb,fee_per_kb_usd,fee_per_kwu,fee_per_kwu_usd,cdd_total,generation,generation_usd,reward,reward_usd,guessed_miner
0,000000000019d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f,2009-01-03 18:15:05,2009-01-03 18:15:05,285,285,1140,1,1,000000000000000000000000000001,4a5e1e4baab89f3a32518a88c31bc87f618f76673e2cc77ab2127b7afdeda33b,2083236893,486604799,1,0000000000000000000000000000000000000000000000000000000100010001,04ffff001d0104455468652054696d65732030332f4a616e2f32303039204368616e63656c6c6f72206f6e206272696e6b206f66207365636f6e64206261696c6f757420666f722062616e6b73,1,0,1,1,0,0,5000000000,0.5,0,0,0,0,0,0,0,5000000000,0.5,5000000000,0.5,Unknown
1,00000000839a8e6886ab5951d76f411475428afc90947ee320161bbf18eb6048,2009-01-09 02:54:25,2009-01-09 02:54:25,215,215,860,1,1,000000000000000000000000000001,0e3e2357e806b6cdb1f70b54c3a3a17b6714ee1f0e68bebb44a74b1efd512098,2573394689,486604799,1,0000000000000000000000000000000000000000000000000000000200020002,04ffff001d0104,1,0,1,1,0,0,5000000000,0.5,0,0,0,0,0,0,0,5000000000,0.5,5000000000,0.5,Unknown
2,000000006a625f06636b8bb6ac7b960a8d03705d1ace08b1a19da3fdcc99ddbd,2009-01-09 02:55:44,2009-01-09 02:54:25,215,215,860,1,1,000000000000000000000000000001,9b0fc92260312ce44e74ef369f5c66bbb85848f2eddd5a7a1cde251e54ccfdd5,1639830024,486604799,1,0000000000000000000000000000000000000000000000000000000300030003,04ffff001d010b,1,0,1,1,0,0,5000000000,0.5,0,0,0,0,0,0,0,5000000000,0.5,5000000000,0.5,Unknown
3,0000000082b5015589a3fdf2d4baff403e6f0be035a5d9742c1cae6295464449,2009-01-09 03:02:53,2009-01-09 02:55:44,215,215,860,1,1,000000000000000000000000000001,999e1c837c76a1b7fbb7e57baf87b309960f5ffefbf2a9b95dd890602272f644,1844305925,486604799,1,0000000000000000000000000000000000000000000000000000000400040004,04ffff001d010e,1,0,1,1,0,0,5000000000,0.5,0,0,0,0,0,0,0,5000000000,0.5,5000000000,0.5,Unknown
4,000000004ebadb55ee9096c9a2f8880e09da59c0d68b1c228da88e48844a1485,2009-01-09 03:16:28,2009-01-09 02:55:44,215,215,860,1,1,000000000000000000000000000001,df2b060fa2e5e9c8ed5eaf6a45c13753ec8c63282b2688322eba40cd98ea067a,2850094635,486604799,1,0000000000000000000000000000000000000000000000000000000500050005,04ffff001d011a,1,0,1,1,0,0,5000000000,0.5,0,0,0,0,0,0,0,5000000000,0.5,5000000000,0.5,Unknown
5,000000009b7262315dbf071787ad3656097b892abffd1f95a1a022f896f533fc,2009-01-09 03:23:48,2009-01-09 03:02:53,215,215,860,1,1,000000000000000000000000000001,63522845d294ee9b0188ae5cac91bf389a0c3723f084ca1025e7d9cdfe481ce1,2011431709,486604799,1,0000000000000000000000000000000000000000000000000000000600060006,04ffff001d0120,1,0,1,1,0,0,5000000000,0.5,0,0,0,0,0,0,0,5000000000,0.5,5000000000,0.5,Unknown
6,000000003031a0e73735690c5a1ff2a4be82553b2a12b776fbd3a215dc8f778d,2009-01-09 03:29:49,2009-01-09 03:02:53,215,215,860,1,1,000000000000000000000000000001,20251a76e64e920e58291a30d4b212939aae976baca40e70818ceaa596fb9d37,2538380312,486604799,1,0000000000000000000000000000000000000000000000000000000700070007,04ffff001d0123,1,0,1,1,0,0,5000000000,0.5,0,0,0,0,0,0,0,5000000000,0.5,5000000000,0.5,Unknown
7,0000000071966c2b1d065fd446b1e485b2c9d9594acd2007ccbd5441cfc89444,2009-01-09 03:39:29,2009-01-09 03:16:28,215,215,860,1,1,000000000000000000000000000001,8aa673bc752f2851fd645d6a0a92917e967083007d9c1684f9423b100540673f,2258412857,486604799,1,0000000000000000000000000000000000000000000000000000000800080008,04ffff001d012b,1,0,1,1,0,0,5000000000,0.5,0,0,0,0,0,0,0,5000000000,0.5,5000000000,0.5,Unknown
8,00000000408c48f847aa786c2268fc3e6ec2af68e8468a34a28c61b7f1de0dc6,2009-01-09 03:45:43,2009-01-09 03:16:28,215,215,860,1,1,000000000000000000000000000001,a6f7f1c0dad0f2eb6b13c4f33de664b1b0e9f22efad5994a6d5b6086d85e85e3,1716931356,486604799,1,0000000000000000000000000000000000000000000000000000000900090009,04ffff001d012c,1,0,1,1,0,0,5000000000,0.5,0,0,0,0,0,0,0,5000000000,0.5,5000000000,0.5,Unknown
►Separate columns:
Instead of killing your spreadsheet, each column can be downloaded separately (the first line is a header). The column "id" (block number) is added to each file (except for id.txt) to detect errors. Make sure the rows are aligned when inserting the columns you need:
File sizes on May 6, 2020. I won't update the size as they grow each day.
Background
In Bitcoin's Empty Blocks Analaysis, mikeywith asked if I can help scrape data for a 10 year analysis. Tranthidung posted a link to Blockchair Database Dumps, which offers compressed block summaries with daily updates.
Updates
All files will be updated daily.
Credits
Blockchair Database Dumps has a staggering amount of data, easily accessible in daily files (at 10 kB/s). All data presented in this topic comes from Blockchair.
I've converted these files to make it easier to access all data at once.
No spam please.
Self-moderated against spam. Discussion and questions are welcome.
Related topics
Bitcoin block data available in CSV format
List of all Bitcoin addresses with a balance
List of all Bitcoin addresses ever used
Jump to: