Bitcoin Forum>Bitcoin>Project Development
Pages:
Author

Topic: Bitcoin block data available in CSV format - page 2. (Read 2118 times)

bitmover
legendary
Activity: 2352
Merit: 6089
bitcoindata.science
Re: Bitcoin block data available in CSV format
December 22, 2020, 02:32:56 PM
#31
Quote from: LoyceV on December 18, 2020, 05:06:56 AM
Quote from: bitmover on December 17, 2020, 05:55:47 PM
Code:
Header set Access-Control-Allow-Origin "*"
I'm not willing to allow "*", so I added this:
Code:
        # Added 21-8-2020 based on https://bitcointalk.org/index.php?topic=5167469.msg55037886#msg55037886
        Header set Access-Control-Allow-Origin "https://bitcointalk.org"
        # Added 18-12-2020 based on https://bitcointalk.org/index.php?topic=5246271.msg55854208#msg55854208
        Header set Access-Control-Allow-Origin "https://bitcoindata.science"
Let me know if it still doesn't work.

It is working. Smiley
THank you.
I think I did something nice. In time I will add more data to it, maybe week, month... ?

I just need to finish building some nice statistics and charts, then I can move on to more data. I will slowly scale it.
LoyceV
legendary
Activity: 3290
Merit: 16489
Thick-Skinned Gang Leader and Golden Feather 2021
Re: Bitcoin block data available in CSV format
December 18, 2020, 05:06:56 AM
#30
Quote from: bitmover on December 17, 2020, 05:55:47 PM
Code:
Header set Access-Control-Allow-Origin "*"
I'm not willing to allow "*", so I added this:
Code:
        # Added 21-8-2020 based on https://bitcointalk.org/index.php?topic=5167469.msg55037886#msg55037886
        Header set Access-Control-Allow-Origin "https://bitcointalk.org"
        # Added 18-12-2020 based on https://bitcointalk.org/index.php?topic=5246271.msg55854208#msg55854208
        Header set Access-Control-Allow-Origin "https://bitcoindata.science"
Let me know if it still doesn't work.
bitmover
legendary
Activity: 2352
Merit: 6089
bitcoindata.science
Re: Bitcoin block data available in CSV format
December 17, 2020, 05:55:47 PM
#29
Hello LoyceV,

I have been working with your CSV and noticied that your server is blocking cross-origin requests (CORS). I don't know if that is your intention or not. My server was blocking CORS as well by default.

I cannot make simple requests like this one, that they will be blocked:
Code:
$.ajax({
    url: 'https://loyce.club/blockdata/blockdata.lastday.txt',
    dataType: 'text',
  })

You can make tests here. This is a public library for CSV parsing. They have a demo, and if you try to load your CSV it will show this

"Access to XMLHttpRequest at 'https://loyce.club/blockdata/blockdata.lastday.txt' from origin 'null' has been blocked by CORS policy: The 'Access-Control-Allow-Origin' header has a value 'https://bitcointalk.org' that is not equal to the supplied origin."

I solved that error in my server just by adding this line to my .htaccess file.
Code:
Header set Access-Control-Allow-Origin "*"

You can read more here:
https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS/Errors/CORSAllowOriginNotMatchingOrigin
https://stackoverflow.com/questions/13421463/htaccess-access-control-allow-origin
bitmover
legendary
Activity: 2352
Merit: 6089
bitcoindata.science
Re: Bitcoin block data available in CSV format
December 13, 2020, 04:29:45 PM
#28
Quote from: LoyceV on December 13, 2020, 04:23:51 PM

Knock yourself out Smiley I can't imagine much interesting graphs from only one day of data though.

The lastday.txt file is meant to add to the full file if you download an update once a day. But it wouldn't make sense to download more than once a day.
I Will try.

I think I can make something interesting  with daily data.
Additionally, as it is a small file it can be easily downloaded.

LoyceV
legendary
Activity: 3290
Merit: 16489
Thick-Skinned Gang Leader and Golden Feather 2021
Re: Bitcoin block data available in CSV format
December 13, 2020, 04:23:51 PM
#27
Quote from: bitmover on December 13, 2020, 04:17:03 PM
How often do you update this file?
https://loyce.club/blockdata/blockdata.lastday.txt
Daily.

Quote
Can I use this in my website? I plan to make some charts with it. Everyone who enter the url would make one request. I dont have many visitors.. yet (I hope lol)
Knock yourself out Smiley I can't imagine much interesting graphs from only one day of data though.

The lastday.txt file is meant to add to the full file if you download an update once a day. But it wouldn't make sense to download more than once a day.
bitmover
legendary
Activity: 2352
Merit: 6089
bitcoindata.science
Re: Bitcoin block data available in CSV format
December 13, 2020, 04:17:03 PM
#26
Hello LoyceV
How often do you update this file?
https://loyce.club/blockdata/blockdata.lastday.txt

Can I use this in my website? I plan to make some charts with it. Everyone who enter the url would make one request. I dont have many visitors.. yet (I hope lol)
LoyceV
legendary
Activity: 3290
Merit: 16489
Thick-Skinned Gang Leader and Golden Feather 2021
Re: Bitcoin block data available in CSV format
December 05, 2020, 03:02:14 AM
#25
Quote from: Vod on December 02, 2020, 08:16:18 PM
Do you have any plans of capturing the transactions or outputs in a similar manner as the blocks?
I'd love to, but I'd need much more server space for this.
I have Blockchair.com's "outputs" downloaded. It takes 115 GB, and Blockchair's "inputs" (which I don't have yet) is ~3.5 times larger.
"Outputs" has files like this:
Code:
block_id        transaction_hash        index   time    value   value_usd       recipient       type    script_hex      is_from_coinbase        is_spendable
653356  dd9ffd382bf83248711ea47abe5b8ac21d9b9a34c6c7abdda6a0b9b1e99adc60        0       2020-10-19 00:02:17     688376206       79156.1016      1KFHE7w8BhaENAswwryaoccDb6qcT6DbYY      pubkeyhash      76a914c825a1ecf2a6830c4401620c3a16f1995057c2ab88ac      1       -1
653356  dd9ffd382bf83248711ea47abe5b8ac21d9b9a34c6c7abdda6a0b9b1e99adc60        1       2020-10-19 00:02:17     0       0       d-0ca2834cdc0b2ca644a7aa6c6c6b571e      nulldata        6a24aa21a9ed79186d8dd65899b51c686badefe90276e5dc2879c2ca48f05d25830989285e3e080000000000000000  1       0
653356  dd9ffd382bf83248711ea47abe5b8ac21d9b9a34c6c7abdda6a0b9b1e99adc60        2       2020-10-19 00:02:17     0       0       d-7ac1238b74a3ecbfab24678cb7010a8d      nulldata        6a4c2952534b424c4f434b3a03b3765c82064ea9fb16ab2211f12c3a6e7b13586cb2e474e7372531002a982c        1       0
653356  dd9ffd382bf83248711ea47abe5b8ac21d9b9a34c6c7abdda6a0b9b1e99adc60        3       2020-10-19 00:02:17     0       0       d-00eb8b6bf9a16a7d66d3739eadcf9c09      nulldata        6a24b9e11b6d01b8daa2d578f9b3cc56d35b0a71ab02e7878e20773ac2d34bf50dce8f9cd4b8    1       0
653356  81f82b69058e7e7d766928183ec05f6dfac5a71b950ac388f5301ba9be2af74e        0       2020-10-19 00:02:17     7890000 907.2679        1GRh1yKNhxjfKmPATYdYAC7xwUHvv6doVN      pubkeyhash      76a914a935462d90eb370ed808de551b57df5eb9016bf588ac      0       -1
653356  1f7c023b5fa5c9389db2977578f6bba48c9aaa3627be2bb87686666949d9a6c7        0       2020-10-19 00:02:17     872314  100.307 1G4Y3mFLamLffj4p248iNWYiWt7A5v2MTS      pubkeyhash      76a914a5350d4988f7607cc8001c7436e2750720dd395c88ac      0       -1
653356  1f7c023b5fa5c9389db2977578f6bba48c9aaa3627be2bb87686666949d9a6c7        1       2020-10-19 00:02:17     97406   11.2007 1G2AUdDQDFBDKgx8zyJMVJ2mh65KFs4Qs5      pubkeyhash      76a914a4c2368f121f9eabf345b3ec7af3c851da7ebaf488ac      0       -1
653356  1f7c023b5fa5c9389db2977578f6bba48c9aaa3627be2bb87686666949d9a6c7        1       2020-10-19 00:02:17     97406   11.2007 1G2AUdDQDFBDKgx8zyJMVJ2mh65KFs4Qs5      pubkeyhash      76a914a4c2368f121f9eabf345b3ec7af3c851da7ebaf488ac      0       -1
653356  1f7c023b5fa5c9389db2977578f6bba48c9aaa3627be2bb87686666949d9a6c7        1       2020-10-19 00:02:17     97406   11.2007 1G2AUdDQDFBDKgx8zyJMVJ2mh65KFs4Qs5      pubkeyhash      76a914a4c2368f121f9eabf345b3ec7af3c851da7ebaf488ac      0       -1
I'm not sure how useful it would be for any purpose to download CSV-files with a billion rows and many gigabytes large.

Someone requested similar data last week, so I made one large CSV-file with this data. See all_Bitcoin_txids.txt.bz2 (19 GB) (this hosting expires January 23, 2021). The header is not included:
Code:
block_id,transaction_hash
The file itself starts with this:
Code:
0,4a5e1e4baab89f3a32518a88c31bc87f618f76673e2cc77ab2127b7afdeda33b
1,0e3e2357e806b6cdb1f70b54c3a3a17b6714ee1f0e68bebb44a74b1efd512098
2,9b0fc92260312ce44e74ef369f5c66bbb85848f2eddd5a7a1cde251e54ccfdd5
3,999e1c837c76a1b7fbb7e57baf87b309960f5ffefbf2a9b95dd890602272f644
4,df2b060fa2e5e9c8ed5eaf6a45c13753ec8c63282b2688322eba40cd98ea067a
5,63522845d294ee9b0188ae5cac91bf389a0c3723f084ca1025e7d9cdfe481ce1
6,20251a76e64e920e58291a30d4b212939aae976baca40e70818ceaa596fb9d37
7,8aa673bc752f2851fd645d6a0a92917e967083007d9c1684f9423b100540673f
8,a6f7f1c0dad0f2eb6b13c4f33de664b1b0e9f22efad5994a6d5b6086d85e85e3
9,0437cd7f8525ceed2324359c2d0ba26006d92d856a9c20fa0241106ee5a597c9
This data is updated until blockchair_bitcoin_outputs_20201127.tsv.gz.

I'd like to have Blockchair's "outputs" on my own VPS, and while I'm at it "inputs" too, so I have it available with high bandwidth when needed. Downloading "inputs" will take about 6 weeks.
But I haven't decided on a webhost yet. It'll instantly fill 0.5 TB and if I start this, I want it to be future proof (and not too expensive).
Vod
legendary
Activity: 3668
Merit: 3010
Licking my boob since 1970
Re: Bitcoin block data available in CSV format
December 02, 2020, 08:16:18 PM
#24
LoyceV,

Do you have any plans of capturing the transactions or outputs in a similar manner as the blocks?

LoyceV
legendary
Activity: 3290
Merit: 16489
Thick-Skinned Gang Leader and Golden Feather 2021
Re: Bitcoin block data available in CSV format
November 04, 2020, 04:10:24 PM
#23
Quote from: BlackHatCoiner on November 04, 2020, 03:43:23 PM
Is there any way to sort blocks from the lowest hash value to the highest?
sort?
Code:
wget https://loyce.club/blockdata/hash.txt.gz -O - | gunzip | grep -v "^id,hash$" | sort -t, -k2 | head

Quote
I want to see what is the lowest hash value human has generated on the proof of work procedure.
000000000000000000000003681c2df35533c9578fb6aace040b0dfe0d446413, block 634842: "Mined on 2020-06-15 09:23 (5 months ago)"

More results:
Code:
id,hash
634842,000000000000000000000003681c2df35533c9578fb6aace040b0dfe0d446413
585774,000000000000000000000019b43763eb4519f4fe65eae9be90fe73117b89026d
625857,000000000000000000000030f8cf8e0a76db53525aff8d56dcfdf4c74fc7878c
622050,000000000000000000000031a10e42c80137b3c3ad3e15c5dfb4ea213c83e497
649141,000000000000000000000084dc46d0da1c920a9b1ce16d91159655cfe1055482
593568,00000000000000000000008b80d0529bc20ac2c580f85acd2aed0baf5b82bdf5
622125,0000000000000000000000a773ce38733b4f36cf2e52581e055e3a4216291256
515910,0000000000000000000000ab789f6d71d9642ae3f697975ccd00afcb98fe6bd2
500174,0000000000000000000000bb5b432a764ad6c7acf677dcd99161abfdf68e698e
643346,0000000000000000000000c87731a1d688f3671d44ef862fe0ee2e74e1a845df
556828,0000000000000000000000ce3e3d62bb5575c1dccfca1ab635412b2c7818b3b8
600225,0000000000000000000000db2867af8040293a4a38db547883a053e0852d85f3
616078,0000000000000000000000e485cca75b9e9a48a6b532d21582ebfb2ebd87470a
575171,0000000000000000000000eb502db6b128d84c1f65943d55b31dc17871528ff9
643496,0000000000000000000000f3c07c18949f337fd5dafc908d3b72bd55684b08d2
620586,0000000000000000000000f4785d6f53ad5adccb7e4cfe31f4b207cf3b5b6329
641181,0000000000000000000000f9794a504f24ef5a8376bd71999f5c1be0cb030178
458091,00000000000000000000011246f099d94f91628d71c9d75ad2f9a06e2beb7e92
599638,00000000000000000000011679cdea1ace135eb018393b0124c22e270a028210
626740,0000000000000000000001295d284da4495a126feb5c803adda37923dd491fa5
625273,00000000000000000000012c360cda3d75b7c06af83c5883422878e2b15c79c0
637882,000000000000000000000132b2092267bd8d9325414627c2d0dc3ff5b4e7b2b2
641109,0000000000000000000001333ed8158bdb219a195102db40b13925ec0ae8fc88
368527,00000000000000000000013712632da34788a7b4ae2cd78f7982c7036610126b
615608,00000000000000000000014bc0b06ecff006a771d9a7b3247dbeb323a9e12d4b
619919,00000000000000000000014e5dc331e8841eb0f0aed8fd22602bfaf93f7a7640
582757,000000000000000000000150ad70b6d369df271f83ca3d6ac859b800f137c9a4
638817,00000000000000000000015b80b2a0848ccefdf5fbf7ad5a24e90f157d751a5a
598482,0000000000000000000001602eb228828476fe030cdcdfb3114f2ee3fdb51d77
622241,000000000000000000000174d9b1478a84a04e978a6250ce18e1799b39ca2d87
652660,0000000000000000000001836835d7c46f837f06a149be9e8b0a24de40210624
532023,00000000000000000000018ba128090ad6dd5fbcd9d666541f060c791f86ad78
605463,0000000000000000000001906fd2657c2aa87d80a1dd31af693c7507cd31179f
572822,00000000000000000000019257b49b3f46d8f7c3d6f3d4153962f1386c19337a
649763,000000000000000000000193a0abf401558a9c125d7f94773d74b6a1329c6b41
577021,0000000000000000000001aa6b9676d6b988c0549f42047dc11ac4d7660b845e
607040,0000000000000000000001b26ca7b70da9a77187933009c7d56893ceba127501
595237,0000000000000000000001bc0f20af28f38472c0eb221137fce12b089c9cd82c
522896,0000000000000000000001d3a96927a8878a4c2b0651cd816594c80b77e29111
598034,0000000000000000000001d5b5ce68ef4f426909c6f6299a945a5a781cd1f18a
587409,0000000000000000000001f57b098911a90b164b9812304f4f7615cf9f91f66a
606267,0000000000000000000001fdb893eeb401358d00c29a9a6246054bd55a9dbf53
524513,00000000000000000000020093ea502dec7cd56a1f5e2e8dd0e3d669c47ebc0c
543672,000000000000000000000206254c56610d6bc9b64c141d7b1705ac9e8fa43b55
512588,00000000000000000000021c043e439b5f4b632389b0062306bf2d4e0b657c7c
613562,00000000000000000000021e32be09862eafb7a2fe78f2b648d7252a022dc132
620410,000000000000000000000227e4a7dbbf84953cdb4beaaabcea9a8309ed5c8e64
522090,00000000000000000000022e7a595abee58621a090c8076dac4a1c3a041079af
513032,000000000000000000000241d3121e74bbbc7a034666d6c6f4d46cac2dc2394a
491270,0000000000000000000002569beb610e7f806774fd0c650c415a78a087b9d979
........
........
........
24372,00000000ff98f141099bd4d415c14152129f0b90994ff4c954beee9f0f435647
20104,00000000ff99a9ecd6a37d2ed796caff46f5ad959d5b6682b3672916375d3090
14280,00000000ff9e75d76ba82637f82ec0ce3c0c24f7d0d7af9a65c5d370f5cdaf97
688,00000000ffa5e19e744552ad9cf9255de8ac6d4c03b9ed259fca020c9a071b1e
10997,00000000ffa6ec3c76e387d9efe6d8c0a0644042b610600ec5c211249e12f0d8
27991,00000000ffab0cc1d059461b509882d0806c1ff38d9617818246c5d884d8b5f4
22465,00000000ffabeba413276a288d957179c7a4a4088de6c22508c71698a6476cd8
25021,00000000ffac6dab77739c3fad6f6853968875c613fdfc886a6056806ccb4d69
2569,00000000ffb05d42c619788bb38636c1e59ec056ad854893cb9fcb6ceabe14dd
18143,00000000ffb062296c9d4eb5f87bbf905d30669d26eab6bced341bd3f1dba5fd
12508,00000000ffb0aaf87b2664c6fe25e112195b0d7836250f7d77493ca1afaead71
23246,00000000ffb0ddfef22ff4b3522c2593c066caa235da1ffa6668b62271258c63
12605,00000000ffb145ba5f8a00e205c1a25998c6e02e4bd3f84a8c152e74d0fff9a7
23896,00000000ffb23c30adbe040379f8a0782364a0afc0dfe3a94d99a650f4e5ea09
32169,00000000ffb62a7f74340a082d00f82dbfdcfbbdf1e0f2f5de0fbe9e91ac0a14
26790,00000000ffb6c4107cbfe90cd60e610de37bcba3ae63a4320d9df3dd904b0001
20725,00000000ffb805b552395141aec921db3877bb1220651f0f7a265e2754629562
23484,00000000ffb874ed66ee7a872921177ea3e96d281c78902e21e497b45aec9785
12798,00000000ffbc3b6502d7df969a2d5546d0fa8a6e72fa69e4e84689d43f093419
2507,00000000ffbc5bf0a1427c2542e9ae43ede6fe888582702f50c5719f67873ca3
4345,00000000ffbdef4f2a349b6ce934e80ac0965cf7f841e7ffc9c03ea78404e11a
1391,00000000ffbe97137d327beae4ec8b7d96b1eb961505d6ef1c079b587e136604
10046,00000000ffc8f2beb8c54d552bf7e0322dd3b70e7901085ed081b78b06a38567
14890,00000000ffca2b3b5df775b36415f5c19ef904dfd70a97c3dc2ea56654e87d73
8750,00000000ffcb592e42740b24ff659f3f2e5d938175fe918e7ef46efa6b86e36d
3340,00000000ffd27d3cd7323dfb9cfda5fb363cbfe283e7fb248a5bdab693055a60
29040,00000000ffd5fbc6cdb40029703159d1b1bbfcef040e76fed978cb4367dad0b0
6765,00000000ffd6cf9652a947452972f4cccfc65eed439f498abac9d250eb844ebe
12461,00000000ffd75a76e7fe0c04754f718ae5fddca404f42f60247ce454f025504e
16751,00000000ffd76c547245e25edb6ce120ffbde1254df21212c050cf81467869f5
1808,00000000ffd7ae8a525d0083ee8384e13a6cbe1f1218f9ae26a637fbc2093a05
27302,00000000ffd9ba907954169f7946abb04f507ee1f6170654ffdacd602e918f34
11561,00000000ffdace7df6579699cdd8ba7be094b5637dcecc7b381af8d83e6d60d1
1601,00000000ffdacee18e2c714d840888e87796855b446f4883c2834c6ad1b9a1c7
10376,00000000ffdb74d25c7238e3fc4d7d7244655357cc750d6be2955770a8fabdf9
17598,00000000ffdd1b6aa100d4abc38ed5618b900df7184e2f9a39d739f515d2375f
10946,00000000ffdf54f4df138ef2b35b1d6a46ff67512d09646a023b92e35ba52069
18333,00000000ffdfe5c62679db8724bd7f5c0b97636b0269a676a5149c0c2e148381
31737,00000000ffe17801335b03ff320346cdfa86609158fd1e39d8e91d77a5f2f519
31577,00000000ffe3cf2a3904cbfe6b3808936d85169a7f520ba413e3652dae432ae1
28614,00000000ffe5255e229781dd72b1626c4df90e77390d38f6337ce949bc3ff300
16250,00000000ffeb2708cfc0aa1b5db3a6eacccc9112eb10cc0de2ec3e98c6c11a7a
9396,00000000ffeb8e847d94f7085c6cc0359c77d8ebe52b9fb59b90e9ef95a23c44
31108,00000000ffeec57f29c3f3df6014774aa1a2f00114ab16bd26b0d270974cd706
1304,00000000fff0728c5da1548b4f15576aa272f0f4952031ae56a5be355310ba0a
30133,00000000fff31576873d6ec35ba6659e72193ddf621ea1edd7faa462336a2211
31664,00000000fff3d56591bc0eae8cee73a214d1d451d704f86126415ed540850df7
27948,00000000fff58158d6ce595a2976d547f16b0ec8aba64b7a9a68c78d469b54b5
5386,00000000fff91949181449d048aca5a1cc6d0e3e3f34e89c00ab2709696b8da0
32009,00000000fff9e01287736bee2fecdd88ae31b5602858c5273d43351cfadecd58
BlackHatCoiner
legendary
Activity: 1512
Merit: 7340
Farewell, Leo
Re: Bitcoin block data available in CSV format
November 04, 2020, 03:43:23 PM
#22
Is there any way to sort blocks from the lowest hash value to the highest? I want to see what is the lowest hash value human has generated on the proof of work procedure.
LoyceV
legendary
Activity: 3290
Merit: 16489
Thick-Skinned Gang Leader and Golden Feather 2021
Re: Bitcoin block data available in CSV format
August 03, 2020, 02:25:10 AM
#21
Quote from: Vod on August 02, 2020, 05:49:40 PM
Ah OK.  From your initial wording, I thought you did one dump, and then just the new data every day.  Smiley
That is what I do Smiley I'm confused: what do you mean by a "full refresh"? As long as the block numbers are still consecutive, I didn't miss any data so there's no need to redo everything.

Did you mean this:
Quote from: LoyceV on July 05, 2020, 01:46:12 AM
Update
For the full blockdata.txt CSV, I've added a daily file with only the new lines:
Quote from: LoyceV on May 06, 2020, 07:14:41 AM
►All-in-one file:
|
|
|
Text file
blockdata.txt
blockdata.lastday.txt
|
|
|
Size (MB)
362
0.107
|
|
|
GNU zip compressed file
blockdata.txt.gz
blockdata.lastday.txt.gz
|
|
|
Size (MB)
129
0.042
|
|
|
The file blockdata.lastday.txt will be overwritten daily.
Both files are updated daily.
Vod
legendary
Activity: 3668
Merit: 3010
Licking my boob since 1970
Re: Bitcoin block data available in CSV format
August 02, 2020, 05:49:40 PM
#20
Quote from: LoyceV on August 02, 2020, 07:18:52 AM
Quote from: Vod on August 01, 2020, 10:59:33 PM
Any chance you can do a full refresh at the start of each month?
I do a full refresh each day.

Ah OK.  From your initial wording, I thought you did one dump, and then just the new data every day.  Smiley
LoyceV
legendary
Activity: 3290
Merit: 16489
Thick-Skinned Gang Leader and Golden Feather 2021
Re: Bitcoin block data available in CSV format
August 02, 2020, 07:18:52 AM
#19
Quote from: Vod on August 01, 2020, 10:59:33 PM
Any chance you can do a full refresh at the start of each month?
I do a full refresh each day.

Quote
(you're late!)
My update starts at the same time, it just takes a while to process.

Quote
And then each file after that I assume is incremental?
Are you looking to download only new additions once a month?
Vod
legendary
Activity: 3668
Merit: 3010
Licking my boob since 1970
Re: Bitcoin block data available in CSV format
August 01, 2020, 10:59:33 PM
#18
Any chance you can do a full refresh at the start of each month?  (you're late!)

And then each file after that I assume is incremental? 
LoyceV
legendary
Activity: 3290
Merit: 16489
Thick-Skinned Gang Leader and Golden Feather 2021
Re: Bitcoin block data available in CSV format
August 01, 2020, 04:49:23 AM
#17
gz.blockchair.com now shows:
Quote
402 Payment Required
nginx/1.14.2
Database Dumps still shows it's available at low speed. I can't find any payment information. This may be the end of my updates.
Update: it works again, running an update now Smiley

I now notice the .gz-files get updated (with a new time stamp) even when fetching data failed. I won't add a check for this, so I'll leave it up to the user to check if column "id" (block number) is complete.
LoyceV
legendary
Activity: 3290
Merit: 16489
Thick-Skinned Gang Leader and Golden Feather 2021
Re: Bitcoin block data available in CSV format
July 08, 2020, 06:22:19 AM
#16
Quote from: TheArchaeologist on July 08, 2020, 05:57:44 AM
For the next couple of days you can also get all funded addresses here (771 megabyte) at a fast download speed:
https://balances.crypto-nerdz.org/balances/balances-bitcoin-20200708-0000-MXSyuyTD.gz
Your quote is slightly off-topic here, but it made me realize I forgot to link List of all Bitcoin addresses with a balance. I have this data available with daily updates and it won't get locked any time soon.

It is fast indeed (104 MB/s)!
TheArchaeologist
sr. member
Activity: 310
Merit: 727
---------> 1231006505
Re: Bitcoin block data available in CSV format
July 08, 2020, 05:57:44 AM
#15
Quote from: LoyceV on May 15, 2020, 12:11:19 PM
There's another very useful data dump on Blockchair that deserves attention: blockchair_bitcoin_addresses_latest.tsv.gz!

On many occasions I found myself searching for a complete snapshot of all funded addresses, for instance to search for a balance when you only have a partial Bitcoin address. This file takes almost a day to download at 10 kB/s.
For the next couple of days you can also get all funded addresses here (771 megabyte) at a fast download speed:
https://balances.crypto-nerdz.org/balances/balances-bitcoin-20200708-0000-MXSyuyTD.gz

Please note: it's not my site/data and link will probably be locked again. So if you need it I suggest you get it now.
LoyceV
legendary
Activity: 3290
Merit: 16489
Thick-Skinned Gang Leader and Golden Feather 2021
Re: Bitcoin block data available in CSV format
July 05, 2020, 08:17:38 AM
#14
Quote from: Vod on July 05, 2020, 06:49:49 AM
What time will you update?
My update starts 4:20 AM Amsterdam time. My server time follows daylight saving time (twice a year it changes one hour).
After I start my update, it takes 2 minutes to complete. However, in rare occasions of prolonged high server load, AWS throttles the CPU down to 20%. Combined with the high load, that can cause the update to take at least ten times longer. So keep some margin after the start of my cronjob Smiley

Full disclosure: during the daily update, I add lines to existing live TXT files, so if you download at "the wrong moment" right when I'm updating, you may miss out on the last lines. The compressed file doesn't have that risk.
Vod
legendary
Activity: 3668
Merit: 3010
Licking my boob since 1970
Re: Bitcoin block data available in CSV format
July 05, 2020, 06:49:49 AM
#13
Quote from: LoyceV on July 05, 2020, 01:46:12 AM
The file blockdata.lastday.txt will be overwritten daily.

What time will you update?
LoyceV
legendary
Activity: 3290
Merit: 16489
Thick-Skinned Gang Leader and Golden Feather 2021
Re: Bitcoin block data available in CSV format
July 05, 2020, 01:46:12 AM
#12
Update
For the full blockdata.txt CSV, I've added a daily file with only the new lines:
Quote from: LoyceV on May 06, 2020, 07:14:41 AM
►All-in-one file:
|
|
|
Text file
blockdata.txt
blockdata.lastday.txt
|
|
|
Size (MB)
362
0.107
|
|
|
GNU zip compressed file
blockdata.txt.gz
blockdata.lastday.txt.gz
|
|
|
Size (MB)
129
0.042
|
|
|
The file blockdata.lastday.txt will be overwritten daily.
Pages:
Bitcoin Forum>Bitcoin>Project Development
Jump to: