Pages:
Author

Topic: Bitcoin block data (728 GB): inputs, outputs and transactions (Read 2842 times)

legendary
Activity: 3262
Merit: 16303
Thick-Skinned Gang Leader and Golden Feather 2021
legendary
Activity: 3262
Merit: 16303
Thick-Skinned Gang Leader and Golden Feather 2021
Bump!

Zensi never made me an offer. Blockchair also has BCH, Dash, Doge, LTC or Zcash data dumps available (for a very slow download).
legendary
Activity: 3262
Merit: 16303
Thick-Skinned Gang Leader and Golden Feather 2021
pls add all bch addresses
You can find the data at gz.blockchair.com/bitcoin-cash/. I don't really want to add hundreds of GB of Forkcoin data to my server. If you need help downloading or processing the data, feel free to make me an offer.
newbie
Activity: 8
Merit: 0
pls add all bch addresses
legendary
Activity: 3262
Merit: 16303
Thick-Skinned Gang Leader and Golden Feather 2021
I've added ethereumdata.loyce.club Abandoned! I'm out of disk space. I have a local backup, if you really need this: ask!
ethereumdata.loyce.club/calls/ (345 GB)
ethereumdata.loyce.club/erc-20_transactions/ (378 GB)
ethereumdata.loyce.club/transactions/ (71 GB)

I only added this centralized shitcoin to utilize the 2 TB storage disk, but adding several gigabytes per week filled it. This gives enough space for a long time again.
legendary
Activity: 3262
Merit: 16303
Thick-Skinned Gang Leader and Golden Feather 2021
Hi Loyce and thanks for adding "merkleroot" to your data. Would you mind to add also the rest ?
I'll create a TSV with this:
Code:
block   hash    version versionHex      merkleroot      time    mediantime      nonce   bits    difficulty      chainwork       nTx     strippedsize    size    weight
I left out the previous and next block hash to reduce the size.

It should be done in about 11 hours, which means I'll share the output tomorrow. I expect the file to be around 150 MB, so I didn't bother making smaller daily files.
Once done, this will get daily updates, up to the same block as Blockchair's data dumps (so not the latest blocks from my own Bitcoin Core).

Update
block_hash_version_versionHex_merkleroot_time_mediantime_nonce_bits_difficulty_ chainwork_nTx_strippedsize_size_weight.tsv.gz (80 MB).

I've updated the OP Smiley
hero member
Activity: 630
Merit: 649
Bitcoin g33k
Hi Loyce and thanks for adding "merkleroot" to your data. Would you mind to add also the rest ?

Code:
  "hash": "0000000000000000000590fc0f3eba193a278534220b2b37e9849e1a770ca959",
  "version": 1073733636,
  "versionHex": "3fffe004",
  "merkleroot": "1f8d213c864bfe9fb0098cecc3165cce407de88413741b0300d56ea0f4ec9c65",
  "time": 1631333672,
  "mediantime": 1631331088,
  "nonce": 2881644503,
  "bits": "170f48e4",
  "difficulty": 18415156832118.24,
  "chainwork": "0000000000000000000000000000000000000000216dd8dc61fdffabb624feeb",
  "nTx": 1276,
  "previousblockhash": "0000000000000000000aa3ce000eb559f4143be419108134e0ce71042fc636eb",
  "nextblockhash": "00000000000000000002f39baabb00ffeb47dbdb425d5077baa62c47482b7e92",
  "strippedsize": 907224,
  "size": 1276422,
  "weight": 3998094,

legendary
Activity: 3262
Merit: 16303
Thick-Skinned Gang Leader and Golden Feather 2021
From how to dump or parse information from all available blocks on a full-node, I realized there's more block data missing from these data dumps.

Example:
Code:
bitcoin-cli getblock $(./bitcoin-cli getblockhash 700000) | head -n25
{
  "hash": "0000000000000000000590fc0f3eba193a278534220b2b37e9849e1a770ca959",
  "confirmations": 57334,
  "height": 700000,
  "version": 1073733636,
  "versionHex": "3fffe004",
  "merkleroot": "1f8d213c864bfe9fb0098cecc3165cce407de88413741b0300d56ea0f4ec9c65",
  "time": 1631333672,
  "mediantime": 1631331088,
  "nonce": 2881644503,
  "bits": "170f48e4",
  "difficulty": 18415156832118.24,
  "chainwork": "0000000000000000000000000000000000000000216dd8dc61fdffabb624feeb",
  "nTx": 1276,
  "previousblockhash": "0000000000000000000aa3ce000eb559f4143be419108134e0ce71042fc636eb",
  "nextblockhash": "00000000000000000002f39baabb00ffeb47dbdb425d5077baa62c47482b7e92",
  "strippedsize": 907224,
  "size": 1276422,
  "weight": 3998094,
  "tx": [
    "1d8149eb8d8475b98113b5011cf70e0b7a4dccff71286d28b8b4b641f94f1e46",
    "ed25927576988e38e4cc8e4b19d1272c480f113fb605271b190df05aa983714e",
    "744556a5586736471d496c928ccca8fd58dadac6071394eca846c180b0dec6fe",
    "adfcbcbd4f87a725337ab0b4eb657f97123806d30ccd50fa0c107b5124692e1d",
    "afe5de49b7a84bb5d79d114601d81645264ebb4fcb8e1b45c280f6d788a8a7ba"
~

If anyone wants a dump of this data, just ask. I don't mind adding it.
legendary
Activity: 3262
Merit: 16303
Thick-Skinned Gang Leader and Golden Feather 2021
Ethereum data

I've added ethereumdata.loyce.club:
ethereumdata.loyce.club/calls/ (329 GB)
ethereumdata.loyce.club/erc-20_transactions/ (363 GB)
ethereumdata.loyce.club/transactions/ (69 GB)

I needed this data from gz.blockchair.com/ethereum/ for a project (which I abandoned before my download completed). Since I now have the data, I'll share it (downloading 762 GB at 100 kB/s took forever 3 months).

Updates
New files are added daily.

Disclaimer
I don't like Ethereum, which is a centralized shitcoin that abandoned it's one unique selling point ("code is law") the moment it was convenient for the creator. Don't waste your money on it!
And since it's a shitcoin, I don't think it deserves it's own topic.

Missing files
Those 2 files are missing from my mirror because they're corrupted:
https://gz.blockchair.com/ethereum/calls/blockchair_ethereum_calls_20200113.tsv.gz
https://gz.blockchair.com/ethereum/calls/blockchair_ethereum_calls_20211110.tsv.gz

I sent Blockchair an email about it 3 months ago, but the response I got was this:
Quote
We've checked and haven't detected the issue with the file.
If anyone has the tools to rebuild those 2 files, I'd love to add them!
legendary
Activity: 3262
Merit: 16303
Thick-Skinned Gang Leader and Golden Feather 2021
I stumbled upon something peculiar. Take blockchair_bitcoin_outputs_20220203.tsv.gz for example:
Code:
721577  f2ec8c7f07725959014613a5cf04dde4cf3079c8948bc011298479e751935fc3        11      2022-02-03 01:43:54     51200000        18970.623       bc1qt2kc82kr0wdyyyqns7qyvl377dap69ygzkpwmc      witness_v0_scripthash   00145aad83aac37b9a4210138780467e3ef37a1d1488    0       -1
721577  f2ec8c7f07725959014613a5cf04dde4cf3079c8948bc011298479e751935fc3        11      2022-02-03 01:43:54     51200000        18970.623       bc1qt2kc82kr0wdyyyqns7qyvl377dap69ygzkpwmc      witness_v0_scripthash   00145aad83aac37b9a4210138780467e3ef37a1d1488    0       -1
721577  f2ec8c7f07725959014613a5cf04dde4cf3079c8948bc011298479e751935fc3        11      2022-02-03 01:43:54     51200000        18970.623       bc1qt2kc82kr0wdyyyqns7qyvl377dap69ygzkpwmc      witness_v0_scripthash   00145aad83aac37b9a4210138780467e3ef37a1d1488    0       -1
721577  f2ec8c7f07725959014613a5cf04dde4cf3079c8948bc011298479e751935fc3        12      2022-02-03 01:43:54     51200000        18970.623       bc1qx9jm85e08jasw75g0drr7y2x9xx45xck6xvxhe      witness_v0_scripthash   00143165b3d32f3cbb077a887b463f1146298d5a1b16    0       -1
721577  f2ec8c7f07725959014613a5cf04dde4cf3079c8948bc011298479e751935fc3        13      2022-02-03 01:43:54     51200000        18970.623       bc1qmx0eraunssvj9ukel8m40cpt8ez4wxj4t2jn4q      witness_v0_scripthash   0014d99f91f793841922f2d9f9f757e02b3e45571a55    0       -1
721577  f2ec8c7f07725959014613a5cf04dde4cf3079c8948bc011298479e751935fc3        13      2022-02-03 01:43:54     51200000        18970.623       bc1qmx0eraunssvj9ukel8m40cpt8ez4wxj4t2jn4q      witness_v0_scripthash   0014d99f91f793841922f2d9f9f757e02b3e45571a55    0       -1
721577  f2ec8c7f07725959014613a5cf04dde4cf3079c8948bc011298479e751935fc3        13      2022-02-03 01:43:54     51200000        18970.623       bc1qmx0eraunssvj9ukel8m40cpt8ez4wxj4t2jn4q      witness_v0_scripthash   0014d99f91f793841922f2d9f9f757e02b3e45571a55    0       -1
721577  f2ec8c7f07725959014613a5cf04dde4cf3079c8948bc011298479e751935fc3        14      2022-02-03 01:43:54     51200000        18970.623       bc1qr4jgu3t5fnjrcux646kssfmavsw5zftmj4tsc6      witness_v0_scripthash   00141d648e45744ce43c70daaead08277d641d41257b    0       -1
721577  f2ec8c7f07725959014613a5cf04dde4cf3079c8948bc011298479e751935fc3        15      2022-02-03 01:43:54     51200000        18970.623       bc1qw6dnjhw8qjyxtszn950l5zh57x60tm5lcsdtkr      witness_v0_scripthash   0014769b395dc7048865c0532d1ffa0af4f1b4f5ee9f    0       -1
721577  f2ec8c7f07725959014613a5cf04dde4cf3079c8948bc011298479e751935fc3        15      2022-02-03 01:43:54     51200000        18970.623       bc1qw6dnjhw8qjyxtszn950l5zh57x60tm5lcsdtkr      witness_v0_scripthash   0014769b395dc7048865c0532d1ffa0af4f1b4f5ee9f    0       -1
721577  f2ec8c7f07725959014613a5cf04dde4cf3079c8948bc011298479e751935fc3        15      2022-02-03 01:43:54     51200000        18970.623       bc1qw6dnjhw8qjyxtszn950l5zh57x60tm5lcsdtkr      witness_v0_scripthash   0014769b395dc7048865c0532d1ffa0af4f1b4f5ee9f    0       -1
There are many duplicated lines! For the compressed filesize it doesn't matter much, but if I remove them, the number of lines drops by 1,046,856-879,157=167,699!
I checked more archives: the older ones have only a few duplicate lines, the newer archives have tens or hundreds of thousands of duplicates.

What could be the reason? And worse: it also makes me wonder if other entries could be missing.
legendary
Activity: 3262
Merit: 16303
Thick-Skinned Gang Leader and Golden Feather 2021
Hi, what's the state of ETH addresses dump? Is it uploaded already?
All funded Ethereum addresses: ethereumdata.loyce.club/blockchair_ethereum_addresses_latest.tsv.gz (2.1 GB). I downloaded this from gz.blockchair.com/ethereum/addresses/ on July 21, 2022. There's currently no update available.
hero member
Activity: 1643
Merit: 683
LoyceV on the road. Or couch.
Hi, what's the state of ETH addresses dump? Is it uploaded already?
I have it, updated until June only, but not online yet. And I'm currently sailing so can't access it.
The data I meant is the full Ethereum transaction data, about 800 GB. That will take my home internet a while to upload.

Quote
Do you want to talk about doing it in DB?
I'm still a total noob, but would be good to learn.
legendary
Activity: 952
Merit: 1367
Update: the server is currenly offline for an upgrade Cheesy With more disk space, I can add Ethereum data from Blockchair (which I have locally already), and later Dogecoin data too.

Hi, what's the state of ETH addresses dump? Is it uploaded already?

Problem:
There's a lot of data, and I don't do databases. I was actually running out of disk space to sort the data, so this upgrade came at the right time.

Do you want to talk about doing it in DB?
legendary
Activity: 3262
Merit: 16303
Thick-Skinned Gang Leader and Golden Feather 2021
I just got a very nice Xeon-powered dedicated server (no more VPS!) from an anonymous donation, so I'm covered for now.
Update: the server is currenly offline for an upgrade Cheesy With more disk space, I can add Ethereum data from Blockchair (which I have locally already), and later Dogecoin data too.



I've been playing with an idea: I want to make a graph of funded (potential) ChipMixer chips over the years, with daily data.

Assumptions:
I'll start by looking for addresses that received 0.512BTC. I'll exclude all addresses that received more than one transaction (ever), and I'll count the chips from the day they were funded until they day they're emptied. In Bitcoin's early years, before ChipMixer even existed, many potential chips were emptied the same day again. Those won't be counted.

Problem:
There's a lot of data, and I don't do databases. I was actually running out of disk space to sort the data, so this upgrade came at the right time.
legendary
Activity: 952
Merit: 1367
That's exactly what I was thinking about.
Is it? It doesn't have the block hashes you asked for, only txids and block numbers.

Yes, I was just stunned by amount of data to download. I need only txids.
With block hashes it is indeed simpler.
(you may delete the file)
legendary
Activity: 3262
Merit: 16303
Thick-Skinned Gang Leader and Golden Feather 2021
Perfect, great, fantastic, thank you!
That's exactly what I was thinking about.
Is it? It doesn't have the block hashes you asked for, only txids and block numbers.

Quote
I owe you beer & frites.
No worries, I've had my fair share today already Cheesy
legendary
Activity: 952
Merit: 1367
See: blockdata.loyce.club/PawGo.tsv or blockdata.loyce.club/PawGo.tsv.gz. It's 27 GB now, hashes don't compress very well. It's scheduled to be deleted in 7 days.

Quote
Otherwise I would have to download packs like from http://blockdata.loyce.club/ , decompress and parse tsv (which seems to be the simplest solution Wink
That's what I would do Smiley

Perfect, great, fantastic, thank you!
That's exactly what I was thinking about. I owe you beer & frites.
legendary
Activity: 3262
Merit: 16303
Thick-Skinned Gang Leader and Golden Feather 2021
Would it be possible to prepare a single file with:
- all block hashes
- all transaction IDs

Or maybe you have such a script I may execute on full node.
What format are you looking for? Just a long list of hashes, or do you need to know which txid belong to which block?
I think the easiest way is to get the data from /transactions/ (55 GB) and /blocks/.

I can save you a 55 GB download if you get me the format you want.
To test, this is running now in /transactions/:
Code:
for file in `ls`; do echo "Now processing $file"; gunzip -c $file | grep -v 'block_id' | cut -f1-2 >> ../PawGo.tsv; done; gzip ../PawGo.tsv
See: blockdata.loyce.club/PawGo.tsv or blockdata.loyce.club/PawGo.tsv.gz. It's 27 GB now, hashes don't compress very well. It's scheduled to be deleted in 7 days.

Quote
Otherwise I would have to download packs like from http://blockdata.loyce.club/ , decompress and parse tsv (which seems to be the simplest solution Wink
That's what I would do Smiley

Quote
Do you plan to backup /blocks/ folder from blockchair?
No need: these files are tiny, so downloading them from Blockchair directly shouldn't take too long anyway.
legendary
Activity: 952
Merit: 1367
Hi Loyce,

Would it be possible to prepare a single file with:
- all block hashes
- all transaction IDs

Or maybe you have such a script I may execute on full node. Otherwise I would have to download packs like from http://blockdata.loyce.club/ , decompress and parse tsv (which seems to be the simplest solution Wink
Do you plan to backup /blocks/ folder from blockchair?
legendary
Activity: 3262
Merit: 16303
Thick-Skinned Gang Leader and Golden Feather 2021
In the list of founded addresses there is unnecessary line "address"  (kind of column header) between addresses "3.." and "bc..", line 31808561. Like you had separated lists and you concatenated them.
Thanks! I'll take this to my other topic.
Pages:
Jump to: