Author

Topic: Reduced final-state blk0001.dat (with pruned index) (Read 6245 times)

legendary
Activity: 1512
Merit: 1036
I have run bitcoin-qt for a week, and it is now synchronized through block 219926.  

I have three files,
blk0001.dat 2,097,361,271 bytes,
blk0002.dat 2,097,295,438 bytes, and
blk0003.dat 1,249,842,320 bytes.

Are these three files the same for everyone?  Or a crisper question might be, are blk001.dat and blk002.dat the same for everyone who is synchronized?

Depending on how orphan blocks were received from the network and how data was reorganized on the hard drive, different users will have different checksums of blockchain files, although the contents are essentially the same. The most recent file, blk0003.dat will be changing in size every time a block is received. Data for downloadable blockchain distributions such as in this thread is specially prepared to be an efficient in-order set of blocks and to be verifiable by others.
legendary
Activity: 1246
Merit: 1002
Perhaps relevant to your interests:

https://bitcointalksearch.org/topic/m.1400898

blk0002.dat is height 210964.

I am new, and reading and assimilating...

I have run bitcoin-qt for a week, and it is now synchronized through block 219926. 

I have three files,
blk0001.dat 2,097,361,271 bytes,
blk0002.dat 2,097,295,438 bytes, and
blk0003.dat 1,249,842,320 bytes.

Are these three files the same for everyone?  Or a crisper question might be, are blk001.dat and blk002.dat the same for everyone who is synchronized?


legendary
Activity: 1512
Merit: 1036
Perhaps relevant to your interests:

https://bitcointalksearch.org/topic/m.1400898

blk0002.dat is height 210964.
legendary
Activity: 2576
Merit: 1186
Can anyone seed? My blk0001 got corrupt somehow Sad
Seeding, if you can get to me.
Thanks!

Could you redo a new torrent with pruned index up to block 210000 (checkpoint)? A torrent with only 39% of the blockchain is becoming less useful to get new users started.
These only work because they're right at 2 GB boundaries. blk0002 might be possible if I get the time...
legendary
Activity: 1512
Merit: 1036
Can anyone seed? My blk0001 got corrupt somehow Sad
Seeding, if you can get to me.

Could you redo a new torrent with pruned index up to block 210000 (checkpoint)? A torrent with only 39% of the blockchain is becoming less useful to get new users started.
legendary
Activity: 2576
Merit: 1186
Can anyone seed? My blk0001 got corrupt somehow Sad
legendary
Activity: 1512
Merit: 1036
This thread is obsolete. You'll want to check out this thread:https://bitcointalk.org/index.php?topic=117982.0
As you can see from my post in that thread: https://bitcointalksearch.org/topic/m.1303629, this torrent might be more appropriate for the time being, because it doesn't have an extra 350MB of blocks that can't be imported into Bitcoin without tricks. As blk0001.dat/block 0-188528 data will never change, and is also the size/height of the actual Bitcoin datadir file after cleanup, sharing this + another .dat torrent with more blocks 188529- for bootstrapping could mean less torrent maintenance.

I tweaked pynode mkbootstrap.py to make the dat file as described above (if height>188528, start second file), and it also produces an identical blk0001.dat, so creating future block checkpoints in a second file without needing to replace the first is easy; we may have a complete blk0002.dat by the time Bitcoin 0.8.0 comes around.
legendary
Activity: 2576
Merit: 1186
I can confirm that I independently downloaded the blockchain off p2p, processed it with mkbootstrap, and reimported the orphan-cleaned blocks into a new Bitcoin and got an identical SHA256 hash for blk0001.dat. The dat file is trustworthy (and makes mine implicity verified too...).

I will add this torrent to my blockchain download thread, where I also will have a blk0002.dat/full index for blocks up to 206000 for those that want to download a drop-in blockchain up to Oct 2012.

Right now the torrent has three seeds and one peer.
This thread is obsolete. You'll want to check out this thread:
https://bitcointalksearch.org/topic/beta-bitcoin-blockchain-torrent-117982
Not really, they serve different purposes.
legendary
Activity: 1204
Merit: 1015
I can confirm that I independently downloaded the blockchain off p2p, processed it with mkbootstrap, and reimported the orphan-cleaned blocks into a new Bitcoin and got an identical SHA256 hash for blk0001.dat. The dat file is trustworthy (and makes mine implicity verified too...).

I will add this torrent to my blockchain download thread, where I also will have a blk0002.dat/full index for blocks up to 206000 for those that want to download a drop-in blockchain up to Oct 2012.

Right now the torrent has three seeds and one peer.
This thread is obsolete. You'll want to check out this thread:
https://bitcointalksearch.org/topic/beta-bitcoin-blockchain-torrent-117982
legendary
Activity: 1512
Merit: 1036
I can confirm that I independently downloaded the blockchain off p2p, processed it with mkbootstrap, and reimported the orphan-cleaned blocks into a new Bitcoin and got an identical SHA256 hash for blk0001.dat. The dat file is trustworthy (and makes mine implicity verified too...).

I will add this torrent to my blockchain download thread, where I also will have a blk0002.dat/full index for blocks up to 206000 for those that want to download a drop-in blockchain up to Oct 2012.

Right now the torrent has three seeds and one peer.
legendary
Activity: 924
Merit: 1004
Firstbits: 1pirata
Great work guys! I hope this will eventually make it to the Satoshi client!

...

+1  Smiley
hero member
Activity: 686
Merit: 500
Bitbuy
Great work guys! I hope this will eventually make it to the Satoshi client!

Btw, the magnet link doesn't seem to be working. Figured I should let you know Wink
legendary
Activity: 1072
Merit: 1181
Orphan blocks (blocks whose parent is not known) are never stored on disk. They are kept in memory while their parents are requested.

Stale blocks (those connected to genesis, but not in the current best chain) are stored on disk, as blk000?.dat + blkindex.dat represents the whole (known part of the) block tree, not just the best chain. You need the side chains in case of reorganisations.

That said, yes optimizations are possible, and you could theoretically remove old stale blocks once the best chain has progressed enough. But as the blk000?.dat files are monolithical, you can't just remove parts of without rewriting the entire file.

I think this is a minor issue only, as the percentage of stale blocks in a typical blk000?.dat file is quite low. I also don't intend to change this in my experimental ultraprune branch. What it will do is use smaller blk000?.dat files (smaller number of blocks per file), and support removing old block files entirely (even when they still contain unspent outputs).
donator
Activity: 1464
Merit: 1047
I outlived my lifetime membership:)
Wow....strong work. Can this become a standard part of the satoshi client?
legendary
Activity: 2576
Merit: 1186
Please enlighten me, what is the technical difference between a normal blk0001.dat and your posted version? You write the blocks are ordered and no orphan blocks are in ... this means a normal node writes blocks unordered and stores orphan blocks in blk0001.dat?
Correct.
Would it be possible to have that behaviour as default for writing the chain-file? I have not a solid Bitcoin background knowledge, but why don't we store blocks in order or remove / don't store orphans (at least when switching to a new blk00x.dat file)?
Because that's how Satoshi did it. sipa's ultraprune may elminate giant blk000*.dat files though.
hero member
Activity: 772
Merit: 500
Please enlighten me, what is the technical difference between a normal blk0001.dat and your posted version? You write the blocks are ordered and no orphan blocks are in ... this means a normal node writes blocks unordered and stores orphan blocks in blk0001.dat?
Correct.

Would it be possible to have that behaviour as default for writing the chain-file? I have not a solid Bitcoin background knowledge, but why don't we store blocks in order or remove / don't store orphans (at least when switching to a new blk00x.dat file)?

Dia
legendary
Activity: 2576
Merit: 1186
Please enlighten me, what is the technical difference between a normal blk0001.dat and your posted version? You write the blocks are ordered and no orphan blocks are in ... this means a normal node writes blocks unordered and stores orphan blocks in blk0001.dat?
Correct.

I loaded it into a bitcoind with BlueMatt's transaction-index-pruning code, and produced a 151 MB blkindex.dat to go with it.
clarification: bitcoind v0.7.x and above
Huh?
legendary
Activity: 873
Merit: 1000
I loaded it into a bitcoind with BlueMatt's transaction-index-pruning code, and produced a 151 MB blkindex.dat to go with it.

clarification: bitcoind v0.7.x and above
hero member
Activity: 772
Merit: 500
Please enlighten me, what is the technical difference between a normal blk0001.dat and your posted version? You write the blocks are ordered and no orphan blocks are in ... this means a normal node writes blocks unordered and stores orphan blocks in blk0001.dat?

Dia
legendary
Activity: 2576
Merit: 1186
I built a "final-state" blk0001.dat with the first 188529 blocks (0 to 188528), in order with no orphan blocks. I loaded it into a bitcoind with BlueMatt's transaction-index-pruning code, and produced a 151 MB blkindex.dat to go with it.

This accomplishes reducing the overall disk space used by my 19 (!) bitcoin data directories:
  • This blk0001.dat can be hard linked between them all - saves about 36 GB
  • blkindex.dat is much smaller, since it has all the spent transactions pruned

So here it is for anyone else who wants it (thanks to jrmithdobbs and wizkid057 for 100mbit and gigabit seeds!):
Tip: If you copy your current blk0001.dat into your download directory and rescan, you may be able to skip a large chunk of the download.

I intentionally did not compress blk0001.dat; since you need this file anyway, the remaining data (compressed) only takes up 64 MB - please continue to seed!

Edit: Please note that if you use git master (0.7+), you must not use the included blkindex.dat, or it will break getrawtransaction - use the 0.7 instructions in the INSTALL.txt file!
Jump to: