Reduced final-state blk0001.dat (with pruned index)

deepceleron

legendary

Activity: 1512

Merit: 1036

Quote from: ProfMac on February 06, 2013, 12:39:01 PM

I have run bitcoin-qt for a week, and it is now synchronized through block 219926.

I have three files,
blk0001.dat 2,097,361,271 bytes,
blk0002.dat 2,097,295,438 bytes, and
blk0003.dat 1,249,842,320 bytes.

Are these three files the same for everyone? Or a crisper question might be, are blk001.dat and blk002.dat the same for everyone who is synchronized?

Depending on how orphan blocks were received from the network and how data was reorganized on the hard drive, different users will have different checksums of blockchain files, although the contents are essentially the same. The most recent file, blk0003.dat will be changing in size every time a block is received. Data for downloadable blockchain distributions such as in this thread is specially prepared to be an efficient in-order set of blocks and to be verifiable by others.

ProfMac

legendary

Activity: 1246

Merit: 1002

Quote from: deepceleron on February 04, 2013, 03:28:18 PM

Perhaps relevant to your interests:

https://bitcointalksearch.org/topic/m.1400898

blk0002.dat is height 210964.

I am new, and reading and assimilating...

I have run bitcoin-qt for a week, and it is now synchronized through block 219926.

I have three files,
blk0001.dat 2,097,361,271 bytes,
blk0002.dat 2,097,295,438 bytes, and
blk0003.dat 1,249,842,320 bytes.

Are these three files the same for everyone? Or a crisper question might be, are blk001.dat and blk002.dat the same for everyone who is synchronized?

deepceleron

legendary

Activity: 1512

Merit: 1036

Perhaps relevant to your interests:

https://bitcointalksearch.org/topic/m.1400898

blk0002.dat is height 210964.

Luke-Jr

legendary

Activity: 2576

Merit: 1186

Quote from: deepceleron on February 04, 2013, 12:36:07 PM

Quote from: Luke-Jr on February 04, 2013, 12:32:06 PM

Can anyone seed? My blk0001 got corrupt somehow Sad

Seeding, if you can get to me.

Thanks!

Quote from: deepceleron on February 04, 2013, 12:36:07 PM

Could you redo a new torrent with pruned index up to block 210000 (checkpoint)? A torrent with only 39% of the blockchain is becoming less useful to get new users started.

These only work because they're right at 2 GB boundaries. blk0002 might be possible if I get the time...

deepceleron

legendary

Activity: 1512

Merit: 1036

Quote from: Luke-Jr on February 04, 2013, 12:32:06 PM

Can anyone seed? My blk0001 got corrupt somehow Sad

Seeding, if you can get to me.

Could you redo a new torrent with pruned index up to block 210000 (checkpoint)? A torrent with only 39% of the blockchain is becoming less useful to get new users started.

Luke-Jr

legendary

Activity: 2576

Merit: 1186

Can anyone seed? My blk0001 got corrupt somehow Sad

deepceleron

legendary

Activity: 1512

Merit: 1036

Quote from: Maged on November 06, 2012, 08:50:00 PM

This thread is obsolete. You'll want to check out this thread:https://bitcointalk.org/index.php?topic=117982.0

As you can see from my post in that thread: https://bitcointalksearch.org/topic/m.1303629, this torrent might be more appropriate for the time being, because it doesn't have an extra 350MB of blocks that can't be imported into Bitcoin without tricks. As blk0001.dat/block 0-188528 data will never change, and is also the size/height of the actual Bitcoin datadir file after cleanup, sharing this + another .dat torrent with more blocks 188529- for bootstrapping could mean less torrent maintenance.

I tweaked pynode mkbootstrap.py to make the dat file as described above (if height>188528, start second file), and it also produces an identical blk0001.dat, so creating future block checkpoints in a second file without needing to replace the first is easy; we may have a complete blk0002.dat by the time Bitcoin 0.8.0 comes around.

Luke-Jr

legendary

Activity: 2576

Merit: 1186

Quote from: Maged on November 06, 2012, 08:50:00 PM

Quote from: deepceleron on November 06, 2012, 11:56:52 AM

I can confirm that I independently downloaded the blockchain off p2p, processed it with mkbootstrap, and reimported the orphan-cleaned blocks into a new Bitcoin and got an identical SHA256 hash for blk0001.dat. The dat file is trustworthy (and makes mine implicity verified too...).

I will add this torrent to my blockchain download thread, where I also will have a blk0002.dat/full index for blocks up to 206000 for those that want to download a drop-in blockchain up to Oct 2012.

Right now the torrent has three seeds and one peer.

This thread is obsolete. You'll want to check out this thread:
https://bitcointalksearch.org/topic/beta-bitcoin-blockchain-torrent-117982

Not really, they serve different purposes.

Maged

legendary

Activity: 1204

Merit: 1015

Quote from: deepceleron on November 06, 2012, 11:56:52 AM

I can confirm that I independently downloaded the blockchain off p2p, processed it with mkbootstrap, and reimported the orphan-cleaned blocks into a new Bitcoin and got an identical SHA256 hash for blk0001.dat. The dat file is trustworthy (and makes mine implicity verified too...).

I will add this torrent to my blockchain download thread, where I also will have a blk0002.dat/full index for blocks up to 206000 for those that want to download a drop-in blockchain up to Oct 2012.

Right now the torrent has three seeds and one peer.

This thread is obsolete. You'll want to check out this thread:
https://bitcointalksearch.org/topic/beta-bitcoin-blockchain-torrent-117982

deepceleron

legendary

Activity: 1512

Merit: 1036

I can confirm that I independently downloaded the blockchain off p2p, processed it with mkbootstrap, and reimported the orphan-cleaned blocks into a new Bitcoin and got an identical SHA256 hash for blk0001.dat. The dat file is trustworthy (and makes mine implicity verified too...).

I will add this torrent to my blockchain download thread, where I also will have a blk0002.dat/full index for blocks up to 206000 for those that want to download a drop-in blockchain up to Oct 2012.

Right now the torrent has three seeds and one peer.

paraipan

legendary

Activity: 924

Merit: 1004

Firstbits: 1pirata

Quote from: Mushoz on August 02, 2012, 12:04:11 PM

Great work guys! I hope this will eventually make it to the Satoshi client!

...

+1

Mushoz

hero member

Activity: 686

Merit: 500

Bitbuy

Great work guys! I hope this will eventually make it to the Satoshi client!

Btw, the magnet link doesn't seem to be working. Figured I should let you know Wink

Pieter Wuille

legendary

Activity: 1072

Merit: 1181

Orphan blocks (blocks whose parent is not known) are never stored on disk. They are kept in memory while their parents are requested.

Stale blocks (those connected to genesis, but not in the current best chain) are stored on disk, as blk000?.dat + blkindex.dat represents the whole (known part of the) block tree, not just the best chain. You need the side chains in case of reorganisations.

That said, yes optimizations are possible, and you could theoretically remove old stale blocks once the best chain has progressed enough. But as the blk000?.dat files are monolithical, you can't just remove parts of without rewriting the entire file.

I think this is a minor issue only, as the percentage of stale blocks in a typical blk000?.dat file is quite low. I also don't intend to change this in my experimental ultraprune branch. What it will do is use smaller blk000?.dat files (smaller number of blocks per file), and support removing old block files entirely (even when they still contain unspent outputs).

bg002h

donator

Activity: 1464

Merit: 1047

I outlived my lifetime membership:)

Wow....strong work. Can this become a standard part of the satoshi client?

Luke-Jr

legendary

Activity: 2576

Merit: 1186

Quote from: Diapolo on July 24, 2012, 03:54:19 AM

Quote from: Luke-Jr on July 24, 2012, 03:44:10 AM

Quote from: Diapolo on July 24, 2012, 02:54:57 AM

Please enlighten me, what is the technical difference between a normal blk0001.dat and your posted version? You write the blocks are ordered and no orphan blocks are in ... this means a normal node writes blocks unordered and stores orphan blocks in blk0001.dat?

Correct.

Would it be possible to have that behaviour as default for writing the chain-file? I have not a solid Bitcoin background knowledge, but why don't we store blocks in order or remove / don't store orphans (at least when switching to a new blk00x.dat file)?

Because that's how Satoshi did it. sipa's ultraprune may elminate giant blk000*.dat files though.

Diapolo

hero member

Activity: 769

Merit: 500

Quote from: Luke-Jr on July 24, 2012, 03:44:10 AM

Quote from: Diapolo on July 24, 2012, 02:54:57 AM

Please enlighten me, what is the technical difference between a normal blk0001.dat and your posted version? You write the blocks are ordered and no orphan blocks are in ... this means a normal node writes blocks unordered and stores orphan blocks in blk0001.dat?

Correct.

Would it be possible to have that behaviour as default for writing the chain-file? I have not a solid Bitcoin background knowledge, but why don't we store blocks in order or remove / don't store orphans (at least when switching to a new blk00x.dat file)?

Dia

Luke-Jr

legendary

Activity: 2576

Merit: 1186

Quote from: Diapolo on July 24, 2012, 02:54:57 AM

Please enlighten me, what is the technical difference between a normal blk0001.dat and your posted version? You write the blocks are ordered and no orphan blocks are in ... this means a normal node writes blocks unordered and stores orphan blocks in blk0001.dat?

Correct.

Quote from: koin on July 24, 2012, 02:56:13 AM

Quote from: Luke-Jr on July 23, 2012, 10:41:23 PM

I loaded it into a bitcoind with BlueMatt's transaction-index-pruning code, and produced a 151 MB blkindex.dat to go with it.

clarification: bitcoind v0.7.x and above

Huh?

koin

legendary

Activity: 873

Merit: 1000

Quote from: Luke-Jr on July 23, 2012, 10:41:23 PM

I loaded it into a bitcoind with BlueMatt's transaction-index-pruning code, and produced a 151 MB blkindex.dat to go with it.

clarification: bitcoind v0.7.x and above

Diapolo

hero member

Activity: 769

Merit: 500

Please enlighten me, what is the technical difference between a normal blk0001.dat and your posted version? You write the blocks are ordered and no orphan blocks are in ... this means a normal node writes blocks unordered and stores orphan blocks in blk0001.dat?

Dia

Luke-Jr

legendary

Activity: 2576

Merit: 1186

I built a "final-state" blk0001.dat with the first 188529 blocks (0 to 188528), in order with no orphan blocks. I loaded it into a bitcoind with BlueMatt's transaction-index-pruning code, and produced a 151 MB blkindex.dat to go with it.

This accomplishes reducing the overall disk space used by my 19 (!) bitcoin data directories:

This blk0001.dat can be hard linked between them all - saves about 36 GB
blkindex.dat is much smaller, since it has all the spent transactions pruned

So here it is for anyone else who wants it (thanks to jrmithdobbs and wizkid057 for 100mbit and gigabit seeds!):

Tip: If you copy your current blk0001.dat into your download directory and rescan, you may be able to skip a large chunk of the download.

I intentionally did not compress blk0001.dat; since you need this file anyway, the remaining data (compressed) only takes up 64 MB - please continue to seed!

Edit: Please note that if you use git master (0.7+), you must not use the included blkindex.dat, or it will break getrawtransaction - use the 0.7 instructions in the INSTALL.txt file!

Topic: Reduced final-state blk0001.dat (with pruned index) (Read 6236 times)