Topic: [BETA] Bitcoin blockchain torrent (Read 57771 times)

Hello! You can find many seedbox providers here , then click on Bitcoin on left side. The lists that appears are seedbox providers who accept Bitcoin.
Kindly,
        Muhammed Zakhir

Can anyone recommend a good seedbox provider that accepts bitcoin? I have used several and have not been happy with any of them yet.

Thanks for this! It is better than syncing through Bitcoin-qt. 
Kindly,
         Muhammed Zakhir

This thread is officially obsolete.

Non-beta torrent thread now posted at https://bitcointalksearch.org/topic/ann-bitcoin-blockchain-data-torrent-145386

I added the torrent on my seedbox. The download speed was already very good @ 1.1 MB/s! Everyone interested should be seeing even faster download speeds.
Long live Bitcoin!

Well, I see 15 seeders and 9 leechers on the current (larger) bootstrap.dat file. Maybe you are not using trackers and/or mainlineDHT or using some restrictive firewall? Also download of the torrent was reasonably fast (= ~380 kB/s on average for the remaining 2.2 GB).

As long as BitcoinQT cannot handle a compressed bootstrap.dat.xz file, I'd rather waste some bandwidth instead of confusing newies even more.

I was able to just drop the 4.7GB bootstrap.dat I already created into the torrent and seed, which I've been doing since 20 minutes after the post - and there are still the same number of seeders.

Oh, for anyone already seeding/downloading the old torrent:
Just remove the seed, open the new torrent and add it at the same location. Your torrent client then verifies the already existing blocks (~first 51.3% of the new torrent) and then loads the rest from other peers.

Currently beta-testing the following updated torrent:

SHA256 bf658c7055b733bfc15ea167f298c5599b89d220b14dbe7c8ef20b18e468c451

magnet:?xt=urn:btih:6fe493ba606847eac163baf35aae9db319735482&dn=bootstrap.dat&tr=udp://tracker.openbittorrent.com:80&tr=udp://tracker.publicbt.com:80&tr=udp://tracker.ccc.de:80&tr=udp://tracker.istole.it:80

or http://gtf.org/garzik/bitcoin/bootstrap.dat.torrent (please use magnet link, if possible)

When sufficient seeds appear, a new PGP-signed forum post will be made, obsoleting this thread.

Yep, your magnet link didn't get me anything.  I had to add some open trackers until I found some peers.

Anyway, I'll be seeding.

Here is a magnet link with a few good trackers added:

If this is an issue, you probably would want to start using multi-part compression (which is another pro fwiw).

I'll move this over here for more discussion of Torrent 2.0


"The 2GB issue", failure to import blocks after 2.0GiB on 32 bit builds, was reported by me, so I guess I'm the one that "raised" it here. Now that Bitcoin can actually use a bigger torrent and 0.8.0 has a higher checkpoint, it's finally time for a new torrent!

You should get the same 4.8GB bootstrap.dat SHA256 at that height, if you don't, we've got a big problem!

The only thing that would make a torrent I create less than "semi-official" would be if I make a compressed torrent without consensus. I am slowly convincing myself even more that a compressed torrent is preferable though. Without compression you are downloading the same amount of data as normal p2p. The compressed binary would only be repeatable on the exact xz/lzma/7zip build version and settings, but I think there are probably four people total that have run mkbootstrap anyway, so this is not important. It takes about an hour to smash the blockchain down to 60% the size at the extreme settings I've used over many trials to optimize settings.

Compressed-Pros:
-2000+ MB of uploading and downloading saved for every user,
-2000+ MB less storage used when seeding,
-won't cause problems if torrent HDD is FAT32 (under 4.0GB for now)

Compressed-Cons:
-Not as simple to use, end-user must decompress with third-party utility (although 7-zip is common and opens xz),
-More work creating torrent (doesn't matter to end-users),
-Cannot "update" torrent by simply replacing data file with newer version with additional blocks (likely to be a rare practice anyway, I am the only one seeding the old torrent right now).

Here is example compression, both require about an hour (but decompress in minutes):

xz utils 5.0.4/Win64, 3GB+ RAM required to compress (2,780,285,148 bytes)
xz --compress --keep --format=xz --check=sha256 --verbose --lzma2=dict=256MiB,nice=273,mf=bt4 bootstrap.dat

7-zip GUI win64 - 6GB+ RAM required to compress (2,769,830,975 bytes)
Format: 7z, Compression Level: Ultra, Compression method: LZMA, Dictionary size: 384MB, Word Size 273

This torrent will be updated soon, to sync with a new checkpoint in the upcoming 0.8 release.

In checkpoints.cpp there is another checkpoint at 210000 - please update the torrent!

By the way, from time to time people still are using this (I'm a more or less permanent seeder here), so I guess the experiment is working.

Additionally, it should be noted that sipa has created a pull request which should avoid seeking, thus avoiding the 2GB problem:

     https://github.com/bitcoin/bitcoin/pull/1962

blkindex.dat is not included in the torrent because that conflicts with Bitcoin's trust model.

If you're going to trust a single someone to do the indexing for you, you may as well trust the network (far less trust required) and run an SPV node (like Multibit). The purpose of the reference client is to be zero trust - not relying on any trusted data. If someone would distributed a faulty (intentional or not) set of block files with included index, and many people - in particular miners - would start using this data without verification, they could end up in a block chain fork if a transactions was included in a legitimate block that conflicts with their index.

Also, 0.8 will switch to an entirely different database layout (there won't be a blkindex.dat anymore), and validate blocks much faster (in particular when they're read from the local disk).

I made a change to the pynode mkbootstrap.py script that generates the torrent, so it splits block files at the same block as above; the hashes of the split output files are identical to the manual split method.

It's a bit less clever then tracking file sizes and auto-splitting for multiple files <2GB.

patch code:

Is it time for a torrent with more blocks (in this method?) Block 206000 is a nice recent number and makes for a 2GB + 1.5GB file.

Note that the first file can't be "bootstrap.dat", because Bitcoin actually imports this last after any other manual import statements. Patch Bitcoin to auto-import any "bootstrap.dat", "bootstrap..001", "bootstrap..002", ... in that order if they exist, instead of fixing >2GB import?

I also think the torrent should be immediately-useable blk0001.dat and blk0002.dat files including a blkindex.dat. That way one could either wait the 6+ hours to import and verify blocks (which is little improvement over p2p), or just drop the whole torrent into a Bitcoin data directory. The "split point" should correspond with the actual (earlier) block at which the Bitcoin client starts blk0002.dat when importing these cleaned blocks (188529 blocks). The index could be a spent-transaction-pruned version like luke-jr created here.

Oh, they're getting stored in a simple flat file. Well that's easy, download blocks from random peers and store them in temporary files. Then when the next block needed has been downloaded, copy it to the final block file.

(Edit - obsolete, Bitcoin issue fixed)

I have a method for splitting the torrent so it can be successfully imported to completion. It can be split into two files using the GNU split utility with this command:

split --bytes 2147570829 bootstrap.dat

This creates two files:

10/28/2012  10:12 AM     2,147,570,829 xaa
10/28/2012  10:12 AM       344,200,733 xab

sha256sum x??
fec4aa1d5a1d07b832cb22a353a689b6bf7fcb09e4822a19d9051160f195a8ef *xaa
fc07e8d67a39152a5bf1920b3bcd99d50515817ba77a28bd7d780d4cc3aa3938 *xab

This splits the torrent at block 189205, the last block that Bitcoin successfully imports out of the full 2.32GB bootstrap.dat. You can then import all torrent blocks with bitcoind or bitcoin-qt with the command:

bitcoind -loadblock=xaa -loadblock=xab -printtoconsole

(printtoconsole so you can watch the progress of block import for several hours. Use the full path to the files if needed.)

Split should be included in any GNU/Linux distribution plus MacOS. A standalone windows compile of the utility is hosted here: split.exe (72,192 bytes) (or get the GNU coreutils and dependencies full packages at http://gnuwin32.sourceforge.net/packages/coreutils.htm).

---

I found another bug/undocumented oddity; Bitcoin first processes -loadblock commands, then it looks for a bootstrap.dat file in the datadir. That means the above command will attempt to load bootstrap.dat a second time if it is located in the datadir (but will skip through all the repeated blocks quickly).

---

BTW: How do I compress the blockchain to 56% of the original size? Have 64 bit 7-zip and lots of RAM:

There's allot of pseudorandom data in the blockchain that can't be compressed, so I don't know if there will be so much difference in the file size result between different algorithms in this case, maybe we can shave off another 100 MB with a better one. It would be interesting to see a comparison you linked to with the blockchain.