Topic: [ANN] Bitcoin blockchain data torrent - page 2. (Read 211096 times)

That's exactly my point - why would I have needed trust someone else, if I had had the opportunity to download pre-verified blockchain as a package, certified by bitcoin.org ? That's just giving the people another way of getting bitcoin node up and running very fast. Precomputing means not only wasted CPU cycles, but also HDD operations, as there was reported that every block requires much more HDD transfer, than just size of the blocks.

There still will be option to run bitcoind from scratch, for any people that just want to "fire and forget", and that is good.

Yes, the overal speed depends on more than just bandwith.


You either used the "bootstrap.dat" in which case you did not safe any computation time, but rather wasted it by seperating download and verification. From the times you states and "precomputation" however I assume you downloaded a preverified blockchain. This is naturally faster, because you dont do all the work. Thus the comparisson is invalid and you need to trust the other party to not have messed with the blockchain.

You should verifiy the blocks to avoid this exact problem, trusting someone else. The same way you verify the binaries you download or the signed message you receive.


Which will require you to trust that person and their ability to secure their PGP key. People are not hesitating to run a full node because of the initial time it takes, but because of the constant time it needs to download and verify data. If it would just be 1-3 days (relativly modern desktop/laptop computer) it would not be an issue.

January 11th 2016 blockchain full download
~~~~~~~~~~~~~~~~~~~~~~~~~
Here I report my most recent performance as it will help others to decide what hardware is sufficient.  
It was to a VBOX virtual machine in linux on a quad-core i5 throttled to <75% x 4cpu cores.
Download of the whole blockchain from block 1 to present took about 2 days.  /.bitcoin/ is now about 65GB
2009 to 2013 parts of the blockchain maxed out the 4x cpu usage.
After that it slowed down, with 4x cpu usage decreased from about 95% x 4 x 2.3 GHz to (4 x 2.3 GHz about 40% of the time).
I suspect that the most recent 40GB of the blockchain were synching at a rate limited by my HDD rw speed and seek time, so a faster HDD such as a SSD would be the best available speedup, and might get a full blockchain download down to less than 40 hours.  What have others been seeing?

I don't think that the original bootstrap.dat is much help now as it only does the "easy" third at the start of the blockchain.

Let me share my story.

I have a small NAS, used also as set-top box to my TV. It runs debian. I decided to give the Bitcoin a try, and set up a node - I have plenty of space there, and this box is used sporadically.

I downloaded fresh version of Bitcoin (at that moment, 0.11.02), and started downloading the blocks normally. I have a decent speed to the internet.

I took over a month to download the whole blockchain, and I have logs to prove it. I have monitored its progress - bitcoind used just a small bandwith, but always 100% of my CPU, on the NAS. It os not very powerful CPU, I admit, so I wasn't surprised. It was the computation that caused the blockchain to load slowly.

When it finished, and started running "normally", the CPU consumption was about 20-40%. It was quite expected. But I decided to give the torrent a try, just for sake of comparison.

The torrent I took from the above linked thread. The torrent downloaded on the same box, with the same CPU and network,  in about 7 hours. Then, I needed to download missing blocks - it took about 36 hours. In total - 42 hours.

In both cases the blockchain is about 62 GB big. The MD5 of the files are different, though.

Now my points:

 - there are computers with slower CPU. Keep this in mind. My case was extreme, but it's just proof, that CPU is also very important during downloading the blockchain, and people having problems with slow downloads perhaps have probelem with CPU, not network.
 - precomputation is just reasonable. Why everyone should compute the same blocks every time they start a new node?
 - if you prepare "certified" blockchain (can be GPG signed, etc.), you will show the people that you CARE about them, and about their time, and freedom of choice of downloading method. This is important for people that are hesitating about running a bitcoin daemon.

Thanks for reading.

I'm currently maintaining an up to date bitcoin blockchain data torrent here: https://bitcointalksearch.org/topic/download-the-blockchain-here-updated-regularly-1310261

HDD is still bottleneck, my system shows 10-20MB of data written every block synced...

the new version 0.12 will synch even faster, thanks to the capacity increased, which will reduce  orphan, and reduce bandwidth

Newer software should be able to sync faster than downloading and verifying the torrent will on an older client.

However, if you insist on installing an old client and upgrading, here is a copy of 0.9.4 that I built and uploaded: https://github.com/achow101/bitcoin/releases/tag/v0.9.4

NEW version is syncing FASTER than ANY older one!

Once the old software is installed, and the blockchain downloaded. I can then update it to the new wallet!
It will take much less time and there is no need to re syncronise the blockchain after this is done with the newer version. It will just runn the veryify blocks process.

I think t's best that you download the entire blockchain from scratch no matter what, it's worth investing into the extra time for the extra security. Why do you want to install an old version anyway tho?

Is there a way that I can install an old verstion of bitcoin core.
Import the torrent file into that and then update the software to the latest version.

Would this work better to syncronise to the network, as I have been told that, when switching between versions, the initial blockchain import is not required? Have I got this correct?

I've just set up a fresh full node using 0.11.2 Core client. Syncing the whole blockchain took 4 days. What is interesting, downloading blocks was not a bottleneck. My 20 Mbit/s connection was never fully utilized. What is slow, is population of the database and probably building base of UTXO. I admit that my setup is rather slow: Core2Duo @ 2.4 GHz and external 1 TB HDD (5400 rpm) on USB 2.0. With this setup HDD seems to be the major bottleneck.

I guess it depends on your location and connection speeds. I still think the torrent is useful for some, not all. It's also nice to have a single file that i can use whenever i need to quickly bootstrap my client. Also when downloading the blockchain with the client i get corrupt database error's all the time, forcing me to constantly reindex the blockchain... what gives?

Bottlenecks where? Bitcoin nodes download at full ISP speed as well. It doesn't have the additional bottleneck of having to verify all of the blocks after the download. The node will verify the blocks as they are downloaded, not after they have all been downloaded.

quote author=knightdk link=topic=145386.msg13357972#msg13357972 date=1451088033]
quote author=hamiltino link=topic=145386.msg13357882#msg13357882 date=1451086860]
quote author=knightdk link=topic=145386.msg13272697#msg13272697 date=1450322869]
quote author=PatrickMacH link=topic=145386.msg13272577#msg13272577 date=1450321761]
It's a standard zip compressed bootstrap.dat file, combining all blk*.dat files out of the blocks directory.
/quote]
The actual bootstrap.dat is not compressed. The one you are downloading is, but it isn't the official one that was maintained by jgarzik.

quote author=PatrickMacH link=topic=145386.msg13272577#msg13272577 date=1450321761]
Maybe we should discuss adding a zip level compression (or better suitable) algorithm to the Bitcoin-Qt client to reduce the amout of data needed to be transferred to update the block chain.
/quote]
There is talk on the mailing list about block compression and ways to speed up syncing, but I don't think that any of it will be coming in the next release. It either isn't ready or there isn't any particular need for it. Using the actual bootstrap.dat (not the one that you used) is not actually faster than simply letting it sync.
/quote]

stop talking out of ur ass, im getting speeds up to 5mb/s with the torrent, i can't get anything near this with the client. This has been a huge time saver for me.
/quote]
I'm not talking out of my ass. From my experience it has been faster to let the client (version 0.10+) sync by itself than it is to download the bootstrap.dat and to import that.
/quote]

This just makes no sense, since torrent clients download at full ISP speed, while the btc server is subject to several bottlenecks.

I'm not talking out of my ass. From my experience it has been faster to let the client (version 0.10+) sync by itself than it is to download the bootstrap.dat and to import that.

stop talking out of ur ass, im getting speeds up to 5mb/s with the torrent, i can't get anything near this with the client. This has been a huge time saver for me.

The actual bootstrap.dat is not compressed. The one you are downloading is, but it isn't the official one that was maintained by jgarzik.

There is talk on the mailing list about block compression and ways to speed up syncing, but I don't think that any of it will be coming in the next release. It either isn't ready or there isn't any particular need for it. Using the actual bootstrap.dat (not the one that you used) is not actually faster than simply letting it sync.

It's a standard zip compressed bootstrap.dat file, combining all blk*.dat files out of the blocks directory.

Maybe we should discuss adding a zip level compression (or better suitable) algorithm to the Bitcoin-Qt client to reduce the amout of data needed to be transferred to update the block chain.