Topic: downloading blocks is too slow (Read 1934 times)

Headers first sync is now talked about for a long time, apparently sipa is working on it at the moment (or not, he didn't push commits regarding this to github in the last 2 weeks). Hopefully it'll be ready for 0.10.0, it was already rumored to be in 0.9 but apparently didn't make the cut.

You would need a really slow CPU for it to be the bottleneck.  zvs is right, your bootstrapping speed will depend heavily based on the peer you connect to.  If that client has sub dialup speed uploads bandwidth (maybe because they are "leet" and have 860 inbound connections on their 1Mbps DSL line).  

It hasn't gotten much focus because it works and it isn't a critical upgrade but as the network becomes larger and the bootstrap times longer the clients will need to become better multitaskers.  Queueing up blocks, analyzing peer connection speeds, downloading in parallel, etc.   Optimally there should be a solid queue of blocks building up because other bottlenecks (probably disk unless you have a fast SSD) can't keep up.  Other ideas to explore would be to download headers only first, find, and verify the best chain.  The node now has a complete set of validated headers from the genesis block to current and can request block bodies from multiple peers and fill in the framework.  It also would make sense to throttle the number of connections a peer has based on its connectivity and maybe have peers distinguish between up to date nodes and bootstrapping nodes (like seeder and leacher terms used in bittorrent).  For a given amount of upstream bandwidth a node can support more "synced" peers than it can "bootstrapping" peers.   One could define the difference as a node who is more than two hours behind the best chain as bootstrapping and nodes would share that with other nodes when the status changes.  

Maybe it's 'cause I have a fast CPU (i7-4700), but my problem was getting peers that uploaded too slow.  Each time it requests blocks from some peer w/ little or no upstream, you waste several minutes..

That's why it's better to just connect to one fast peer.

No it doesn't.

If you had blocks from 0 to 10000, you would send something like.

I have (hashes of)

block 10000
block 9999
block 9998
....
block 9990
block 9900
block 9000
block 8000
block 5000
block 2000
block 500
block 0

The peer then scans the list and starts with the one first on the main chain.  If your block 10000 wasn't on the main chain, but block 9990 was, then it would start at block 9990.

Normally, it starts from the most recent block you have, unless you are somehow on a fork of the main chain.


This is generally the big one.

Are there plans to do block verification (of multiple blocks) in parralel?
Especially during bootstrap.dat loading this would be very useful (because we all have multi core machines) - possibly even utilising the gpu to speed up the process some more.

Network speed is rarely the bottleneck unless you just have the bad luck of continually connecting to a really slow peer.   Downloading 20 GB @ 5Mbps is only ~8 hours.

Downloading and verification aren't independent steps.  To avoid being spoofed by other nodes your client will download a block, verify it, and then request the next block.  The blockchain is downloaded, verified, and indexed one block at a time.

Use

bitcoin-qt (or bitcoind) --maxconnections=1 --connect=5.9.24.81

until you finish syncing the entire blockchain.

This will help w/ two things, 1) you won't use any upstream (& potentially have downstream limited by saturated upstream) by having or connecting to a client that's also downloading the blockchain, 2) you'll receive blocks as fast as your CPU can process them (I have gigabit connection on 5.9.24.81).

Check your network bandwidth. Block downloads depends on network speed.

Verification and write speed, after downloading, your computer need to verify the blocks individually. SSD greatly improves the speed. Download speed is dependent on your internet speed, not on hardware. 

Trying to use SSD, it will improve the download speed obviously.

Actually a high latency, or a slow computer, can cause very slow downloads in the bitcoin blockchain, even if you have good bandwidth.

The way the initial block download works, your client goes and asks somebody, "what's the current block", and they say, BLOCK ID XXXXX.  And then since you don't have XXXXX, you ask them for that and they send it to you, but after you download it, then you look at its block header and discover that the previous block was WWWWW - and you don' t have that one either!  So you ask, "Hey could you give me  a copy of Block WWWWW?"  And they send that, and you look at it, and its parent was VVVVVV, and you don' t have that either, so you ask.....  and so on, back to the Genesis block.

The deal here is that you don't know which additional block you need next (and therefore can't ask for it) until you've gotten the current block, so you can only have one request for a block out there at a time.  If you have a high latency, or the peer you asked the block for has a high latency, it can take a few seconds to get each and every block, even if you both support very high transfer rates.

Then there's a CPU bottleneck to worry about.  Your peer does all kinds of cryptographic checks on each downloaded block, so if your CPU is slow, that may even dominate the latency time. 

It's worthwhile to get the bitcoin blockchain using bittorrent; with the bittorrent protocol at least you can have requests out there and active for more than one block at a time. 

Cool~ Thank you!!!

Note: You need to use torrent client in order to get the blockchain file. Try using a torrent client. As to how to import it, navigate to ~/.bitcoin/ and paste your bootstrap.dat or any other name that u have finished torrenting. Personally I like this: http://www.utorrent.com/downloads/linux
Wish you good luck!

Thank you Cahirlet. I'm not sure where I should put the bootstrap.dat.torrent file. under /root/bitcoin I have .git; contrib; doc; qa; share; src folders. Which one should contain bootstrap? And to start this bootstrap, can i just run '$bitcoin = new jsonRPCClient('http://admin:[email protected]:8332/');' ?

Many many thanks,
J

Mine took a few days. If you want it to be faster, download the bootstrap from the official site.
Go to https://bitcoin.org/en/download this and scroll to the bottom.
The QT will verify the blocks when starting up. If you do not want to download the blocks, try a light weight client like multibit 

You'll have to install a torrent client, like transmission.

this is bullshit. bitcoin core only downloads from one peer at a time. having more peers does not improve download speed.

Sorry, really new for bitcoin. How can I do that on CentOS server?
Many many thanks~
J

The link to the bootstrap torrent can be found directly on bitcoin.org download page. https://bitcoin.org/en/download

You might have poor node connectivity. How many connections do you have? Check that you've setup port forwarding correctly in your router so other nodes can "find" yours easier. Failing that, you can change the source code's outgoing connection # and incoming connection # to be 100 instead of ~8. Improves connectivity and block downloads if you're running a server. Just be careful it doesn't rape your bandwidth.

In future you might want to use a bootstrap.dat. It's a pre-downloaded list of blocks that can vastly speed up how fast you can get a client "up to date." There are some good torrents for this.