Topic: Excessive disk usage during block chain download (Read 5784 times)
Also note... Defragment after downloading the block chain. Even on an SSD... it fragments into billions of tiny fragments, stuffing itself into every nook and cranny, in UNSEQUENTIAL order... thus, this, like steam-games and torrents, gets an instant boost for loading, by defragmenting after downloading.
I think I may have stumbled across a solution to this problem, disable barrier writes on ext4 for the first blockchain sync
It involves remounting the filesystem like so sudo mount -o remount,nobarrier /
For those using btrfs filesystem, disabling CoW (Copy on Write) for the .bitcoin folder might also help alleviate this problem.
chattr +C
It involves remounting the filesystem like so sudo mount -o remount,nobarrier /
For those using btrfs filesystem, disabling CoW (Copy on Write) for the .bitcoin folder might also help alleviate this problem.
chattr +C
I'm running a bitcoin client in a virtual machine. I put the VM image on a mechanical disc first, but the performance was too low. It took forever to download the block chain.
What kind of VM?
I'm running a bitcoin client in a virtual machine. I put the VM image on a mechanical disc first, but the performance was too low. It took forever to download the block chain. Moved the image to an SSD and performace increased with an order of magnitude.
I have the same issue when trying to sync the blockchain after a fresh installation. The disk usage is so intense it makes the system unusable and literally takes days, sometimes a full week to finish. And this is on a system with an SSD!!
This is a serious issue that is going to drive away new non-technical users. I will try the ramdisk work around in the mean time.
This is a serious issue that is going to drive away new non-technical users. I will try the ramdisk work around in the mean time.
whenever i need to redownload the blockchain or need to catch up a lot, I'll just use the connect option to connect to one (fast) node... seems a lot quicker like that
you can create a 4GB RAM disk
That's a kind of what I had to do. 
The Bitcoin client does not simply download blocks and store them. It adds information about blocks, transactions, and addresses to a database with an index, and it is the building of this index, along with verification of the cryptographic hashes and signatures of all transactions that have taken place, which heavily utilizes storage and CPU resources.
If you wish to re-download the blockchain for the sake of re-verification, you can create a 4GB RAM disk (if you have 6+ GB of RAM), and utilize this as your -datadir. This will remove the bottleneck of limited I/O operations per second from a spinning hard drive with moving heads. Then you will be limited by the CPU speed of the system running Bitcoin. Of course move the downloaded blockchain to a more permanent location before powering off your computer...
Later blocks have many more transactions, thus take longer to download, the top graph shows the current block number after the elapsed time downloading, the bottom graph shows the blocks per second at a particular block height (notice how abysmal it gets at the very end):
If you wish to re-download the blockchain for the sake of re-verification, you can create a 4GB RAM disk (if you have 6+ GB of RAM), and utilize this as your -datadir. This will remove the bottleneck of limited I/O operations per second from a spinning hard drive with moving heads. Then you will be limited by the CPU speed of the system running Bitcoin. Of course move the downloaded blockchain to a more permanent location before powering off your computer...
Later blocks have many more transactions, thus take longer to download, the top graph shows the current block number after the elapsed time downloading, the bottom graph shows the blocks per second at a particular block height (notice how abysmal it gets at the very end):
Didn't the first computer verify it already?
In this particular case, it did not. In any case, I sometimes may wish to verify data. 
why not just copy the blockchain from the other computer on your network that already has it?
For sake of verification.Meh. Didn't the first computer verify it already?
why not just copy the blockchain from the other computer on your network that already has it?
For sake of verification.
If you're doing this over a local network why not just copy the blockchain from the other computer on your network that already has it?
Everyone agrees. It's a long-term goal to keep improving performance.
If your goal is to just get started then you can just download a recent copy of the blockchain
This is a way to go. However, I opine that bootstrapping Bitcoin instances from instances that already exist in the same LAN should be a reasonable technique. I'd like the Bitcoin client to be able to perform initial block chain download smoothly.
If your goal is to just get started then you can just download a recent copy of the blockchain from here:
http://eu1.bitcoincharts.com/blockchain/
http://eu1.bitcoincharts.com/blockchain/
Thanks, Revalin. I temporarily moved files blk0002.dat, blkindex.dat, and the folder "database" to RAM storage. This resulted in significant reduction in disk usage.
My Bitcoin client is currently downloading block chain over 100 megabit local network at about 35 blocks per minute. Hard disk indicator is flashing all the time. Responsiveness of many programs is very bad. "top" reports that over 40% of CPU time is being spent on I/O waiting. At such low speed, it will take hours to download the full chain, and finish torturing my HDD. This is annoying.
Download of at least first 100000 blocks went quite fast. It seems like the problems began after a certain block. Both bitcoind and bitcoin-qt exhibit the problem. I ran bitcoind with strace for some time, and it didn't show that any disk-related syscalls take too long time. My guess is that excessive disk activity may be caused by large number of munmap calls.
Relevant system information:
HDD: Seagate ST3500413AS SATA
MB: A8N-E
CPU: AMD Athlon 64 X2 4200+
OS: Slackware Linux 13.37 2.6.37.6 x86_64
FS: ext4
BDB: 5.3.21
Bitcoin: 0.6.3
Download of at least first 100000 blocks went quite fast. It seems like the problems began after a certain block. Both bitcoind and bitcoin-qt exhibit the problem. I ran bitcoind with strace for some time, and it didn't show that any disk-related syscalls take too long time. My guess is that excessive disk activity may be caused by large number of munmap calls.
Relevant system information:
HDD: Seagate ST3500413AS SATA
MB: A8N-E
CPU: AMD Athlon 64 X2 4200+
OS: Slackware Linux 13.37 2.6.37.6 x86_64
FS: ext4
BDB: 5.3.21
Bitcoin: 0.6.3
Jump to: