Topic: Again a topic about wallet sync (Read 341 times)

I think you meant HDD speed, not HDD storage. Blockchain size is only 250GB which is small compared to today's customer HDD capacity (0.5 - 8TB)


At least for Bitcoin Core client, there are many optimization on synchronization. The biggest hurdle now are using HDD to store Bitcoin's blockchain or/and slow internet connection which hardly can be optimized.

No, I was indeed talking about the HDD storage but you are right that today 250GB is not so much and 1TB is slowly a standard or a bare minimum for every more advanced computer user.

Actually, my biggest HDD is 250GB and I don't think I will fill it completely at any time soon.

Everything is changing because a few years I've always had too little memory and had to buy extra or larger ones if they were available.

Filling any memory for me wasn't a problem because I was storing a lot of video files, but nowadays with video streaming and all these movie services just don't have such a need.

I was afraid that the number of full nodes may decrease drastically taking into consideration the size of the blockchain which is technically too high for a hard disk of a PC.

I was running a full node for almost 4 years because Bitcoin-QT was the only one available wallet when I started to use and later mine BTC. Additionally, I always heard that the best thing we can do to support the network is to run a full node but in today's reality, I rather doubt if average people decide to do it through downloading the whole blockchain because of the size. There is no comparison to the times when a few hours of synchronization and a medium-class computer was enough to be a happy owner of a full node.

Despite everything, the number of full nodes increases every year and today we have already 9137 according to https://bitnodes.earn.com/. It looks like a phenomenon to me.



It would be great if some solution could be implemented to speed up the synchronization because it would be only worst during the time.

I don't think these posts were overlooked.

OP take these suggestions seriously and even changed his drive to SSD which significantly accelerated the synchronization speed as @Bitcryptex advised.

Additionally, @Jackg suggested alternatives like SPV wallets (electrum) for people who don't really need a full node and this was also discussed above.


Of course, one should do all what possible to check such file before download but this is true that security risk is the main reason that bootstrap files should not be used if possible.

Bitcoin has bootstrap file too in the past. But since about version 0.10 something has change and the verify blocks steps seems to be the bottleneck, being slower than the actual download. And yeah, bootstrap.dat is a security risk. But that doesn't stop people post and use bootstrap. Old habits don't die easy.


And whoever doesn't have the time, space, willingness to do this sync properly, there are plenty of good SPV wallets. That's what I also use.

Some nice advice has been given, but i'd like to put some emphasis on part of BitCryptex and jackg's initial posts... They are correct, but it seems their posts have been overlooked...

Running a full node isn't necessary if you don't want to... If you have problems with the time it takes to do a full sync, or the amount of data you need to store, start looking at an SPV wallet.
An SPV wallet only downloads the block headers, not the complete blocks. A "full sync" of all headers happens in a matter of minutes and only uses a couple Mb on disk, and i find SPV wallets sufficient for most day-to-day task.
Let's be honest: it's a good thing for the network if a lot of people run full nodes, it's good for an enduser to verify all blocks and all transactions, but it isn't absolutely necessary for everybody to run such a full node if they are only going to perform enduser tasks. If we have several thousand full nodes, endusers can use SPV wallets without significant impact on the network's security (imho).

Granted, if you use an SPV wallet, you do rely on other nodes and you give away a small part of your privacy... But security-wise SPV wallets seem to be fine.
If you worry so much about synctime and diskusage, download and verify the latest version of electrum, or buy a ledger or trezor (both spv hardware wallets). By using a HW wallet, you'll even add an extra layer of security....

Always when there were problems with synchronization of the wallets in many projects, the first solution provided was the bootstrap file with all blockchain data, mostly ending on the last checkpoint. But to be honest, I have never seen this solution for Bitcoin Core client. When I have used bootstrap files a couple of times, always the synchronization was smooth and took only a fraction of the time needed if I would try to sync without the file. It was like a couple of days, to a few hours difference.

Keep thinking about this and maybe bootstrap file works better (faster) in POS coins because if I remember correctly, I have used it only for alts like Pivx, Dash, Bitsend, Masternodecoin and never in POW mined coins like our old good Bitcoin?

I think they optimised it too so it's faster to do it by core of its at minimum the same. Unless you buy hardware with the blockchain on it (and verify a few merkle roots, hashes and tx) then it'll still take the same amount of time. 

That won't help at all, you still need to verify all blocks from the bootstrap file. The speed when you download from other nodes (for Bitcoin Core or P2P client) and centralized server more/less is same.

Besides, you need to trust the uploader not to give malicious/altcoin bootstrap file.

Such a summary was needed at the end of this topic, exactly when I wanted to ask about the time difference between synchronization on HDD and SDD.

I think 2 days is acceptable, taking into consideration slow internet, sync can be possibly improved to 24 hours or even less on really good performing machines.

Still, I am surprised that there is no other way to sync faster. I was sure this is possible with the help of bootstrap files but maybe not for Bitcoin Core client?

Tried the SDD (970 EVO 1Tb) and it did make a HUGE difference. I synced the complete blockchain in two days. My 20bps internet was the next limiting factor, but two days is ok for me.

Me too, in another 5-10 years it’s going to become ridiculous to keep up (storage wise).

I can keep my node synchronised over public WiFi without much difficulty. That's on a Windows 10 machine with an HDD. As others have said, it is much faster if I use a Linux machine with an SSD. I can handle the current situation, and I appreciate the extra security that comes from it. I may well switch to keeping a pruned node in the future though.

That's not a "maybe", it's a given

I keep my "blocks" directory on my old hdd, and the rest of Bitcoin Core on ssd. Especially the chainstate-directory benefits a lot from much faster storage.

That would be a waste: once synced, you'll never have to go through this again. Just keep Bitcoin Core running and it downloads 6 blocks per hour to remain fully synced.

You're not transferring data, you're downloading and validating it. The later being very resource heavy.

You seem to have room for a large dbcache, that should help. Try setting it to 2 or 3 GB.

In most cases the HDD is the most limiting factor in sync speed, yes.

No, sequential read and write speeds are relevant only to large files. While synchronizing, you are dealing with a large amount of small files which are being verified by the client one by one.

So sync performance could be improved if the wallet would cluster the read/write operations. Not sure if this would be possible, I never looked at the wallet code.

Sequential reads and writes are not important. You should be more concerned about input/output operations per second (IOPS). HDDs usually reach ~75 IOPS while SATA III SSDs can go up to 120000 IOPS or more. NVMe SSDs are even better in this aspect.

When I look at the system performance it seems the HD is the bottleneck. CPU is only at 15%, RAM 3.9G/8G, Ethernet below 5%, but the HD is 100% flat.
So maybe SDD will improve sync, I will try that.
But it raises a question for me, why is syncing so hard on the HD? Blockchain is about 250G, that's only a few minutes for a SATA6 drive

There are a lot of alternatives to running a full node and waiting this long to download. I think the favourite for desktop at the moment is electrum (from electrum.org/#download).

You can wait for core to finish syncing though if you want to it's much faster than it used to be when I started. You might want to tell us your system specs too a standard laptop should handle it in a few days, you could try upgrading your hardware if you want to run a full node like putting an ssd in your hard drive slot. You could try looking at other ways to improve performance from the confignfile if you have spare ram.