Bitcoin core slow sync/download of the blockchain

apogio

hero member

Activity: 560

Merit: 1060

Quote from: canudinho on November 03, 2023, 01:00:01 PM

Hello everyone

Gonna be fast...

I'm trying to download the Blockchain (by btc core) on an external HD, using my laptop (256 ssd, 8gb RAM). My internet connection is around 300mbps, and it is doing great, but the same way it keeps downloading 0,03% per hour... What can I do for optimizing this process ?? Just gotta download this and it won't work this by this speed.

Best regards

Hi mate.

The fastest way since you have an SSD in your machine is to copy the Chainstate directory and the Indexes directory to your SSD and create a symlink to look at those directories.

Here is a video with instructions. I saw a huge difference when I followed those steps:

https://youtu.be/q3zY9BbyfEU?si=DaRyT0cXRD7QbPzc

Keep in mind to copy the directories back to the original location when you are done with IBD. Also remove the symlinks.

If you need further assistance, let me know.

satscraper

hero member

Activity: 714

Merit: 1298

@ OP, if sync is still an issue then add to bitcoin.conf file the following entries:

Quote

dbcache=

# where M equals to 1/4 RAM, for instance if you RAM = 8Gb then dbcache=2048

blocksonly=1

# this will increase the sync speed as it disables node listening and "stop requesting and relaying transactions unless they are part of the block". After full initial sync you may remove this entry.

philipma1957

legendary

Activity: 4326

Merit: 8950

'The right to privacy matters'

you can get away with a sata ssd 2tb 8gb ram and an i5 third gen laptop.

I have a 2tb crucial running linux mint on an asus third gen i5 (let me check )

could be an i7 third gen.

This was a toss away laptop cost free. ssd was 85 bucks.

I downloaded the entire oct 2023 chain in 22 hours . my net speed is 200 max via optimum.

https://bitcointalksearch.org/topic/so-i-got-a-free-laptop-and-am-looking-to-setup-a-linux-os-for-a-node-5468600

here is the thread.

it was a third gen i7 laptop 💻

DaveF

legendary

Activity: 3500

Merit: 6320

Crypto Swap Exchange

Another side note, not related to the OP since they are on a USB device.
But for internal drives, I am really starting to see BIG differences between the top of the line SSD / NVMe and the cheap generic ones.

There was always a performance difference, but the spread of performance seems to be getting larger.
If you are just running core on the machine and noting else it's not a real concern once the IBD is done.

But with a full LN node, and a few other things the generic I have is the bottleneck. The exact same spec hardware with a Samsung is not having the same slowdown.
However....the Samsung was probably 50% more expensive so there is that....

-Dave

af_newbie

legendary

Activity: 2702

Merit: 1468

Quote from: DaveF on November 05, 2023, 06:22:32 AM

1) What CPU?
2) USB 3 or 2 on the external drive.

What others have discussed is true but running an older Celeron and an old spinning external drive on USB 2 is going to be a totally different experience then an 6th gen i5 and a newer 7200RPM external on USB3

-Dave

+1

When you use NVMe 4.0+ internal drives, the CPU becomes the 'bottleneck', especially towards the more recent blocks.

I recently downloaded the whole blockchain using a notebook (T480, i5 1.8GHz, 2TB NVMe 4.0, 64GB) and it took ~25 hours
to download the whole thing.

I saw CPU usage over 90% towards the end of the download, around the 2023 data.

Memory, network, and disk usage never went over 15%.

A kick-ass desktop with something like i9-13900K and NVMe 5.0 would probably take half a day to download the whole thing.

DaveF

legendary

Activity: 3500

Merit: 6320

Crypto Swap Exchange

1) What CPU?
2) USB 3 or 2 on the external drive.

What others have discussed is true but running an older Celeron and an old spinning external drive on USB 2 is going to be a totally different experience then an 6th gen i5 and a newer 7200RPM external on USB3

-Dave

nc50lc

legendary

Activity: 2618

Merit: 6452

Self-proclaimed Genius

Quote from: LoyceV on November 05, 2023, 03:10:30 AM

Quote from: nc50lc on November 05, 2023, 02:25:56 AM

Don't worry too much about that displayed 0.03% per hour rate, it tends to underestimate a lot.

My experience is the opposite when I start from scratch: the rate is quite uptimistic for the first hours, but becomes slower once it reaches more recent years of data.

That's normal for the older nearly empty blocks where there's little to no data to verify.
If he's getting that 0.03% per hour estimate at that part, the issue could be something more than just the hardware bottleneck.

LoyceV

legendary

Activity: 3290

Merit: 16489

Thick-Skinned Gang Leader and Golden Feather 2021

Quote from: BitMaxz on November 03, 2023, 01:45:50 PM

Code:

dbcache=4096

Is that even enough? Considering the size of chainstate, I'd say 8192 MB dbcache is better, but OP doesn't have enough RAM for that.

Quote from: nc50lc on November 05, 2023, 02:25:56 AM

Don't worry too much about that displayed 0.03% per hour rate, it tends to underestimate a lot.

My experience is the opposite when I start from scratch: the rate is quite uptimistic for the first hours, but becomes slower once it reaches more recent years of data.

@OP: syncing Bitcoin Core is more than just downloading the blockchain. It includes verifying every Bitcoin transaction ever made, which is very demanding for your hardware. An external drive is far from ideal, and it's risky. If you ever accidentally unplug it, you risk corrupting your data.
My advice: decide what you need first. You've also opened a topic about Electrum, which works great on your hardware. Or prune the blockchain if you don't have a specific reason to keep all data, so it fits on your internal SSD.

nc50lc

legendary

Activity: 2618

Merit: 6452

Self-proclaimed Genius

Quote from: canudinho on November 03, 2023, 01:00:01 PM

My internet connection is around 300mbps, and it is doing great, but the same way it keeps downloading 0,03% per hour... What can I do for optimizing this process ?? Just gotta download this and it won't work this by this speed.

Don't worry too much about that displayed 0.03% per hour rate, it tends to underestimate a lot.
For reference, when I'm catching up my node to the tip with about 7 days worth of blocks to download, it'll show me that it would take 2 days to sync at a rate that's close to yours,
but in reality, it always only takes less than an hour to finish.

But aside from that, your setup is really getting a bottleneck with that external hard drive as others have mentioned.
Follow their suggestion if you want faster sync speed.

You may also want to check if you're connected to enough peers to download the blockchain from since in some cases, nodes cannot connect to others.

af_newbie

legendary

Activity: 2702

Merit: 1468

Quote from: canudinho on November 03, 2023, 01:00:01 PM

Hello everyone

Gonna be fast...

I'm trying to download the Blockchain (by btc core) on an external HD, using my laptop (256 ssd, 8gb RAM). My internet connection is around 300mbps, and it is doing great, but the same way it keeps downloading 0,03% per hour... What can I do for optimizing this process ?? Just gotta download this and it won't work this by this speed.

Best regards

Follow these steps:

- get a laptop with at least i5/i7 CPU, 2GHz+
- install PCIe NVMe 4.0+, 2TB or larger
- install 64GB DDR4/3200+ memory
- install your favourite OS (Windows 11 or whatever)
- install all the updates etc., remove/stop bloatware

after you get the OS cleaned up:

- get the latest core from https://bitcoincore.org/en/download/
- block all incoming connections (all protocols and ports)
- block all protocols for outgoing connections, except IPv4
- open TCP/UDP port 53 and 68 and TCP port 8333 outgoing connections (for bitcoin-qt only)
- block all other outgoing connections (all protocols remote ports, except 53,68,8333 above) for all other applications
- disable power management

If you use your router as a DNS, open 53&68 for that IP only.

With ~100 mbps Internet connection, T480 Thinkpad, I am getting ~25GB per hour blockchain downloads.
(Average utilization in my setup: CPU: 35%, Memory: 10%, Network: 10%, Disk: 2%)

NeuroticFish

legendary

Activity: 3668

Merit: 6382

Looking for campaign manager? Contact icopress!

I've written this before, but I don't have now the patience to look it up, however, there are plenty of tutorials for this:

If you want faster initial sync, it's not only the dbcache, CPU, internat speed that are of help. You can make use of your SSD too.
Yes, since it's a small SSD you can't store the blockchain there, but you can store the chainstate there (7.6 GB now) and if you use indexes then those too (indexes = 43.5 GB, but you probably don't want txindex).

I've used symlinks for the job, but maybe there are other options too. And yes, after initial sync has ended I've moved the data from SSD to HDD onto the correct locations.

BitMaxz

legendary

Activity: 3374

Merit: 3095

Playbet.io - Crypto Casino and Sportsbook

Let me ask if your external drive is HDD or SSD?
if all Bitcoin core data is in HDD then that might be the bottleneck.
You can try to add dbcache under the bitcoin.conf file and set it to half of your total ram memory or just 3000mb.

Code:

dbcache=4096

Or you can try splitting the data by manually adding datadir on bitcoin.conf file for the SSD and then blocksdir into your external drive.

Check the reference below for a solution
- https://github.com/bitcoin/bitcoin/issues/14595

canudinho

newbie

Activity: 14

Merit: 8

Hello everyone

Gonna be fast...

I'm trying to download the Blockchain (by btc core) on an external HD, using my laptop (256 ssd, 8gb RAM). My internet connection is around 300mbps, and it is doing great, but the same way it keeps downloading 0,03% per hour... What can I do for optimizing this process ?? Just gotta download this and it won't work this by this speed.

Best regards

Topic: Bitcoin core slow sync/download of the blockchain (Read 215 times)