Bitcoin Core IBD slow | Bitcointalksearch.org

apogio

hero member

Activity: 560

Merit: 1060

Quote from: LoyceV on December 05, 2023, 06:49:55 AM

Nope. A hdparm speed test doesn't care about the filesystem.

So the current speeds have gone up to:

Code:

/dev/sda:
Timing cached reads: 1666 MB in 2.00 seconds = 834.15 MB/sec
Timing buffered disk reads: 328 MB in 3.15 seconds = 104.15 MB/sec

I feel stupid, but here is what happened... I was using the USB 2.0 port and not the 3.0 port.

LoyceV

legendary

Activity: 3290

Merit: 16489

Thick-Skinned Gang Leader and Golden Feather 2021

Quote from: apogio on December 05, 2023, 06:40:38 AM

Perhaps a format will solve the issue...

Nope. A hdparm speed test doesn't care about the filesystem.

apogio

hero member

Activity: 560

Merit: 1060

Quote from: LoyceV on December 05, 2023, 06:21:21 AM

I can't find an English review for this exact model, but my Intenso USB stick is faster than what you get.

Perhaps a format will solve the issue...

LoyceV

legendary

Activity: 3290

Merit: 16489

Thick-Skinned Gang Leader and Golden Feather 2021

Quote from: apogio on December 05, 2023, 06:09:21 AM

This is my disk: https://www.amazon.de/-/en/Intenso-Internal-Inch-SATA-Black/dp/B0B63HPPG3

I can't find an English review for this exact model, but my Intenso USB stick is faster than what you get.

Quote

Could the format be an issue? My disk is exfat

The filesystem matters, but not for raw speed tests.

apogio

hero member

Activity: 560

Merit: 1060

Quote from: LoyceV on December 05, 2023, 05:48:28 AM

Which SSD do you have exactly?

This is my disk: https://www.amazon.de/-/en/Intenso-Internal-Inch-SATA-Black/dp/B0B63HPPG3

Quote from: LoyceV on December 05, 2023, 05:48:28 AM

I'm just guessing here: is it possible you're using a very low quality USB cable?
Have you tried Gnome Disks? It has a Benchmark feature.
Or have a look at GSmartControl, see if it can give you a reason for the low speed.

No errors unfortunately... with GSmartControl

Could the format be an issue? My disk is exfat

EDIT: I think I will backup my drive and then format to ext4

LoyceV

legendary

Activity: 3290

Merit: 16489

Thick-Skinned Gang Leader and Golden Feather 2021

Quote from: apogio on December 05, 2023, 05:36:47 AM

No my IBD is finished and my node is stopped. So essentially my RPi is connected to an SSD which has the whole bitcoin-core structure, but bitcoind is not running.

Which SSD do you have exactly?
I'm just guessing here: is it possible you're using a very low quality USB cable?
Have you tried Gnome Disks? It has a Benchmark feature.
Or have a look at GSmartControl, see if it can give you a reason for the low speed.

Quote

I also ran fstrim -v on the mounted directory and it trimmed some data, but it didn't work afterwards. I mean no amelioration was seen.

That shouldn't matter for "hdparm -tT", it doesn't use the file system.

apogio

hero member

Activity: 560

Merit: 1060

Quote from: LoyceV on December 05, 2023, 05:30:22 AM

Most microSD cards are faster than this nowadays. It should be 10 times faster:

Did you run your disk speed test during IBD? If so, that's probably why it's slow.

The type of SSD matters too: some perform just as you'd expect, and others are only fast "in short burst", and after writing the burst data, it gets much slower, up to the point that combined read and writes can make the entire system freeze for a while. That doesn't matter much for normal computer usage, but it matters a lot during heavy tasks (such as the IBD).

Hi Loyce.

No my IBD is finished and my node is stopped. So essentially my RPi is connected to an SSD which has the whole bitcoin-core structure, but bitcoind is not running.

I also ran fstrim -v on the mounted directory and it trimmed some data, but it didn't work afterwards. I mean no amelioration was seen.

LoyceV

legendary

Activity: 3290

Merit: 16489

Thick-Skinned Gang Leader and Golden Feather 2021

Quote from: apogio on December 05, 2023, 05:17:15 AM

Code:

/dev/sda:
Timing buffered disk reads: 62 MB in 4.67 seconds = 13.28 MB/sec

Isn't it too slow for an SSD ? Especially the second timer.

Most microSD cards are faster than this nowadays. It should be 10 times faster:
Image loading...

(image source)

Did you run your disk speed test during IBD? If so, that's probably why it's slow.

The type of SSD matters too: some perform just as you'd expect, and others are only fast "in short burst", and after writing the burst data, it gets much slower, up to the point that combined read and writes can make the entire system freeze for a while. That doesn't matter much for normal computer usage, but it matters a lot during heavy tasks (such as the IBD).

apogio

hero member

Activity: 560

Merit: 1060

I wanted to test it further, so I did:

Code:

sudo hdparm -tT /dev/sda

To my surprise, I got:

Code:

/dev/sda:
Timing cached reads: 1776 MB in 2.00 seconds = 888.80 MB/sec
Timing buffered disk reads: 62 MB in 4.67 seconds = 13.28 MB/sec

Isn't it too slow for an SSD ? Especially the second timer.

apogio

hero member

Activity: 560

Merit: 1060

To sum up:
The IBD went from 0 to 790,000 in 5 days, then from 790,000 to 810,000 in 3 days, then from 810,000 to 814,000 in 2 days and currently, from 814,000 to 816,000 in 10 hrs. I guess it will reach 819,000 soon enough.

Thank you all for the answers, apparently it was what Loyce said above regarding the ordinals and the larger transactions.

Now, I am at 816,000 and it moves quickly. It must be finished later during the day.

Finally, I did increase my dbcache to 6000 (from a total of 8GB RAM), as all of you suggested. However, I did not use the assumevalid directive, because I wanted to read more about it before using it, but unfortunately I didn't have the time.

ABCbits

legendary

Activity: 2870

Merit: 7490

Crypto Swap Exchange

Quote from: apogio on November 30, 2023, 07:04:37 AM

Quote from: ?? on ??

Edit: be careful with default linux swap behavior which perform some swap even though there are some free/unused RAM capacity.

I am sorry I don't follow you on this. You mean perhaps I should add less than 7000 and go for 6000 for example?

I was referring to swappiness value (each distro probably have different default value) which affect swap behavior. That means i suggest you to check that value and optionally reduce it. Here's an example guide, https://linuxize.com/post/how-to-change-the-swappiness-value-in-linux/.

DaveF

legendary

Activity: 3500

Merit: 6320

Crypto Swap Exchange

Quote from: apogio on November 30, 2023, 12:24:41 PM

Quote from: DaveF on November 30, 2023, 11:03:05 AM

Have you done any checks on the SSD? I spent a couple of hours the other day trying to figure out why a PC was so slow and it turned out the almost new SSD was failing SMART checks.
Was looking for malware, was checking running apps, etc. Till I finally ran crystaldisk and poof. But neither the PC or Windows put up a smart error.

-Dave

Hi Dave, no I haven't checked the disk. I think it's an exFAT, but I am not sure. Also I run Raspbian and I am not aware of what crystal disk is. Is it a tool?

Crystaldiskinfo is a SMART https://en.wikipedia.org/wiki/Self-Monitoring,_Analysis_and_Reporting_Technology checker for drives for windows.
I don't know of an equivalent for RPi, will have to look.

But it does show what the drive thinks is going on it with it, not what Windows or what the PC itself thinks. They can for whatever reason hide / not report SMART issues.

-Dave

apogio

hero member

Activity: 560

Merit: 1060

Quote from: DaveF on November 30, 2023, 11:03:05 AM

Have you done any checks on the SSD? I spent a couple of hours the other day trying to figure out why a PC was so slow and it turned out the almost new SSD was failing SMART checks.
Was looking for malware, was checking running apps, etc. Till I finally ran crystaldisk and poof. But neither the PC or Windows put up a smart error.

-Dave

Hi Dave, no I haven't checked the disk. I think it's an exFAT, but I am not sure. Also I run Raspbian and I am not aware of what crystal disk is. Is it a tool?

DaveF

legendary

Activity: 3500

Merit: 6320

Crypto Swap Exchange

Have you done any checks on the SSD? I spent a couple of hours the other day trying to figure out why a PC was so slow and it turned out the almost new SSD was failing SMART checks.
Was looking for malware, was checking running apps, etc. Till I finally ran crystaldisk and poof. But neither the PC or Windows put up a smart error.

-Dave

apogio

hero member

Activity: 560

Merit: 1060

Quote from: NotATether on November 30, 2023, 08:21:00 AM

Quote from: apogio on November 30, 2023, 07:04:37 AM

I am sorry I don't follow you on this. You mean perhaps I should add less than 7000 and go for 6000 for example?

Try 3/4ths of the total RAM. Your disk doesn't seem to be the problem I thought you had an internal SSD and externally-attached 2TB SSD... my bad.

Also combine it with the assumevalid directive that nc50lc posted about, maybe set it to 800000 blocks, to really speed up things, as these things must have been verified thousands of times by now.

I will set it to 800,000 and also set dbache to 6000 and I will see if it makes a difference.

NotATether

legendary

Activity: 1568

Merit: 6660

bitcoincleanup.com / bitmixlist.org

Quote from: apogio on November 30, 2023, 07:04:37 AM

I am sorry I don't follow you on this. You mean perhaps I should add less than 7000 and go for 6000 for example?

Try 3/4ths of the total RAM. Your disk doesn't seem to be the problem I thought you had an internal SSD and externally-attached 2TB SSD... my bad.

Also combine it with the assumevalid directive that nc50lc posted about, maybe set it to 800000 blocks, to really speed up things, as these things must have been verified thousands of times by now.

apogio

hero member

Activity: 560

Merit: 1060

Quote from: LoyceV on November 30, 2023, 06:58:02 AM

I'm curious if that helps. More dbcache also means less file cache for the OS.

Exactly, but I haven't found any explanation online, so I thought perhaps one would know in this forum. Feels like above 4GB doesn't help too much. It's like there is a soft-cap and then the return you get is diminishing. Perhaps I am wrong though.

Quote from: ?? on ??

Edit: be careful with default linux swap behavior which perform some swap even though there are some free/unused RAM capacity.

I am sorry I don't follow you on this. You mean perhaps I should add less than 7000 and go for 6000 for example?

Quote from: nc50lc on November 30, 2023, 07:03:34 AM

It's not necessary bad, security-wise, because it's just the script verifications that are skipped.
Everything in the assumed valid blocks are still being verified by your node.

The topic is quite famous after its implementation so you may find a lot of helpful related topics across the internet.
For example: https://bitcoin.stackexchange.com/a/88666

Thanks mate, much appreciated!

nc50lc

legendary

Activity: 2534

Merit: 6080

Self-proclaimed Genius

Quote from: apogio on November 30, 2023, 06:22:27 AM

Quote from: nc50lc on November 30, 2023, 06:15:23 AM

-snip-
It's not recommended but if you think it's necessary to speed up your node's IBD, you can set a custom assumevalid block by setting -assumevalid=.

Thanks, I will see the reference because it is something I ignored until today.

Wouldn't it be bad for security though? Is this why you say it is not recommended?

It's not necessary bad, security-wise, because it's just the script verifications that are skipped.
Everything in the assumed valid blocks are still being verified by your node.

The topic is quite famous after its implementation so you will find a lot of helpful related topics across the internet.
For example: https://bitcoin.stackexchange.com/a/88666

LoyceV

legendary

Activity: 3290

Merit: 16489

Thick-Skinned Gang Leader and Golden Feather 2021

Quote from: apogio on November 30, 2023, 06:38:38 AM

Perhaps all of it? 8192?

That leaves nothing for other programs, so it's too much.

Quote

I set it to 7000 and let's see...

I'm curious if that helps. More dbcache also means less file cache for the OS.

apogio

hero member

Activity: 560

Merit: 1060

Quote from: ?? on ??

It leads to same result, although i find it's unusual to see configuration on multiple places. And since your device has 8GB RAM, you could set higher value for dbcache (assuming you don't run other memory-intensive software on that device).

Like how much? Perhaps all of it? 8192? I don't run anything else on this device. My plan, in general, is to run Bitcoin & Monero node on the same machine.

EDIT:
I set it to 7000 and let's see...

Topic: Bitcoin Core IBD slow (Read 246 times)