Pruned mode syncing doesn't cache file writes? - page 2.

jnano

member

Activity: 301

Merit: 74

Quote from: Lauda on June 19, 2019, 08:52:32 AM

You think you are entitled to demand

I'm not demanding anything. I'm describing a problem that I (and others) encounter, the same as any other report from the userbase. What's done with that is up to whoever is capable and willing.

Even if your suggestion works for you, it doesn't work for everyone. Besides myself, I also find it an issue when I suggest to people, especially layman, how they should use Bitcoin. I do mention Core for some of its benefits, but on the other hand it's a burden to normal users. 200GB+ of storage, or pruned mode that brings your computer down if you're on an HDD.

Quote from: ?? on ??

Currently not, but in past i use pruned mode where i didn't experience such problem. Take note i use 3.5" 7200 RPM HDD on Linux OS. Meanwhile you're using 2.5" 5400 RPM which is slower mine

Perhaps something changed in how pruned mode or caching is handled since v0.15? Or maybe some difference in settings (I don't remember the details, but I think Sjors saw cases where a larger dbcache degrades performance). Anyway, the specific HDD might make a small difference but it doesn't change the big picture. The problem is documented well enough and was also encountered by core devs.

Quote from: Lauda on June 19, 2019, 08:52:32 AM

You need maybe $10-worth-of-SSD

Regardless of this issue, if you could point me in the right direction I'd love to get a reliable 1TB SSD for $10 to take the place of the 1TB 2.5" HDD.

Lauda

legendary

Activity: 2674

Merit: 2965

Terminated.

Quote from: jnano on June 18, 2019, 01:43:55 PM

Quote from: Lauda on June 19, 2019, 03:08:28 AM

Stop using HDDs, problem solved and no dev-time gets wasted.

Yeah, people should adapt to software rather than the other way around. Great way to encourage full nodes, too.

Yes, that's exactly the best way. You think you are entitled to demand Core to implement something for you, well you are not. There is more important work that needs to be doing. What makes matters worse here, you complain about the suggestion yet you run in pruned mode. You need maybe $10-worth-of-SSD to run it in the pruned mode with your settings.

jnano

member

Activity: 301

Merit: 74

Quote from: ?? on ??

In my experience, HDD alone is sufficient for IBD as long as the HDD is used exclusively for IBD.

Perhaps you're not running pruned?

Quote from: ?? on ??

In your case, IBD most likely is very slow because it's also used for Windows OS

It's purely due to Core's caching behavior, there's no other appreciable disk activity.

Quote from: Lauda on June 19, 2019, 03:08:28 AM

Stop using HDDs, problem solved and no dev-time gets wasted.

Yeah, people should adapt to software rather than the other way around. Great way to encourage full nodes, too.

Anyway, it's mainly a question of whether there's developer interest, and luckily that seems to exist.

In particular, a while back Sjors has been spending time benchmarking and trying things out. There was also a PR from esotericnonsense. What got merged into 0.17 was from luke-jr. MarcoFalke or jamesob wanted to regularly benchmark pruned behavior, and seem to care about HDD performance as evident on BitcoinPerf, which graphs IBD time on HDD versus SSD, though currently just non-pruned.

There are two current open tickets:
#11315 Prune undermines the dbcache
#12058 Slow catchup for recent blocks on non-SSD drive

moneymakescents777

newbie

Activity: 42

Merit: 0

Prunes always make me rush off to the toilet, and I always like Canyacoin CAN

jnano

member

Activity: 301

Merit: 74

I had some hope that v0.17's PR #11658 would help, but there's no appreciable difference.

Without using a RAM drive, and unless a proper caching system is implemented for the chainstate, it seems that decent performance requires an SSD. Maybe HDD is sufficient when using non-pruned, but I haven't tried that.

jnano

member

Activity: 301

Merit: 74

Temporary solution:

I tried what LoyceV suggested.

I copied the whole "chainstate" directory to a RAM drive, which I symlinked to the "chainstate" directory under Core's data dir.
That way, sync speed turned an order of magnitude faster. When syncing was done I copied the RAM drive contents back into the physical "chainstate" directory, and restarted Core.

jnano

member

Activity: 301

Merit: 74

Quote from: achow101 on October 21, 2017, 12:46:00 PM

I had turned off my pruned test on the RAM drive, testing it now (blocks from April 10, 2017): My 200 block test took just over 5 minutes, that's even slower than my HDD-test this morning.

So, pruned on RAM drive is slower than the HDD test? Is the HDD test pruned as well?

But two differences between us: you're on v14.2.0 and Linux, I'm on 15.0.1 (x64) and Windows.

Quote from: achow101 on October 21, 2017, 11:58:18 AM

Then you have probably set the parameter incorrectly. How did you set dbcache? Can you post the contents of your bitcoin.conf file?

The dbcache setting is acknowledged in the settings window:

This is bitcoin.conf with comment lines removed:

Code:

prune=2000
minimizetotray=1
dbcache=6000

Quote

Can you post the contents of your debug.log file?

Lots of lines. Anything specific? The cache parts read:

Code:

2017-10-20 14:21:35 * Using 2.0MiB for block index database
2017-10-20 14:21:35 * Using 8.0MiB for chain state database
2017-10-20 14:21:35 * Using 5990.0MiB for in-memory UTXO set (plus up to 286.1MiB of unused mempool space)

Do the small figures refer to something other than caching? I thought the UTXO set is the same as chainstate db.

Quote

What is your CPU? CPU can also be a major bottleneck given the amount of computation that needs to be done to validate blocks.

CPU usage is single-digit on average, with peaks crossing 10% (the screenshot in the earlier post shows a typical pattern. Here's another startup.)

Quote

what kind of HDD do you have?

Internal 2.5" 5400rpm. Not quick, but all things considered, definitely not much different than any other HDDs when it comes to seeking.

The I/O patterns and lack of caching are the root cause. It's constantly reading at a few MB/sec, mixed with writes at a few 100s KB/s, with an occasional >1MB write.

The following ticket mentions changes in caching in v15. Maybe the issue is related?
https://github.com/bitcoin/bitcoin/issues/10647

Another ticket says prune mode flushes UTXO cache constantly in a v15 beta:
https://github.com/bitcoin/bitcoin/issues/11315

fabioganga

full member

Activity: 478

Merit: 113

Quote from: achow101 on October 21, 2017, 12:46:00 PM

When you have pruning enabled with the default settings (prune=550), Bitcoin Core will try to keep at most 550 MB of block and undo data on disk. This means that ~225 MB of blocks are actually stored. After that, it begins deleting blocks starting with the oldest.

When you are syncing, Bitcoin Core is downloading blocks as fast as possible, doing a basic check on them, and then writing them to disk. But validating the blocks and connecting them takes longer than the download and quick check, so the 225 MB of blocks allocated fills up quickly, but blocks can't then be deleted because the validation has not yet completed on the older blocks, so the download pauses so that the validation can catch up. Then it resumes and repeats this process. Because of this, syncing a pruned node may take longer.

When you don't have pruning enabled, Bitcoin Core can just keep downloading and checking blocks as fast as possible without any consideration as to where the validation is at.

Thanks for the clear explanation. I have raised my prune value to prune=2048. Would this speed things up a bit or not?

achow101

staff

Activity: 3458

Merit: 6793

Just writing some code

Quote from: LoyceV on October 21, 2017, 11:58:22 AM

I've never installed a full Bitcoin Core on my SSD (it's not big enough). Syncing also uses a lot of CPU, so I'm very happy with 1 block (1 MByte) per second.

I had turned off my pruned test on the RAM drive, testing it now (blocks from April 10, 2017): My 200 block test took just over 5 minutes, that's even slower than my HDD-test this morning.

It's an interesting pattern though: sometimes it stops using much CPU, I think it then downloads many blocks ahead at 5 Mbyte/s, and after this continues processing them. CPU load goes from 20 to 350 % (of virtual 4 cores). Then, after a while, it stops downloading, while still processing blocks. It seems inefficient not downloading and syncing continuously at the same time. I'm still using Bitcoin Core version v0.14.2.0, so this might be faster in the latest version.

I can't tell you what's the cause of your slow sync, it's a huge speed difference with my test. If you ever figure it out, please post your results here.

When you have pruning enabled with the default settings (prune=550), Bitcoin Core will try to keep at most 550 MB of block and undo data on disk. This means that ~225 MB of blocks are actually stored. After that, it begins deleting blocks starting with the oldest.

When you are syncing, Bitcoin Core is downloading blocks as fast as possible, doing a basic check on them, and then writing them to disk. But validating the blocks and connecting them takes longer than the download and quick check, so the 225 MB of blocks allocated fills up quickly, but blocks can't then be deleted because the validation has not yet completed on the older blocks, so the download pauses so that the validation can catch up. Then it resumes and repeats this process. Because of this, syncing a pruned node may take longer.

When you don't have pruning enabled, Bitcoin Core can just keep downloading and checking blocks as fast as possible without any consideration as to where the validation is at.

LoyceV

legendary

Activity: 3290

Merit: 16489

Thick-Skinned Gang Leader and Golden Feather 2021

Quote from: jnano on October 21, 2017, 08:06:02 AM

Thanks for checking. 70 seconds is much more reasonable (though still could be quicker). Were similar syncs quicker on your SSD install?

I've never installed a full Bitcoin Core on my SSD (it's not big enough). Syncing also uses a lot of CPU, so I'm very happy with 1 block (1 MByte) per second.

I had turned off my pruned test on the RAM drive, testing it now (blocks from April 10, 2017): My 200 block test took just over 5 minutes, that's even slower than my HDD-test this morning.

It's an interesting pattern though: sometimes it stops using much CPU, I think it then downloads many blocks ahead at 5 Mbyte/s, and after this continues processing them. CPU load goes from 20 to 350 % (of virtual 4 cores). Then, after a while, it stops downloading, while still processing blocks. It seems inefficient not downloading and syncing continuously at the same time. I'm still using Bitcoin Core version v0.14.2.0, so this might be faster in the latest version.

I can't tell you what's the cause of your slow sync, it's a huge speed difference with my test. If you ever figure it out, please post your results here.

achow101

staff

Activity: 3458

Merit: 6793

Just writing some code

Quote from: jnano on October 20, 2017, 03:32:31 PM

RAM is not a problem. Bitcoin Core anyway never used much more than the 300MB you see in the screenshot.

Then you have probably set the parameter incorrectly. How did you set dbcache? Can you post the contents of your bitcoin.conf file? Can you post the contents of your debug.log file?

Quote from: LoyceV on October 21, 2017, 12:42:04 AM

I'm not sure how it works internally, click "Help > Debug window" to see it's Memory usage for the Memory Pool, I always assumed that's the amount of cache it uses (but I'm not entirely sure), and memory pool doesn't have so many unconfirmed transactions.

The mempool and the dbcache are unrelated to each other. They have their own memory allocations and are configured with different options.

OP, what kind of HDD do you have? What is your CPU? CPU can also be a major bottleneck given the amount of computation that needs to be done to validate blocks.

jnano

member

Activity: 301

Merit: 74

Thanks for checking. 70 seconds is much more reasonable (though still could be quicker). Were similar syncs quicker on your SSD install?

The memory pool is unconfirmed transactions not yet in blocks, not the total memory usage of the client.

Windows might use free memory for its own caching, but it's relinquished if software needs it.

LoyceV

legendary

Activity: 3290

Merit: 16489

Thick-Skinned Gang Leader and Golden Feather 2021

Quote from: jnano on October 20, 2017, 04:14:43 PM

RAM's 8GB. BitcoinCore isn't restricting itself to 300MB because it tries to be polite, most memory is free.

The OS uses the free memory for file caching.

Quote from: LoyceV on October 20, 2017, 04:03:25 PM

I have the same setup on HDD (although I didn't prune it), and barely turn it off. I've turned it off now, tomorrow morning I'll turn it on again and tell you how long it takes to sync.

It took 70 seconds to sync 11 hours of blocks. That's about 1 block per second

jnano

member

Activity: 301

Merit: 74

RAM's 8GB. BitcoinCore isn't restricting itself to 300MB because it tries to be polite, most memory is free.
I don't think it limits the cache size dynamically, does it? The explicit dbcache setting suggests it just uses as much as it wants, up to that limit.

I think a RAM drive will indeed solve it, but shouldn't the software just know how to use RAM directly?

LoyceV

legendary

Activity: 3290

Merit: 16489

Thick-Skinned Gang Leader and Golden Feather 2021

Quote from: jnano on October 20, 2017, 03:32:31 PM

The only bottleneck here is the HDD, combined with the lack of caching, which leads to a lot of random small reads and writes.

I have the same setup on HDD (although I didn't prune it), and barely turn it off. I've turned it off now, tomorrow morning I'll turn it on again and tell you how long it takes to sync.

On my RAM drive experiment, I'm still doing several blocks per second. It's now less than 2 years behind, when blocks were not yet full.

Quote

RAM is not a problem. Bitcoin Core anyway never used much more than the 300MB you see in the screenshot.

Your OS uses RAM too, and the more you have, the more file cache it can use. Hence my question: how much RAM do you have? I noticed a huge overall improvement when I went from 4 to 12 on Linux.

Quote

Page faults are normal. I guess it's the disk accesses due to what the software does. The 3 minutes are CPU time, not total runtime.
A random reference point: launching Notepad entails about 2000 page faults in 50ms.

I see. It was worth a try

jnano

member

Activity: 301

Merit: 74

That particular sync was to catch up after 12 hours offline.

The only bottleneck here is the HDD, combined with the lack of caching, which leads to a lot of random small reads and writes.
RAM is not a problem. Bitcoin Core anyway never used much more than the 300MB you see in the screenshot.

Page faults are normal. I guess it's the disk accesses due to what the software does. The 3 minutes are CPU time, not total runtime.
A random reference point: launching Notepad entails about 2000 page faults in 50ms.

LoyceV

legendary

Activity: 3290

Merit: 16489

Thick-Skinned Gang Leader and Golden Feather 2021

Quote from: jnano on October 20, 2017, 10:27:41 AM

Another check, this time with dbcache=6000. The cache isn't working. I'm using 15.0.1 x64 on Windows 8.

After 15 initial minutes, sync rate is less than 1 block/minute. Memory usage peaked at about 300 MB.

What hardware specs do you have? It seems the site was under DDOS earlier, by the time I could edit my post, Bitcoin Core had downloaded 15% already. And that's just an old i3, but with 12 GB ram.

How far behind are you? 1 block/minute is only 10 times real time, that means it could take months to catch up.

Quote

After a few more minutes:

I see 159,271 Page Faults in just 3 minutes. That's 843 per second. I'm no expert on Windows, and I'm not sure what this means exactly, but this seems high to me. From Wiki:

Quote

Major page faults on conventional computers (which use hard disk drives for storage) can have a significant impact on performance. An average hard disk drive has an average rotational latency of 3 ms, a seek time of 5 ms, and a transfer time of 0.05 ms/page. Therefore, the total time for paging is near 8 ms (= 8,000 μs). If the memory access time is 0.2 μs, then the page fault would make the operation about 40,000 times slower.

Your sync speed looks a lot like my old Atom netbook with 1 GB ram (and SSD). Are you limited on RAM?

jnano

member

Activity: 301

Merit: 74

Another check, this time with dbcache=6000. The cache isn't working. I'm using 15.0.1 x64 on Windows 8.

After 15 initial minutes, sync rate is less than 1 block/minute. Memory usage peaked at about 300 MB.
I/O is more reads than writes, and there's a constant opening and closing of chainstate files.

Watching file accesses for about a minute, there were a few 100s of I/Os to block data (one file and its rev file),
and more than 100K I/Os to chainstate files.

Startup:

After a few more minutes:

LoyceV

legendary

Activity: 3290

Merit: 16489

Thick-Skinned Gang Leader and Golden Feather 2021

Quote from: jnano on October 19, 2017, 06:14:31 PM

Is there a way to get write caching?

In general, your Operating System should take care of this on a disk level.
I have good experiences running a pruned Bitcoin Core from a RAM drive. If you have enough RAM, you can do this, and copy it to HDD once it's done syncing.

On Linux, this works for me:

Code:

mkdir /dev/shm/prunedBitcoin # create new directory in /dev/shm, which by default uses up to 50% of available RAM
chmod 700 /dev/shm/prunedBitcoin # basic security on a multi-user-system
bitcoin -datadir=/dev/shm/prunedBitcoin -prune=550 # now we wait
mv /dev/shm/prunedBitcoin ~ # move to your home-directory after you close Bitcoin Core

It's now downloading at the maximum speed my Wifi allows, Estimated time left: 9 hours (although I know from experience it will take a bit longer, as it's limited by my CPU later on).

achow101

staff

Activity: 3458

Merit: 6793

Just writing some code

Quote from: jnano on October 19, 2017, 08:44:26 PM

dbcache was set to 3000. I don't think I saw the process use anywhere close to that much memory, but I'll look more closely next time.

Is dbcache only for the UTXO set? Is it for reads or writes?

The dbcache is used for holding the UTXO set in memory. Once the dbcache is full, the UTXO set will be flushed to the disk. Having a higher dbcache means that it is flushed less so it will reduce disk IO from the UTXO set side of things. However during the syncing process, blocks are being written and deleted from disk so that is a major source of disk IO and slowdown.

Topic: Pruned mode syncing doesn't cache file writes? - page 2. (Read 1455 times)