Pruned mode syncing doesn't cache file writes?

jamikuli

brand new

Activity: 0

Merit: 0

ABCbits

legendary

Activity: 2870

Merit: 7490

Crypto Swap Exchange

Quote from: jnano on December 15, 2020, 07:46:27 AM

By the way, another problem with these uncached writes is that it wastes SSD write cycles.

I'm seeing about 2.4 GB chainstate data written per each 1 day of blockchain (blocks from recent months). That's chainstate data only, not blocks.

Can anyone confirm?

2.4GB write/day is relative low IMO, since most SSD have hundred TB write endurance. Even assuming you're using your SSD for 10 years, you only write about 8.55TB

I don't know how to monitor specific directory, but i could this command to monitor Bitcoin Core process for short time.

Code:

sudo iotop -a

jnano

member

Activity: 301

Merit: 74

Quote from: ABCbits on December 16, 2020, 07:19:14 AM

I forget entry-level and small capacity SSD have lower write endurance.

Some QLC drives are even worse. Corsair MP400 has 200 TB TBW for the 1TB model. I'm not aware of small QLC drives currently, but that might change.

ABCbits

legendary

Activity: 2870

Merit: 7490

Crypto Swap Exchange

Here's the result with command sudo iotop -a with txindex enabled for about 80 minutes.

Code:

    TID  PRIO  USER     DISK READ DISK WRITE   SWAPIN      IO    COMMAND                 
  14736 be/4 user      0.00 B     20.50 M  0.00 %  0.00 % bitcoin-qt [b-scheduler]
  15497 be/4 user   1904.77 M      8.77 M  0.00 %  4.67 % bitcoin-qt [b-msghand]

Unfortunately iotop separate based PID and remove total read/write if the PID is closed, so i doubt my result is useful at all.

Quote from: jnano on December 16, 2020, 06:56:46 AM

Quote from: ABCbits on December 16, 2020, 05:55:42 AM

SSD have hundred TB write endurance

Some 120GB SSDs, like the Crucial BX500 or Kingston A400, have only 40 TBW. Twice that much in the 240GB models. A nice mid-ranger like the MX500 500GB has 150 TBW.

I forget entry-level and small capacity SSD have lower write endurance.

jnano

member

Activity: 301

Merit: 74

I symlinked only the chainstate directory to the SSD, then watched the drive's SMART stats.
(Maybe there's also a configuration switch to specify the chainstate directory separately from the whole data dir?)

That 2.4GB/day I'm seeing is "client writes" from the PC. I haven't checked yet SSD write amplification. And would be interesting to compare to a non-pruned node.

Assuming 2.4GB/day, that's approaching 1TB/year without block and index data, and ignoring WA. If you're not testing multiple full IBDs maybe it's acceptable, but it's not great, especially if you use the SSD for more than just Bitcoin Core. And it isn't inherently "needed", it's just a side effect of the write caching problem.

Quote from: ABCbits on December 16, 2020, 05:55:42 AM

most SSD have hundred TB write endurance

Some 120GB SSDs, like the Crucial BX500 or Kingston A400, have only 40 TBW. Twice that much in the 240GB models. A nice mid-ranger like the MX500 500GB has 150 TBW.

Quote from: NotATether on December 15, 2020, 09:00:41 PM

My chainstate folder is capped at 4 gigabytes

4GB is more or less its current storage size, but Core does a whole lot of small fragmented writes there while running. At least in pruned mode.

NotATether

legendary

Activity: 1568

Merit: 6660

bitcoincleanup.com / bitmixlist.org

Quote from: jnano on December 15, 2020, 07:46:27 AM

By the way, another problem with these uncached writes is that it wastes SSD write cycles.

I'm seeing about 2.4 GB chainstate data written per each 1 day of blockchain (blocks from recent months). That's chainstate data only, not blocks.

Can anyone confirm?

Interesting. My chainstate folder is capped at 4 gigabytes, but my node has been running for a couple days now. I don't have a way to monitor write amounts from only the chainstate folder, but I made a script that monitors write amounts from the whole process if you're interested in it. How did you capture the amount of chainstate data written?

jnano

member

Activity: 301

Merit: 74

By the way, another problem with these uncached writes is that it wastes SSD write cycles.

I'm seeing about 2.4 GB chainstate data written per each 1 day of blockchain (blocks from recent months). That's chainstate data only, not blocks.

Can anyone confirm?

jnano

member

Activity: 301

Merit: 74

Might help eventually:
https://github.com/bitcoin/bitcoin/pull/17487

Lauda

legendary

Activity: 2674

Merit: 2965

Terminated.

Quote from: jnano on August 08, 2019, 12:49:19 PM

or perhaps SegWit I/O patterns.

This is not the case.

jnano

member

Activity: 301

Merit: 74

I decided to benchmark it more carefully, on a 2.5" 5400 rpm laptop HDD, on Windows.

I did two full IBDs, one with the chainstate in a RAM drive, the other only on HDD. Block data was on the HDD in both cases. Rather than a straight run it was a few hours each time, then quitting Core. For the HDD run I defragged before every run. For the RAM run I may have as well, but not sure if every time. So as far as fragmentation goes I think it's better than typical real-life scenarios.

Occasionally I used the computer for other things at the same time, but light usage.

I partially watched CPU usage. It wasn't generally a bottleneck. For most part it looked like Core limits itself to about 50% CPU, and only in more recent blocks it reached 80%+ and then maybe hit some CPU limits. Network speed was not a bottleneck.

On pure-HDD, besides the slower IBD times, it also seriously harms the usability of the computer. In the RAM drive case it's rarely if ever a major issue.

The most interesting result is how the total RAM and HDD runtimes diverge:

It seems the HDD run is greatly affected by maximum block size, or perhaps SegWit I/O patterns.

Earlier blocks were mostly indifferent to HDD. It was even slightly faster part of the time. My guess is fragmentation differences (I may have been less careful in the RAM run). Either way, it's insignificant.

A graph of how much slower the HDD run was:

Average time/block:

LoyceV

legendary

Activity: 3290

Merit: 16489

Thick-Skinned Gang Leader and Golden Feather 2021

Quote from: ?? on ??

LoyceV, for reference, could you tell us about these things when you sync :
1. CPU usage? Was it near 100%?
2. Was Bitcoin Core use most of your internet bandwidth?

1. Most of the time it took all available CPU power.
2. On and off: it started at 3-5 MB/s continuously, but close to the end it took 8 MB/s for a couple of seconds, then stopped downloading to process the data.

jnano

member

Activity: 301

Merit: 74

Thanks for the testing LoyceV. When it's all done, it'd be interesting to know how long it all took.

This got me curious, so I'm going to compare a fresh IBD to RAM-drive versus HDD. It's going to a take a while, as I won't be able to dedicate a computer 24/7. Just for good measure, for the HDD tests I'll defrag before each separate launching of Core.

But again, regardless of how much you can improve it on HDD, there is an inherent inefficiency in how the caching works.
I'm not sure if anyone checked the GitHub issue I linked earlier

, so I'll quote Sjors directly, who seems like a pretty good authority on Bitcoin Coin behavior.

Quote from: Sjors

It often takes more than half an hour to catch up on less than a day of blocks.

Lauda

legendary

Activity: 2674

Merit: 2965

Terminated.

Quote from: LoyceV on June 22, 2019, 12:41:24 PM

Could it be SegWit blocks take longer to check?

Could it be that bigger blocks take longer to check?[1] I don't know. Use your brain? Luke would be very angry with you.

[1] FTFY.

LoyceV

legendary

Activity: 3290

Merit: 16489

Thick-Skinned Gang Leader and Golden Feather 2021

Bump with an update: I've turned off Bitcoin Core for many hours today (to speed up processing Trust data), then turned it on again. It has 7500 blocks and 2 hours to go, progressing just under 1% per hour.
Could it be SegWit blocks take longer to check?

If I wouldn't touch my PC, I think it'll complete a pruned download to hdd in approximately 24 hours. At well over 200 GB, I'm not disappointed!
I tested this via Wifi, on a system that's over 4 years old and was nowhere near state of the art when I bought it.

To conclude:

Quote from: jnano on October 19, 2017, 06:14:31 PM

I've launched it after one offline day, and syncing really grinds the HDD and barely progresses. It's doing something like 1 block per minute.

The bottleneck is probably somewhere else in your system, and not with Bitcoin Core.

LoyceV

legendary

Activity: 3290

Merit: 16489

Thick-Skinned Gang Leader and Golden Feather 2021

Quote from: jnano on June 19, 2019, 10:37:35 AM

~, but on the other hand it's a burden to normal users. 200GB+ of storage, or pruned mode that brings your computer down if you're on an HDD.

I can't believe this is still an issue after almost 2 years!

Quote from: jnano on June 19, 2019, 02:02:45 PM

It could be noticeable, but it would be far smaller than the 10 times difference you get with proper caching (simulated using a RAM drive).

I still think this might be an Operating System problem. Are you still only using Windows for this?

Quote from: jnano on June 19, 2019, 10:51:17 AM

Reading some discussions on Github, it seems the importance of pruned mode (and other space or time optimization methods like assumeutxo) increases as the size of the blockchain increases.

I can confirm this for my case: I pruned my blockchain to 99 GB (the maximum amount allowed) because I was running low on disk space. I don't want the hassle of getting a bigger disk until I replace my laptop entirely.

I'm going to run some tests myself Tongue

It will be a very rough estimate, as my PC is currently processing Merit data and will be processing Trust data tomorrow.

This is what I'm going to do:
1. Test syncing a pruned blockchain on HDD (starting now).
2. Test on RAM drive (starting when the previous is done):

Quote from: LoyceV on October 20, 2017, 06:23:29 AM

On Linux, this works for me:

Code:

mkdir /dev/shm/prunedBitcoin                            # create new directory in /dev/shm, which by default uses up to 50% of available RAM
chmod 700 /dev/shm/prunedBitcoin                        # basic security on a multi-user-system
bitcoin -datadir=/dev/shm/prunedBitcoin -prune=550      # now we wait
mv /dev/shm/prunedBitcoin ~                             # move to your home-directory after you close Bitcoin Core

I'm using database cache 300 MB (my usual).

Update: The pruned download to hdd is now 48% done. Most of the day it downloaded several MB per second, with around 6% progress increase per hour. Then, I started a torrent download (Knoppix DVD) on the same hdd, which completely took all my bandwidth, and dropped the progress increase to 0.06% per hour.
When the torrent was done, Bitcoin Core didn't speed up. I restarted it, and after a while was still barely using any CPU power, but eventually the download speed went up, later CPU usage went up, and it's now processing several blocks per second again (Last block time Feb 2017).
I'm not using the latest version of Bitcoin Core yet, but this seems to be faster than I remember from when I tested the same on a RAM drive. I think my Wifi got faster.

If it keeps going like this, the pruned download to hdd will be finished tomorrow morning. In that case I won't bother to compare doing the same to a RAM drive.
I can't reproduce OP's problem with very slow syncing of a pruned blockchain on hdd so far.

jnano

member

Activity: 301

Merit: 74

I bet some tweaks could improve HDD performance, but I doubt any would be enough to turn it into a non-issue.

Still hoping for a caching solution one day. In the meantime, I think the best solution is for me to script the RAM drive procedure.

jnano

member

Activity: 301

Merit: 74

It could be noticeable, but it would be far smaller than the 10 times difference you get with proper caching (simulated using a RAM drive).

I haven't done real reaserch, but my brief search shows that 2.5" 5400 RPM drives might actually be pretty similar to 3.5" 7200 RPM drives as far as random 4K I/O is concerned, depending on the specific models compared. See the following in StorageReview (a rather reputable storage-centric site): 2.5" HDD review and 3.5" HDD review.

Never mind the specific models, the reviews also include comparative data for other drives. The IOMeter 4K Random I/O tables show 2.5" drives, probably 5400 RPM, getting 50-160 IOPS. The 3.5" 7200 drives are getting 50-180 IOPS. The top 7200 drives (WD RE4 and Caviar Black) are 80-180, so faster, but only by about 15%. The only drives showing real difference are the 10K RPM Velociraptors, with 140-370 IOPS, but these drives are far from typical.

It could be that 2.5" drives gain some advantage over 3.5" drives due to the shorter distance the heads need to travel (equivalent to short-stroking), then lose some due to mobile-use optimizations. Also the RPM difference could contribute.

SSDs are of course much faster (though even there proper caching may have some benefits).

Lauda

legendary

Activity: 2674

Merit: 2965

Terminated.

Quote from: jnano on June 19, 2019, 10:51:17 AM

Reading some discussions on Github, it seems the importance of pruned mode (and other space or time optimization methods like assumeutxo) increases as the size of the blockchain increases. That, assuming there's the goal of not driving more users to SPV or centralized bitcoin web wallets. Bitcoin Core is really not user-friendly when you add it all up, which is a shame.

Nah, pruning is a done deal. The other things that are being worked on are not related to improving pruning (but they might, down the road).

Quote from: jnano on June 19, 2019, 10:51:17 AM

About that SSD, I'd need the capacity to at least match that of the HDD it would replace in the laptop.

The problem is your laptop HDD, i.e. the 5400 RPM that ETFbitcoin mentioned.

jnano

member

Activity: 301

Merit: 74

Reading some discussions on Github, it seems the importance of pruned mode (and other space or time optimization methods like assumeutxo) increases as the size of the blockchain increases. That, assuming there's the goal of not driving more users to SPV or centralized bitcoin web wallets. Bitcoin Core is really not user-friendly when you add it all up, which is a shame.

About that SSD, I'd need the capacity to at least match that of the HDD it would replace in the laptop.

Lauda

legendary

Activity: 2674

Merit: 2965

Terminated.

Quote from: jnano on June 19, 2019, 10:37:35 AM

Quote from: Lauda on June 19, 2019, 08:52:32 AM

You think you are entitled to demand

I'm not demanding anything. I'm describing a problem that I (and others) encounter, the same as any other report from the userbase. What's done with that is up to whoever is capable and willing.

Even if your suggested solution works for you, it doesn't work for everyone. Besides myself, I also find it an issue when I suggest to people, especially layman, how they should use Bitcoin. I do mention Core for some of its benefits, but on the other hand it's a burden for normal users. 200GB+ of storage, or pruned mode that brings your computer down if you're on HDD.

Alright, no demanding. You need to keep in mind that pruning is not a priority like full nodes are.

Quote from: jnano on June 19, 2019, 10:37:35 AM

Quote from: Lauda on June 19, 2019, 08:52:32 AM

You need maybe $10-worth-of-SSD

Regardless of this issue, if you could point me in the right direction I'd love to get a reliable 1TB SSD for $10 to take the place of the 1TB 2.5" HDD.

You are using pruned mode, you said that yourself. Therefore: You don't need 1 TB, you need <10-20 GB. FYI you can get a 1 TB SSD for ~$130, and I'm talking about the best-reliable SSDs (e.g. Samsung Evo). It's actually a great time to pick up one!

Topic: Pruned mode syncing doesn't cache file writes? (Read 1455 times)