Topic: Are High Disk Writes Expected? (Read 438 times)

This is interesting discovery. Usually i would suggest people to consider ElectrumX or Fulcrum when they want to run Electrum server, but it'll make me prioritize to recommend Fulcrum instead.


If you strictly follow the guide, i think it means ElectrumX have default poor configuration rather you misconfigured ElectrumX.

I think I'm just going to use Fulcrum. It's working well, and these results are promising. I have been running iotop for about 14 hours now (first column is cumulative disk writes, second column is process name):


Between all the Fulcrum variations in the list above, it looks like just under 1 GB was written in 14 hours. In the same timeframe, electrumx would have written around 70 gigabtyes to disk (seemed to average around 5 GB per hour). Crazy difference.

There are three possible causes I can think of for now:

1. I have electrumx misconfigured somehow. If other people were observing this, then this is probably not the true. However, nobody else running electrumx reported back to me, so I have no evidence that this is true.
2. Some of the code is poorly implemented somewhere or there is a bug in electrumx. I also have no evidence for this.
3. This is a difference between leveldb and rocksdb. I only tested electrumx with leveldb. Fulcrum uses rocksdb. Each of these simply may handle things differently. It's possible I wouldn't observe this if I used electrumx with rocksdb instead.

At this point, I'm not really willing to sink any more time into this to get to the root cause, since Fulcrum is working great. I'm just going to continue using it for now.

Thanks for your help, everyone.

Yes, it create log file. But based on previous discussion and example on https://electrumx.readthedocs.io/en/latest/HOWTO.html#understanding-the-logs, i expect bunch of these which isn't very helpful.


But if you or someone else know how to make ElectrumX perform flush less frequently, it probably can reduce it's write activity.

doesn't electrumx have a log file? why not open it and take a look at what it's doing? maybe it will tell  you when it is doing disk writes and what they are for. any reputable software would.

Both. I wasn't seeing any difference in the RAM usage or any difference in the disk writes when using 16 GB of RAM relative to 8 GB, even after increasing the dbcache in Bitcoin Core and the CACHE_MB in electrumx.

Also, Fulcrum finally fully finished indexing/processing all the blocks, so I'll let it run for a couple hours and monitor the disk writes. It will be interesting to compare to electrumx.

Did you miss this part?

No difference in memory usage, or no difference in disk writes? I guess it depends on what your Electrum server is doing: Bitcoin Core would benefit from larger dbcache, I don't know anything about Electrum server.

Agreed.

I'm not sure about this. As I mentioned, not very much RAM was being used according to Xubuntu's built-in task manager, and when I put in 16 GB instead of 8 GB of RAM, I noticed absolutely no difference. I highly doubt the processor would impact the amount of disk writes.

Once again, it's not like the performance is bad; it's actually been entirely sufficient for me. I just can't figure out why ElectrumX is writing so much to disk. This very well could simply be how it was designed, and everybody, regardless of hardware, would experience this.

1tb ssd
32gb ram
I5 8500  or newer

using the Celeron and 8gb ram is simply a shit show.


I have run a ton of different  wallets in fact I will be doing a new one as I now have 1 gb  vs my old internet

I'm giving Fulcrum a try right now. The database is currently building. It uses rocksdb while electrumx (by default) used leveldb, so maybe the write activity will be different. I will report back once it's all finished.

Using HDD seems to be poor choice. If i were you, i would try different Electrum server implementation such as Fulcrum or Electrs. Sparrow Wallet developer even write a benchmark about it on https://www.sparrowwallet.com/docs/server-performance.html.

I wouldn't move the writes to a HDD. That's terrible for performance. Note that cheap SSDs are usually lower quality, both in endurance as well as performance. I'd just let it be, keep backups of the data, and by the time your SSD is "worn" in a few years, replace it with a bigger one.

At the moment I cannot find any settings to change for electrumx that would lower the disk writes. As a temporary solution for now I'm thinking I might use an extra 2.5 inch HDD from one of my old laptops and dedicate it to just the electrumx db folder (this is where all the writes are happening) so I can keep it separate from the SSD the OS and Bitcoin Core are installed on. Obviously performance will be worse but I'm not sure it will be bad enough to really matter for me. Either that or I will use a much cheaper extra 256 GB SSD instead.

Assuming the output comes after you use 16GB RAM and set higher dbcache value, it's not surprising since Bitcoin Core would use disk less frequently. I saw you asked on GitHub whether changing CACHE_MB value may solve the problem. But description under CACHE_MB also mention ElectrumX write something called "unflushed history" frequently which doesn't seem could be changed.


I recall their node software also recommends using SSD if you store the data on HDD. Although i don't know if they still show that recommendation today.

I've never used Electrum server, but I'm curious: what could it possibly be doing that requires so much disk writes?

This has nothing to do with Monero.

and that's after he'd checked out factory specs. hopeless case

Funny you say that because the electrumx devs specifically recommended using an SSD. (As I pointed out above, it's electrumx that is doing all the disk writes, not Bitcoin Core.)

say monero clearly stated that their node will ruin ssd. i'll rather go with sas hba and sas hdd. used sas1 are cheap now as  are sas2 hdd's, it's enterprise-grade things.
by hba i mean raid card flashed to IT mode (it'll be LSI one most probably). Because if it's not flashed already you'll need to do this https://www.reddit.com/r/DataHoarder/comments/lck2ou/comment/gm93zpp/      on your own.

Thanks for suggesting iotop. I've been running it for about 10 minutes and I already get the impression electrumx is doing all the disk writes, not even Bitcoin core. After just 10 minutes I'm seeing:

Disk Writes          Command
3.36 G                 python3 /home/user/Desktop/electrumx/electrumx_server
17.97 M               [jdb2/sda2-8]
6.52 M                 systemd-journald
3.84 M                 bitcoin-qt -min -chain=main [b-scheduler]
3.75 M                 bitcoin-qt -min -chain=main [b-msghand]
3.36 M                 tor --defaults-torrc /usr/share/tor/tor-service-defaults-torrc -f /etc/tor/torrc --RunAsDaemon 0

So it's almost certainly my elecrumx configuration I'll need to scrutinize, not Bitcoin core.

Edit: Here's what I'm seeing after one hour:

Disk Writes          Command
8.49 G                 python3 /home/user/Desktop/electrumx/electrumx_server
56.66 M               [jdb2/sda2-8]
23.80 M               systemd-journald
20.05 M               bitcoin-qt -min -chain=main [b-msghand]
16.27 M               bitcoin-qt -min -chain=main [b-scheduler]
13.89 M               tor --defaults-torrc /usr/share/tor/tor-service-defaults-torrc -f /etc/tor/torrc --RunAsDaemon 0

Each new block requires verification, which includes reading and writing to chainstate.

Usually the mainboard is the limiting factor.

With 16 GB RAM, that doesn't leave much for the rest of your OS. I'd go for no more than half your RAM, so try 8 GB dbcache if you install 16 GB RAM.

That should be enough. For what it's worth: my system has 16 GB RAM (and the mainboard can't handle more), and during IDB I increase dbcache, but when normally running I reduce it to 1024 again. I don't want Bitcoin Core to occupy the majority of my RAM, I need it for other things.
I just checked: I've written 7.5 TB in the past year (I didn't do a new IBD on it). That's 21 GB written per day, for Bitcoin Core, swap and everything else running on my system. I keep chainstate on this SSD, and blocks on my HDD. At this rate, I'll reach my SSD's rated writes after 80 years.

I'm getting the impression that adding more RAM mostly reduces disk writes before the node is fully synced, not after. But I have extra RAM and I can bump it up to 16GB so I can give it a shot. The Celeron supports up to 64 GB of RAM according to Intel's website. I could also put in 32 GB later (I can swap it out from another old computer), but I'll start with 16 for now.

From what I can tell, I should set this line in bitcoin.conf (12288 = 12 * 1024):


Is that the only thing I need to change? Or are there other settings I should try too?

Adding more RAM should largely reduce your disk writes, but that may not be possible considering it's a Celeron.

That makes sense: with enough RAM, chainstate can stay in memory until you shut down Bitcoin Core. So all that's written is the full blockchain, which must have been 561 GB back then.

At a glance, i find it's reasonable since you only have 8GB of RAM when current UTXO size is about 11.6GB according to https://statoshi.info/d/000000009/unspent-transaction-output-set and you need to share with both OS and ElectrumX. But for starter, you probably want to run iotop --accumulated and wait for few days to know which process perform most write.


In addition, according to Jameson Lopp benchmark, the write is much lower if you have much bigger RAM and configure Bitcoin Core to use lots of RAM. Although his benchmark doesn't enable txindex either.

The system has 8GB of RAM on an Intel 8th gen Celeron CPU (2 cores, 2 threads). The node isn't pruned. Xubuntu's built-in task manager is showing Memory at 25% (1.9 GiB / 7.6 GiB) and Swap at 34% (1.4 GiB / 4 GiB) at this exact moment, having had both Bitcoin Core and ElectrumX running 24/7 for about 3 or 4 days. I guess I don't really understand why the Swap is being used significantly given how much memory appears to be available, but maybe this is normal. If not, let me know.

I don't have anything in the Bitcoin.conf file specifically changing the database cache. However, from what I can see in the GUI using bitcoin-qt, under Settings -> Options -> Size of database cache, I see it is set to 450 MiB.

Also, I checked smartmontools again and confirmed the sector size is 512 bytes, so my math should be correct.

Can you share how much RAM you have, how much swap is in use, and Bitcoin Core's dbcache setting?

The 500 GB version can handle 300 TBW. Even at 150 GB per day, you're looking at 5 years.

I have been running Bitcoin Core 27.0 with txindex=1 and ElectrumX 1.16.0 on a Samsung 860 Evo SSD on Xubuntu. This is a dedicated machine doing nothing but running those two programs all day. I've also been using smartmontools to monitor the total disk writes once a day over time, and I've noticed the numbers are shockingly high. Obviously this is a concern since SSDs can only perform so many writes before they die.

Unfortunately, I only have a few data points right now, but all of these were recorded AFTER both the initial block download and after the transaction index was finished being built:

30 total gigabytes written to the SSD from June 24th to June 25th
108 total gigabytes written to the SSD from June 25th to June 26th
158 total gigabytes written to the SSD from June 26th to June 27th

To be clear, this is the difference in total GiB (512 bytes * Total LBAs Written /1024**3) written to the disk from day to day, not actual capacity being used. The amount of disk space being used is about what I would expect (increasing a few hundred megabytes per day).

I'm getting the LBAs written from smartmontools using:
sudo smartctl -a /dev/sda
and pulling the Total_LBAs_Written field, in case anybody else wants to try to monitor this for themselves and compare.

Obviously there's a lot more testing I need to do before I can determine what's causing this for sure but of course my intuition is that it's either related to Bitcoin Core or ElectrumX since the machine is dedicated to running them.

It is also possible I have the LBA size wrong for this drive and I'm inflating the gigabytes written without realizing it, but 512 bytes is standard from what I can tell.

Do these disk write numbers seem reasonable to you, or are these way too high?