Topic: Are High Disk Writes Expected? (Read 362 times)

This is interesting discovery. Usually i would suggest people to consider ElectrumX or Fulcrum when they want to run Electrum server, but it'll make me prioritize to recommend Fulcrum instead.


If you strictly follow the guide, i think it means ElectrumX have default poor configuration rather you misconfigured ElectrumX.

I think I'm just going to use Fulcrum. It's working well, and these results are promising. I have been running iotop for about 14 hours now (first column is cumulative disk writes, second column is process name):


Between all the Fulcrum variations in the list above, it looks like just under 1 GB was written in 14 hours. In the same timeframe, electrumx would have written around 70 gigabtyes to disk (seemed to average around 5 GB per hour). Crazy difference.

There are three possible causes I can think of for now:

1. I have electrumx misconfigured somehow. If other people were observing this, then this is probably not the true. However, nobody else running electrumx reported back to me, so I have no evidence that this is true.
2. Some of the code is poorly implemented somewhere or there is a bug in electrumx. I also have no evidence for this.
3. This is a difference between leveldb and rocksdb. I only tested electrumx with leveldb. Fulcrum uses rocksdb. Each of these simply may handle things differently. It's possible I wouldn't observe this if I used electrumx with rocksdb instead.

At this point, I'm not really willing to sink any more time into this to get to the root cause, since Fulcrum is working great. I'm just going to continue using it for now.

Thanks for your help, everyone.

Yes, it create log file. But based on previous discussion and example on https://electrumx.readthedocs.io/en/latest/HOWTO.html#understanding-the-logs, i expect bunch of these which isn't very helpful.


But if you or someone else know how to make ElectrumX perform flush less frequently, it probably can reduce it's write activity.

doesn't electrumx have a log file? why not open it and take a look at what it's doing? maybe it will tell  you when it is doing disk writes and what they are for. any reputable software would.

Both. I wasn't seeing any difference in the RAM usage or any difference in the disk writes when using 16 GB of RAM relative to 8 GB, even after increasing the dbcache in Bitcoin Core and the CACHE_MB in electrumx.

Also, Fulcrum finally fully finished indexing/processing all the blocks, so I'll let it run for a couple hours and monitor the disk writes. It will be interesting to compare to electrumx.

Did you miss this part?

No difference in memory usage, or no difference in disk writes? I guess it depends on what your Electrum server is doing: Bitcoin Core would benefit from larger dbcache, I don't know anything about Electrum server.

Agreed.

I'm not sure about this. As I mentioned, not very much RAM was being used according to Xubuntu's built-in task manager, and when I put in 16 GB instead of 8 GB of RAM, I noticed absolutely no difference. I highly doubt the processor would impact the amount of disk writes.

Once again, it's not like the performance is bad; it's actually been entirely sufficient for me. I just can't figure out why ElectrumX is writing so much to disk. This very well could simply be how it was designed, and everybody, regardless of hardware, would experience this.

1tb ssd
32gb ram
I5 8500  or newer

using the Celeron and 8gb ram is simply a shit show.


I have run a ton of different  wallets in fact I will be doing a new one as I now have 1 gb  vs my old internet

I'm giving Fulcrum a try right now. The database is currently building. It uses rocksdb while electrumx (by default) used leveldb, so maybe the write activity will be different. I will report back once it's all finished.

Using HDD seems to be poor choice. If i were you, i would try different Electrum server implementation such as Fulcrum or Electrs. Sparrow Wallet developer even write a benchmark about it on https://www.sparrowwallet.com/docs/server-performance.html.

I wouldn't move the writes to a HDD. That's terrible for performance. Note that cheap SSDs are usually lower quality, both in endurance as well as performance. I'd just let it be, keep backups of the data, and by the time your SSD is "worn" in a few years, replace it with a bigger one.

At the moment I cannot find any settings to change for electrumx that would lower the disk writes. As a temporary solution for now I'm thinking I might use an extra 2.5 inch HDD from one of my old laptops and dedicate it to just the electrumx db folder (this is where all the writes are happening) so I can keep it separate from the SSD the OS and Bitcoin Core are installed on. Obviously performance will be worse but I'm not sure it will be bad enough to really matter for me. Either that or I will use a much cheaper extra 256 GB SSD instead.

Assuming the output comes after you use 16GB RAM and set higher dbcache value, it's not surprising since Bitcoin Core would use disk less frequently. I saw you asked on GitHub whether changing CACHE_MB value may solve the problem. But description under CACHE_MB also mention ElectrumX write something called "unflushed history" frequently which doesn't seem could be changed.


I recall their node software also recommends using SSD if you store the data on HDD. Although i don't know if they still show that recommendation today.

I've never used Electrum server, but I'm curious: what could it possibly be doing that requires so much disk writes?

This has nothing to do with Monero.

and that's after he'd checked out factory specs. hopeless case

Funny you say that because the electrumx devs specifically recommended using an SSD. (As I pointed out above, it's electrumx that is doing all the disk writes, not Bitcoin Core.)

say monero clearly stated that their node will ruin ssd. i'll rather go with sas hba and sas hdd. used sas1 are cheap now as  are sas2 hdd's, it's enterprise-grade things.
by hba i mean raid card flashed to IT mode (it'll be LSI one most probably). Because if it's not flashed already you'll need to do this https://www.reddit.com/r/DataHoarder/comments/lck2ou/comment/gm93zpp/      on your own.

Thanks for suggesting iotop. I've been running it for about 10 minutes and I already get the impression electrumx is doing all the disk writes, not even Bitcoin core. After just 10 minutes I'm seeing:

Disk Writes          Command
3.36 G                 python3 /home/user/Desktop/electrumx/electrumx_server
17.97 M               [jdb2/sda2-8]
6.52 M                 systemd-journald
3.84 M                 bitcoin-qt -min -chain=main [b-scheduler]
3.75 M                 bitcoin-qt -min -chain=main [b-msghand]
3.36 M                 tor --defaults-torrc /usr/share/tor/tor-service-defaults-torrc -f /etc/tor/torrc --RunAsDaemon 0

So it's almost certainly my elecrumx configuration I'll need to scrutinize, not Bitcoin core.

Edit: Here's what I'm seeing after one hour:

Disk Writes          Command
8.49 G                 python3 /home/user/Desktop/electrumx/electrumx_server
56.66 M               [jdb2/sda2-8]
23.80 M               systemd-journald
20.05 M               bitcoin-qt -min -chain=main [b-msghand]
16.27 M               bitcoin-qt -min -chain=main [b-scheduler]
13.89 M               tor --defaults-torrc /usr/share/tor/tor-service-defaults-torrc -f /etc/tor/torrc --RunAsDaemon 0

Each new block requires verification, which includes reading and writing to chainstate.

Usually the mainboard is the limiting factor.

With 16 GB RAM, that doesn't leave much for the rest of your OS. I'd go for no more than half your RAM, so try 8 GB dbcache if you install 16 GB RAM.

That should be enough. For what it's worth: my system has 16 GB RAM (and the mainboard can't handle more), and during IDB I increase dbcache, but when normally running I reduce it to 1024 again. I don't want Bitcoin Core to occupy the majority of my RAM, I need it for other things.
I just checked: I've written 7.5 TB in the past year (I didn't do a new IBD on it). That's 21 GB written per day, for Bitcoin Core, swap and everything else running on my system. I keep chainstate on this SSD, and blocks on my HDD. At this rate, I'll reach my SSD's rated writes after 80 years.

I'm getting the impression that adding more RAM mostly reduces disk writes before the node is fully synced, not after. But I have extra RAM and I can bump it up to 16GB so I can give it a shot. The Celeron supports up to 64 GB of RAM according to Intel's website. I could also put in 32 GB later (I can swap it out from another old computer), but I'll start with 16 for now.

From what I can tell, I should set this line in bitcoin.conf (12288 = 12 * 1024):


Is that the only thing I need to change? Or are there other settings I should try too?