how-to clean up chainstate folder | Bitcointalksearch.org

nc50lc

legendary

Activity: 2534

Merit: 6080

Self-proclaimed Genius

Quote from: citb0in on January 31, 2024, 05:02:23 AM

Quote from: nc50lc on January 31, 2024, 01:06:01 AM

The command: --reindex-chainstate deletes the chainstate and creates a new chainstate from the blockchain.

Thanks a bunch in advance. What I still don't understand how two fully-synchronized non-pruned nodes can have different sizes of the chainstate. That makes no sense to me.

Go to the link in my previous reply to see the comment of a Bitcoin Developer in bitcoin.stackexchange site regarding the chainstate database size difference.

Quote from: citb0in

So will that result in a smaller or significant difference folder size after such a reindex ? Please provide some folder sizes before/after if possible.

I've already mentioned in the first reply that it'll not be a significant difference.

ABCbits

legendary

Activity: 2870

Merit: 7490

Crypto Swap Exchange

Quote from: citb0in on January 31, 2024, 05:02:23 AM

Quote from: nc50lc on January 31, 2024, 01:06:01 AM

The command: --reindex-chainstate deletes the chainstate and creates a new chainstate from the blockchain.

Quote from: ABCbits on January 31, 2024, 04:58:46 AM

Although as reminder, --reindex-chainstate will redownload whole blockchain.

So will that result in a smaller or significant difference folder size after such a reindex ? Please provide some folder sizes before/after if possible. Thanks a bunch in advance. What I still don't understand how two fully-synchronized non-pruned nodes can have different sizes of the chainstate. That makes no sense to me.

citb0in

Sorry, but i don't want to run reindex-chainstate since i only use HDD. But i don't expect much difference, where you only gain about 1GB free space. You better upgrade your storage, clear all cache on your OS and delete unused files.

BlackHatCoiner

legendary

Activity: 1512

Merit: 7340

Farewell, Leo

Quote from: citb0in on January 31, 2024, 05:02:23 AM

So will that result in a smaller or significant difference folder size after such a reindex ?

It will result in a small difference. If you have a storage problem, reindexing for the sake of restructuring your chainstate is definitely not a recommended option.

Quote from: citb0in on January 31, 2024, 05:02:23 AM

What I still don't understand how two fully-synchronized non-pruned nodes can have different sizes of the chainstate. That makes no sense to me.

Due to heuristics, two LevelDB databases may have the same content, but it is not guaranteed that they are byte-to-byte identical, with the exact same size. Nodes running for different durations exhibit different write patterns. For instance, a node running for years may have more frequent but smaller writes, whereas a node completing the IBD may have fewer but larger writes.

Maybe a Bitcoin Core developer can provide us a better insight.

citb0in

hero member

Activity: 630

Merit: 731

Bitcoin g33k

Quote from: nc50lc on January 31, 2024, 01:06:01 AM

The command: --reindex-chainstate deletes the chainstate and creates a new chainstate from the blockchain.

Quote from: ABCbits on January 31, 2024, 04:58:46 AM

Although as reminder, --reindex-chainstate will redownload whole blockchain.

So will that result in a smaller or significant difference folder size after such a reindex ? Please provide some folder sizes before/after if possible. Thanks a bunch in advance. What I still don't understand how two fully-synchronized non-pruned nodes can have different sizes of the chainstate. That makes no sense to me.

citb0in

ABCbits

legendary

Activity: 2870

Merit: 7490

Crypto Swap Exchange

Quote from: nc50lc on January 31, 2024, 01:06:01 AM

Quote from: ABCbits on January 30, 2024, 05:35:49 AM

Do you have personal experience or know documentation that --reindex-chainstate actually reduce chainstate size? After all, chainstate basically just LevelDB (key/value) database.

His chainstate folder is 11GB in size and based from his post history, he's been using his node for a few years now.
On the other hand, I have a recently fully synced node with 9.8GB chainstate folder.

Good point. I just found out size of my chainstate folder has smaller size.

Quote from: nc50lc on January 31, 2024, 01:06:01 AM

The command: --reindex-chainstate deletes the chainstate and creates a new chainstate from the blockchain.

Here's a post in bitcoin stackexchange that explains how it work: https://bitcoin.stackexchange.com/a/117435

I already know LevelDB has some optimization, but didn't know it doesn't fully optimize/reduce size of it's DB files. Although as reminder, --reindex-chainstate will redownload whole blockchain.

nc50lc

legendary

Activity: 2534

Merit: 6080

Self-proclaimed Genius

Quote from: ABCbits on January 30, 2024, 05:35:49 AM

Do you have personal experience or know documentation that --reindex-chainstate actually reduce chainstate size? After all, chainstate basically just LevelDB (key/value) database.

His chainstate folder is 11GB in size and based from his post history, he's been using his node for a few years now.
On the other hand, I have a recently fully synced node with 9.8GB chainstate folder.

The command: --reindex-chainstate deletes the chainstate and creates a new chainstate from the blockchain.

Here's a post in bitcoin stackexchange that explains how it work: https://bitcoin.stackexchange.com/a/117435

ABCbits

legendary

Activity: 2870

Merit: 7490

Crypto Swap Exchange

Quote from: nc50lc on January 29, 2024, 07:47:53 AM

If your node is running for a few years now, re-syncing Core or --reindex-chainstate may produce a more "optimized" chainstate
but it wont be a significant change in size, maybe a reduction of 1GB at best.

Do you have personal experience or know documentation that --reindex-chainstate actually reduce chainstate size? After all, chainstate basically just LevelDB (key/value) database.

Quote from: citb0in on January 29, 2024, 06:10:49 AM

This of course goes beyond the actual purpose of a pruned node.

But it's technical limitation of pruned node, since you need all "spendable" UTXO to verify all new TX/block.

Quote from: citb0in on January 29, 2024, 06:10:49 AM

Why does the folder content of chainstate rise to this immense height?

1. Ordinals (mainly BRC-20 token).
2. Rising Bitcoin popularity.

Quote from: citb0in on January 29, 2024, 06:10:49 AM

What is the best way to proceed in such a case?

Allocate more storage. I don't expect Bitcoin Core will implement compressed chainstate feature.

pooya87

legendary

Activity: 3472

Merit: 10611

Quote from: NotATether on January 29, 2024, 07:57:09 AM

Especially Ordinals. I think all this minting is contributing to make the UTXO set bigger at an even faster rate since selling and buying an Ordinal makes the same outputs as it does inputs (IIRC).

Pretty soon, I think we will need drop provably-unspendable outputs from the chainstate.

Unfortunately the junk outputs Ordinals attackers create are not provably unspendable. A lot of them are just dust outputs that the standard rules prevent from being spent. Other broken scripts* (including those used for Ordinals Attack) don't contribute enough to the bloat to justify the effort of writing the code to "purge" them from the database.

The OP_RETURN outputs (the only provably unspendable outputs) have always been dropped as a.chow said.

* Example of a provably unspendable broken script that I'm talking about:

Code:

OP_DUP OP_HASH160  OP_EQUALVERIFY OP_CHECKSIG

achow101

staff

Activity: 3458

Merit: 6793

Just writing some code

Quote from: NotATether on January 29, 2024, 07:57:09 AM

Pretty soon, I think we will need drop provably-unspendable outputs from the chainstate.

We already do.

NotATether

legendary

Activity: 1568

Merit: 6660

bitcoincleanup.com / bitmixlist.org

Quote from: nc50lc on January 29, 2024, 07:47:53 AM

Quote from: citb0in on January 29, 2024, 06:10:49 AM

This of course goes beyond the actual purpose of a pruned node. Why does the folder content of chainstate rise to this immense height?

Your pruned node is a full node too, it needs the chainstate to verify all the transactions that it continuously receiving from its peers.
It's increasing (at a very slow rate) because of UTXOs that are being spent and created.

Especially Ordinals. I think all this minting is contributing to make the UTXO set bigger at an even faster rate since selling and buying an Ordinal makes the same outputs as it does inputs (IIRC).

Pretty soon, I think we will need drop provably-unspendable outputs from the chainstate.

nc50lc

legendary

Activity: 2534

Merit: 6080

Self-proclaimed Genius

Quote from: citb0in on January 29, 2024, 06:10:49 AM

And how can this immensely high disk usage of the chainstate folder be cleaned up? Simply deleting it is certainly not a good option. What is the best way to proceed in such a case?

You can't clean it, it contains the "UTXO set" based from the blocks that you've verified.

If your node is running for a few years now, re-syncing Core or --reindex-chainstate may produce a more "optimized" chainstate
but it wont be a significant change in size, maybe a reduction of 1GB at best.

Quote from: citb0in on January 29, 2024, 06:10:49 AM

This of course goes beyond the actual purpose of a pruned node. Why does the folder content of chainstate rise to this immense height?

Your pruned node is a full node too, it needs the chainstate to verify all the transactions that it continuously receiving from its peers.
It's increasing (at a very slow rate) because of UTXOs that are being spent and created.

citb0in

hero member

Activity: 630

Merit: 731

Bitcoin g33k

In short: a pruned node that uses the minimum possible value "prune=550" shows a folder structure of

./blocks = 704M
./chainstate = 11G

This of course goes beyond the actual purpose of a pruned node. Why does the folder content of chainstate rise to this immense height? And how can this immensely high disk usage of the chainstate folder be cleaned up? Simply deleting it is certainly not a good option. What is the best way to proceed in such a case?

Topic: how-to clean up chainstate folder (Read 193 times)