Absolute minimal node disk use configuration

BlackHatCoiner

legendary

Activity: 1512

Merit: 7340

Farewell, Leo

Quote from: user32485724 on March 27, 2024, 09:41:27 AM

Is there a setting in bitcoin.conf that get's rid of chainstate?

There's a reason why you can't prune the chainstate. It's this one thing you absolutely need to verify that the transactions do not go against the protocol. The most you could do to save up space is enable prune=550 (doesn't go less than that).

It is theoretically possible to run a full node from home (where you probably have sufficient storage), and configure your private server to communicate with your home node. But, that's just too much effort. Just buy more storage on your private server.

DaveF

legendary

Activity: 3500

Merit: 6320

Crypto Swap Exchange

Did you TEST your app with testnet 1st? Why play with getting it working on the live chain and dealing with these issues before you know it is going to work.
If you have issues now you will not know if it's a programming / app problem or a configuration issues because you tried to squeeze it into something too small.

Also, you really should be using a hosted solution with a lot more storage & power. If it does work and you release it if you are that limited on storage and RAM it's going to get clobbered once it's in the real world.

-Dave

ranochigo

legendary

Activity: 3038

Merit: 4418

Crypto Swap Exchange

Quote from: LoyceV on April 11, 2024, 02:33:41 AM

Quote

Even if RAM is extremely reliable and that the server is never shutdown, Core would still attempt to flush chainstate to disk as the IBD progresses. Hence, it is necessary for the disk to have sufficient space to accomodate the chainstate, no way around that.

I've synced a pruned node in /dev/shm in the past. Works fine Tongue

That's including swap I suppose. If you're trying to keep your chainstate in your RAM, then that's the expected behavior (flushing) which is quite different from using your ram as a tmpfs or ramdisk. If you're using /dev/shm as a ram disk, then you can only use half of the RAM, and you'll have to consider the different overheads as well. Likely would require over 32GB of ram, which would probably mean that it's going to be a couple of times more than just adding half a terabyte to your server.

LoyceV

legendary

Activity: 3290

Merit: 16489

Thick-Skinned Gang Leader and Golden Feather 2021

Quote from: ranochigo on April 11, 2024, 01:18:40 AM

Regardless, the behavior for the chainstate flushing during IBD of a pruned node should be that the flushing occurs everytime that the blocks are removed from disk.

I've never tested it. It would be interesting to try: compare disk writes during IBD on a system with little RAM, to a system with 32 GB dbcache. I don't have the RAM to test this though.

Quote

Even if RAM is extremely reliable and that the server is never shutdown, Core would still attempt to flush chainstate to disk as the IBD progresses. Hence, it is necessary for the disk to have sufficient space to accomodate the chainstate, no way around that.

I've synced a pruned node in /dev/shm in the past. Works fine Tongue

ranochigo

legendary

Activity: 3038

Merit: 4418

Crypto Swap Exchange

Quote from: LoyceV on April 11, 2024, 12:44:49 AM

Any failure in your RAM pretty much fucks up your computer. Luckily, it's rare for RAM to fail after it's properly installed.

Yeah, that's why mission critical configurations have ECC and not for normal consumers. Even a slight corruption in the chainstate would be bad.

Quote from: LoyceV on April 11, 2024, 12:44:49 AM

Each time Bitcoin Core verifies a block, it reads/writes a lot to chainstate. If you're low on RAM, the disk will be the bottleneck.

Bitcoin Core only stores as much chainstate on the ram as you allow it. Keeping the entire chainstate on the RAM, permanently and not as a cache, would probably cause more problems than it solves.

Regardless, the behavior for the chainstate flushing during IBD of a pruned node should be that the flushing occurs everytime that the blocks are removed from disk. Even if RAM is extremely reliable and that the server is never shutdown, Core would still attempt to flush chainstate to disk as the IBD progresses. Hence, it is necessary for the disk to have sufficient space to accomodate the chainstate, no way around that.

LoyceV

legendary

Activity: 3290

Merit: 16489

Thick-Skinned Gang Leader and Golden Feather 2021

Quote from: ranochigo on April 10, 2024, 10:46:05 PM

Storage is far cheaper than RAM

That's true, but if you're low on RAM, more storage isn't going to help you.

Quote

note that since your entire chainstate is in RAM, any corruption to the RAM would result in a catastrophic failure and thus another reindex.

Any failure in your RAM pretty much fucks up your computer. Luckily, it's rare for RAM to fail after it's properly installed.

Quote

I doubt it wouldn't try to reserve the space for chainstate on the disk at all times. So it'll be far easier to just get more storage.

Each time Bitcoin Core verifies a block, it reads/writes a lot to chainstate. If you're low on RAM, the disk will be the bottleneck.

ranochigo

legendary

Activity: 3038

Merit: 4418

Crypto Swap Exchange

Quote from: NotATether on April 09, 2024, 02:49:16 AM

I *think* that if you set the dbcache to a high enough value then it's just going to keep the entire chainstate in RAM and then only flush it to disk on program exit (which I know is not very helpful but maybe you can delete the folder when the program is finished. Caveat: I'm don't know if this will make Bitcoin Core complain.)

Storage is far cheaper than RAM, and note that since your entire chainstate is in RAM, any corruption to the RAM would result in a catastrophic failure and thus another reindex. I doubt it wouldn't try to reserve the space for chainstate on the disk at all times. So it'll be far easier to just get more storage.

LoyceV

legendary

Activity: 3290

Merit: 16489

Thick-Skinned Gang Leader and Golden Feather 2021

Quote from: ABCbits on April 10, 2024, 03:49:44 AM

It'd require OP to rent VPS which have at least 12GB though. My Bitcoin Core's chainstate currently has size about 10.8GB, you also need 300MB of RAM for mempool and some more for OS.

And you can add 1 or 2 GB RAM per year to keep up with the growth Wink

ABCbits

legendary

Activity: 2870

Merit: 7490

Crypto Swap Exchange

Quote from: NotATether on April 09, 2024, 02:49:16 AM

I *think* that if you set the dbcache to a high enough value then it's just going to keep the entire chainstate in RAM and then only flush it to disk on program exit (which I know is not very helpful but maybe you can delete the folder when the program is finished. Caveat: I'm don't know if this will make Bitcoin Core complain.)

It'd require OP to rent VPS which have at least 12GB though. My Bitcoin Core's chainstate currently has size about 10.8GB, you also need 300MB of RAM for mempool and some more for OS.

Quote from: NotATether on April 09, 2024, 02:49:16 AM

Quote from: ABCbits on March 28, 2024, 04:40:37 AM

There are few third party services which let you perform RPC operation which supported by full node software they run. I'm only aware of getblock.io which offer such thing, but take note i never use that service and i don't know whether it's legit or fake service. So i strongly recommend you to do your own research.

Getblock is not a fake service, I have used it before.

That's good to know, i said that since i recall they used to perform somewhat aggressive advertising here (where some of them deleted by moderator).

NotATether

legendary

Activity: 1568

Merit: 6660

bitcoincleanup.com / bitmixlist.org

I *think* that if you set the dbcache to a high enough value then it's just going to keep the entire chainstate in RAM and then only flush it to disk on program exit (which I know is not very helpful but maybe you can delete the folder when the program is finished. Caveat: I'm don't know if this will make Bitcoin Core complain.)

Quote from: ABCbits on March 28, 2024, 04:40:37 AM

Quote from: user32485724 on March 27, 2024, 09:41:27 AM

Alternatively, is there a public rpc server that allows calls every 5 minutes or so?

There are few third party services which let you perform RPC operation which supported by full node software they run. I'm only aware of getblock.io which offer such thing, but take note i never use that service and i don't know whether it's legit or fake service. So i strongly recommend you to do your own research.

Getblock is not a fake service, I have used it before.

ranochigo

legendary

Activity: 3038

Merit: 4418

Crypto Swap Exchange

Most blockchain explorers offer a public API for users to query but that'll mean that you would have to trust their information to be accurate. If that fits your purpose, then you can try to use those: BlockCypher, SoChain for example. Caveat is that since you're not able to run your own node and verify the data, there is no way of knowing if they are accurate.

ABCbits

legendary

Activity: 2870

Merit: 7490

Crypto Swap Exchange

Quote from: user32485724 on March 27, 2024, 09:41:27 AM

Can that be shrunk further? I am looking for the minimum hard drive footprint because my cloud droplet only comes with 25 gigs and 12 gigs just aren't available. Is there a setting in bitcoin.conf that get's rid of chainstate?

Impossible, chainstate (which contain UTXO) is necessary to verify whether transaction violate Bitcoin protocol or not.

Quote from: user32485724 on March 27, 2024, 09:41:27 AM

Alternatively, is there a public rpc server that allows calls every 5 minutes or so?

There are few third party services which let you perform RPC operation which supported by full node software they run. I'm only aware of getblock.io which offer such thing, but take note i never use that service and i don't know whether it's legit or fake service. So i strongly recommend you to do your own research.

user32485724

newbie

Activity: 2

Merit: 0

Quote from: LoyceV on March 27, 2024, 10:06:32 AM

Quote from: user32485724 on March 27, 2024, 09:41:27 AM

Is there a setting in bitcoin.conf that get's rid of chainstate?

I don't think that's possible. Without chainstate, Bitcoin Core can't know which transactions in your mempool can't exist.

Quote from: Pieter Wuille on StackExchange

The chainstate directory contains the state as of the latest block (in simplified terms, it stores every spendable coin, who owns it, and how much it's worth).

thanks. If you're right, it boils down to mounting an extra drive or finding a public bitcoin node that allows external RPC calls. I need one call every 5 minutes, that's 288 per day

LoyceMobile

hero member

Activity: 1659

Merit: 687

LoyceV on the road. Or couch.

Quote from: Amphenomenon on March 27, 2024, 10:11:50 AM

I believe the maxuploadtarget is what you need for this

Wrong. Bandwidth has nothing to do with disk usage.

Amphenomenon

sr. member

Activity: 700

Merit: 470

Hope Jeremiah 17vs7

Quote from: ?? on ??

Here's what I use on my server:

Code:

~/bitcoin-26.0/bin/bitcoind -maxuploadtarget=500000 -dbcache=4096

Note that 4 GB dbcache is low for the IBD, but I don't want to give it 16 GB continuously. Normal file cache can handle it anyway. I simply extracted the compressed file in a home directory, and run it with user permissions. There's no need to edit config files, all options can be added to the starting command.
The 500 GB upload target is a daily limit, only set to prevent crazy things, but never came close to that amount.

I believe the maxuploadtarget is what you need for this

LoyceV

legendary

Activity: 3290

Merit: 16489

Thick-Skinned Gang Leader and Golden Feather 2021

Quote from: user32485724 on March 27, 2024, 09:41:27 AM

Is there a setting in bitcoin.conf that get's rid of chainstate?

I don't think that's possible. Without chainstate, Bitcoin Core can't know which transactions in your mempool can't exist.

Quote from: Pieter Wuille on StackExchange

The chainstate directory contains the state as of the latest block (in simplified terms, it stores every spendable coin, who owns it, and how much it's worth).

user32485724

newbie

Activity: 2

Merit: 0

Hi,

I am working on a web app for bitcoiners. The app needs to make rpc calls to a node to get information from the node. I am only interested in the mempool, noting else.

To that end, I would like to run a minimal node configureation on the webserver. I played around with configurations locally an found that pruning to the minimum still leaves a 12 gig heavy .bitcoin folder, because chainstate is 11 gigs.

Can that be shrunk further? I am looking for the minimum hard drive footprint because my cloud droplet only comes with 25 gigs and 12 gigs just aren't available. Is there a setting in bitcoin.conf that get's rid of chainstate?

This is the config that gets me 12gigs today:

server=1
daemon=1
prune=550
maxmempool=100
rpcuser=redacted
rpcpassword=alsoredacted
rpcallowip=127.0.0.1
rpcport=8332

Alternatively, is there a public rpc server that allows calls every 5 minutes or so?

Topic: Absolute minimal node disk use configuration (Read 278 times)