Downloading pruned blockchain inefficient?

-ck

legendary

Activity: 4088

Merit: 1631

Ruu \o/

Quote from: gmaxwell on July 14, 2017, 09:46:22 PM

Quote from: -ck on July 12, 2017, 03:20:53 PM

If you want to run a pruned node but have the storage capacity to download the whole blockchain it is indeed much faster to download the whole blockchain and then restart bitcoind in pruned mode to do the pruning afterwards since it is constantly being pruned during the download process otherwise.

Have you actually measured that? If it's true-- it's a bug. the pruning should delete a whole blockfile at a time (128MB) and shouldn't take much time at all. It's totally plausible to me that there is a bug here, but I've not noticed it myself or heard it reported before.

Only when pruning was first available did I experience this and I did not do a fair comparison of identical hardware so no I cannot say for certain. I've not downloaded the whole blockchain since.

gmaxwell

staff

Activity: 4326

Merit: 8951

Quote from: -ck on July 12, 2017, 03:20:53 PM

If you want to run a pruned node but have the storage capacity to download the whole blockchain it is indeed much faster to download the whole blockchain and then restart bitcoind in pruned mode to do the pruning afterwards since it is constantly being pruned during the download process otherwise.

Have you actually measured that? If it's true-- it's a bug. the pruning should delete a whole blockfile at a time (128MB) and shouldn't take much time at all. It's totally plausible to me that there is a bug here, but I've not noticed it myself or heard it reported before.

SebastianJu

legendary

Activity: 2674

Merit: 1083

Legendary Escrow Service - Tip Jar in Profile

Thanks for the answer. It might be useful for the future to implement something like
if more than 100 blocks behind then download normally 50 blocks then prune on those. Repeat.

tspacepilot

legendary

Activity: 1456

Merit: 1083

I may write code in exchange for bitcoins.

Quote from: achow101 on July 12, 2017, 02:53:47 PM

Quote from: tspacepilot on July 12, 2017, 02:13:59 PM

@achow101: where I can read about the pruning implementation (obviously I can study the source, but I suppose there's a writeup somewhere)?

AFAIK, there is no write up of how pruning is implemented. You will have to look in the source code for that. Since manual pruning is now available via an RPC command, I would recommend that you start looking there to see what it does to prune: https://github.com/bitcoin/bitcoin/blob/master/src/rpc/blockchain.cpp#L849

Thanks! I also rhappened to re-look over the original whitepaper and I noticed (for the first time) the section on "reclaiming disk space". I guess I'm trying to figure out what the disadvantages are (if any) over running a pruned node vs non-pruned.

EDIT: I found some further notes about pruning in the release notes for 0.11: https://github.com/bitcoin/bitcoin/blob/v0.11.0/doc/release-notes.md#block-file-pruning

-ck

legendary

Activity: 4088

Merit: 1631

Ruu \o/

If you want to run a pruned node but have the storage capacity to download the whole blockchain it is indeed much faster to download the whole blockchain and then restart bitcoind in pruned mode to do the pruning afterwards since it is constantly being pruned during the download process otherwise.

achow101

staff

Activity: 3458

Merit: 6793

Just writing some code

Quote from: tspacepilot on July 12, 2017, 02:13:59 PM

@achow101: where I can read about the pruning implementation (obviously I can study the source, but I suppose there's a writeup somewhere)?

AFAIK, there is no write up of how pruning is implemented. You will have to look in the source code for that. Since manual pruning is now available via an RPC command, I would recommend that you start looking there to see what it does to prune: https://github.com/bitcoin/bitcoin/blob/master/src/rpc/blockchain.cpp#L849

tspacepilot

legendary

Activity: 1456

Merit: 1083

I may write code in exchange for bitcoins.

I was away from following development on core for some time. Last time I was up-to-speed pruning was just starting to be implemented. @achow101: where I can read about the pruning implementation (obviously I can study the source, but I suppose there's a writeup somewhere)?

achow101

staff

Activity: 3458

Merit: 6793

Just writing some code

Pruning will be more taxing on your disk as it has to constantly read and write to the disk far more than without pruning. This is because it deletes things on the fly. That means that, for each block, it will need to additionally compute what blocks should be deleted, look up where the block is, delete it, and remove it from the database. That will use extra CPU, RAM, and disk IO as there are simply more things that Core is doing in order to prune.

TexasHuck

full member

Activity: 126

Merit: 100

Ⓚ Kore Projects CTO Ⓚ

This has been my experience as well. I was thinking there could be some type of issue with the indexing of the values retained from the prune. I would interested to see a play by play of the prune in action. Even pseudo code would be helpful... thanks for posting!

Huck

SebastianJu

legendary

Activity: 2674

Merit: 1083

Legendary Escrow Service - Tip Jar in Profile

Hello,

I just download the blockchain in pruned mode. I downloaded the unpruned blockchain some days ago. It was way faster and I notice that downloading a pruned blockchain now takes way longer. It looks like takes more cpu and it uses the harddisc more often. It isn't the swap file also and I gave bitcoin-qt 6gb RAM and 16 threads. The same setup. Antivirus is also not running.

So can it be that the pruning is constantly done and writing and reading from the harddisc in an inefficient way? The same way like it works with single blocks when it is already synchronized? If so then maybe it would make sense to make the prune work just every couple of minutes. This might save ressources then.

Greetings!

Topic: Downloading pruned blockchain inefficient? (Read 1038 times)