Topic: [ANN] AEON [2019-09-27: Upgrade to version 0.13.0.0 ASAP HF@1146200 Oct 25] - page 225. (Read 625666 times)

Pruning FAQ

Q: What is pruning?

A: Pruning refers to removing unnecessary information from the blockchain once it is no longer needed.

Q: What are the advantages?

A: Pruning reduces the amount of storage needed for the blockchain. If the blockchain is loaded into RAM (which is the case in the current implementation), then it also reduces RAM usage. Finally, it also reduces the rate that these resources are consumed over time as the blockchain grows. In the current implementation, each time one new block is accepted, one old block is pruned, on average freeing up a large portion of the resources needed for the new block.

Q: How does pruning affect functionality?

A: The only inherent functional limitation of AEON's pruning is the inability to restore (also known as rescan) a wallet which was used for spending transactions. All other functionality including sending and receiving coins, mining, updating a wallet after any period of inactivity, cold storage, mining, etc. remain fully supported. There is a current limitation that after extended period of disconnection (>28 days) a node may have trouble resynchronizing with the network, and would need to be reset. This is not inherent and should be addressed later.

Q: Does pruning reduce the effectiveness of ring signatures for transaction privacy?

A: No, ring signatures and privacy are unaffected.

Q: How is pruning enabled and disabled?

A: Using the --pruning option on the daemon command line. With this option the daemon will prune the blockchain and will also switch from using the blockchain.bin file for storage to blockchain-pruned.bin. To switch an existing node into pruned mode, copy blockchain.bin to blockchain-pruned.bin before starting the daemon with --pruning. To switch pruning off, remove the --pruning option. Do not, however, copy blockchain-pruned.bin to blockchain.bin. This will not work. You will need to have an unpruned blockchain.bin file or resync unpruned from the network.

Q: How does AEON's pruning compare with Boolberry's pruning?

A: AEON prunes slightly more information from the blockchain, so the required storage is slightly smaller (given equivalent usage), though the difference is likely not particularly significant. BBR prunes the blockchain on the entire network while AEON prunes on each node individually (and only if pruning is enabled). This means that new nodes can come online faster with BBR, but those new nodes are unable to independently verify the entire blockchain. It is possible to download an unpruned BBR blockchain from a web site to independently verify it, but in that case the amount of data downloaded would be the same as AEON. It also means that every BBR node is able to serve the chain to new users but in AEON this function falls to nodes that are unpruned, also known as "archive nodes" (or alternately via a trusted bootstrap file).

Q: How does AEON's pruning compare with Bitcoin's pruning?

A: In Bitcoin 0.11, the same model of pruning is implemented as in AEON. That is, nodes prune blocks locally, after syncing from an unpruned chain and verifying the chain independently. Like AEON, Bitcoin pruned nodes can't rescan or reindex wallets. Bitcoin 0.11 does not support wallets on pruned nodes at all, so it currently has more limitations that AEON. Old Bitcoin blocks must be retrieved from unpruned archive nodes, as with AEON.

Q: What other coins implement pruning?

A: Other than Boolberry, Bitcoin, and possibly some coins which have inherited their implementation by being Bitcoin forks, no other coins implement pruning at this time.

Q: What's this about "archive nodes"? How can I run one?

A: When nodes connect to the network, they retrieve blocks from other nodes. If only blocks within the most recent 10 000 (approx.  28 days) are needed for syncing, even pruned nodes can provide them. However, in the case of nodes which are brand new (syncing from the genesis block) or which have been offline for >28 days, pruned nodes will be unable to supply the older blocks. Instead this task falls to unpruned nodes, also known as archive nodes. For the time being the project-run seed nodes will always run in unpruned mode, and others with sufficient RAM and storage space who wish to support the network are also encouraged to do so. To run an archive node, simply start the daemon in the normal manner, without the --pruning option.

Q: What are the numbers? How much does pruning reduce the amount of memory and storage needed?

A: The exact numbers will vary according to OS, compiler, etc. and also depend on the usage of the blockchain in the future. One early report from BoscoMurray stated, "RAM usage down from 4.8GB to 2.4GB and blockchain file size down from 3.2GB to 1.7GB"

Q: That doesn't seem like a big reduction. Why is the benefit not greater?

A: To explain why the reduction is not larger and understand what this means for the future, let's first review some basics of how a blockchain works.

Every time a coin is spent, a digital signature accompanies the transaction in order to show that the owner of that coin authorized spending it. Once this signature is checked, it is no longer needed. This is the largest portion of what is being pruned. In the cryptonote signature scheme (used by AEON), ring signatures used for anonymity, which means while a signature shows the coin owner authorized the transaction, unlike conventional signatures, it does not identify the specific coins being spent, and therefore does not allow tracing and analyzing the blockchain. As a side effect of this functionality, these signatures are much larger than ordinary digital signatures (a fact sometimes described as "cryptonote bloat"). Thus in AEON, pruning offers great benefit and eliminates the "cryptonote bloat".

So why is the savings not larger? Because in the early history of AEON, there was a very large number of very large transactions and many of those transactions did not using ring signatures. This happened for several reasons, including a major flaw in the early versions of cryptonote mining pool software. Thus, while the chain is relatively large, a relatively small amount of storage is saved by pruning the early transactions.

The newer transactions are a different story. The pool software flaw was fixed long ago, and most transactions now do use ring signatures. So the savings from pruning going forward should be much higher (perhaps 75-80%). Of course, since ring signatures are now being routinely used, it means that that actual anonymity of the coin in practice is greatly improved, and with pruning there is no long term storage cost (i.e. no "cryptonote bloat") for most nodes (other than archive nodes). We get the best of both worlds!

Q: What about a database or disk-based block storage? Doesn't that also reduce memory usage?

A: Storing the blockchain out-of-memory in a database or in cache files reduces memory usage but does not reduce storage usage nor reduce the rate of growth of storage usage over time. In AEON, the plan is to later support out-of-memory storage of the blockchain along with pruning, so memory usage will be further reduced, and storage usage will remain low and grow very slowly over time.

It's working on Ubuntu 14 x64. RAM usage down from 4.8GB to 2.4GB and blockchain file size down from 3.2GB to 1.7GB.

Pruning (aka Light Full Node) test release

This release optionally prunes the blockchain by dropping no-longer-needed information once it is processed and verified by your node. This reduces both memory and storage requirements, and perhaps more importantly reduces the rate at which both increase over time. With pruning enabled, the daemon continues to function as a full node on the network in nearly every way. I will be posting a FAQ to answer some other questions about the details of the pruning functionality.

Please test, especially if you have a smaller memory configuration. Very small memory configurations (<2 GB? -- I'm not sure) won't work with this, but if you have something that has been barely able to run the daemon previously, this should improve your performance quite a bit. Certainly some smaller-memory systems (at a minimum 3-4 GB of RAM, if not smaller) should be able to successfully run a node now. I'm not sure if any 32-bit operating systems can handle it, although it is possible.

Since converting an existing blockchain requires enough RAM to at least load it first, resyncing from scratch with the --pruning option may be a better option for smaller-memory systems until pruned bootstraps become available.

Instructions:

1. Copy your blockchain.bin file to blockchain-pruned.bin in the same location. The daemon uses different files to store the pruned and unpruned versions of the blockchain. Once testing is complete if you want to delete the unpruned blockchain to free up disk space that is fine, but I don't recommend it right away EDIT: there have been some issues with compatibility of the .bin files so I recommend backing up all of those files before using a test release.

2.
3. Start node with the --pruning option. You should get a message at startup about the blockchain being pruned (this is done using the blockchain-pruned.bin file; your original blockchain.bin file is unchanged).

4. Exit node to save the pruned blockchain and reset RAM usage.

5. Start node again with the --pruning option.

6. If any problems occur you can switch back to unpruned (start daemon without --pruning)

Report any issues (or even if no issues)

Limitations: a pruned node can not be used to rescan or restore a wallet that is older than the pruning period (10 000 blocks, or about 28 days). Also, if a pruned node is disconnected from the network for more than the duration of this pruning period, it will potentially not be able to recover on its own, and will need to be resynced from scratch or with a bootstrap. The second restriction is fixable and can be addressed later. Otherwise it should be functionality complete.

Neither this test release nor the pruning feature is recommended for important nodes such as pools, exchanges, etc. Please continue to use the standard version at this time for those purposes.

Thanks for testing!

We are waiting for news and will test.

Pruning update

I further reviewed the issues surrounding pruning and I was able to remove most of the limitations discussed in the previous update. A normal wallet will work even if you are offline for an arbitrary period of time (specifically longer than the pruning period of 10 000 blocks or about 28 days).

The only remaining limitation is that you will not be able to restore (or rescan) a wallet using a pruned node (this function will require an unpruned aka archive node). This is equivalent to the limitation of pruning in Bitcoin (which does not allow reindexing; essentially equivalent to our restore or rescan), although we are slightly ahead of Bitcoin in terms of functionality since they currently can't support a wallet at all in pruned mode


I'm finishing the last of my own testing on the revised implementation and will be pushing a test release to github within 1-2 hours. I look forward to any and all test reports!

We need in touch with minergate, it is the most convenient way of mining and a great community.

Sounds good, mining there now. Nice thing about Minergate is that anyone can easily set up and mine on most commonly used computers. 

As I like this one long term, CPU mining even makes sense.

Seems to be yes. For a while they claimed to be up but their pool stats page was still broken. Now it appears to show a recently found block, so I guess it is fixed.

I think, also minergate is alive.

Just a note to say that my pool appears to be comfortably past 50% at the moment, and that while I'm not planning on 51%ing aeon, you wouldn't just trust some guy on the internet not to, right ?
While Arux's pool appears to be down, the daemon can solo mine, which should still be a reasonable option for large hash miners that can handle the RAM usage.

Yes, there is a --pruned option. Without the option it runs in the traditional unpruned aka "archive node" manner.


That is planned, just waiting for that code to become fully stable with the last remaining issues worked out.

Also, the plan is for pruning to coexist with a database, and that has factored into the design. The benefit in that case is reduced, but perhaps more importantly slow-growing, database size. This will allow small devices to continue to run nodes long term without filling up storage.

Thanks for the explanation, Smooth. If this is configurable on the daemon, I'll be happy to support an archive node. Merging LMDB support would avoid the lengthy and dreaded 12-hour "saving blockchain" pause. 

With the current model of pruning there will never be a case that the whole network contains only pruned nodes. There still need to be some unpruned (also known as archive nodes) to provide the old blocks to new users. This is the same model used by Bitcoin currently. My long term plan is provide a mechanism of payment to ensure that people want to provide these archive nodes. Short term the seed nodes at least will run archive nodes and I'm recommending that important services (exchanges, pools, etc.) not run pruned nodes at this time so those will be there too.

Longer term there will be a different model of pruning and wallet syncing that will fully work with pruned nodes for both regular and lightweight wallets. Lightweight wallets will work this way first, probably.

Smooth, in the scenario where eventually the whole network contains only pruned nodes and users have cold wallets, if the wallets were inactive for over 28 days and users decide to buy more coins, what would happen?

I guess I'm not understanding the issue here, sorry. The requirement of having to refresh a wallet every 28 days won't work with cold wallets, I'd say.

 

@smooth, fwiw, I concur with your reasoning on how to address this wallet refresh issue and pruning strategy. Good stuff! 

One last follow up before I drop a release (soon!)

I investigated the wallet problem. There are two ways of fixing it. One is to reduce the degree of pruning and keep more old data, and the other is to disallow syncing an old wallet from a pruned node. After implementing the first I reconsidered and decided to go with the second. Not only does the first approach reduce the amount of pruning now, it will further reduce how much pruning can be improved later.

The new approach will only support "active" wallets off a pruned  node. The pruning window is 10 000 blocks or around 28 days, so you will need to sync (open and then exit) your wallet at least once per 28 days from a pruned node to keep it up to date, otherwise you will need an unpruned node to sync the wallet. If there is no activity at all (received payments) during the offline period, then it will still be okay to sync from a pruned node, even after 28 days.

I feel this limitation is acceptable for most if not all use cases.

Awesome. Thanks for the hard work.

crazy trading traffic for aeon on bittrex  that way my otc order will never get filled 

To follow up, I did discover one issue with wallet refreshes in testing. It is easily reproducible so I'm sure I will track down the cause and get it fixed soon.

EDIT: I did find the problem in the wallet and unfortunately some more work is going to be needed so the test release of pruning will be pushed back a day or two.