Here is my idea for avoiding blockchain bloat while doing 1000TPS. I am surprised nobody mentioned this, so it is either because it is a really bad idea for some reason, or too obvious nobody bothered??
From my understanding, it will be possible to have regular snapshots of the entire state of the NXT blockchain so that you wont need to parse all the blocks from genesis.
Secondly, very little of this "entire state of NXT" data has a life beyond 1440 blocks.
So, my proposal is to generate a daily snapshot, peer reviewed by nodes, checksummed, fingerprinted, signatured, whatever we need to make sure it is a valid unmolested snapshot. We actually dont even need these snapshots, but while we are doing this, might as well avoid having to download the entire blockchain. Bandwidth savings alone makes it worthwhile.
OK, so one way or another, let us assume the node is current. Now the problem is keeping up with 1000TPS (lets make the overall network adaptive so we can handle bursts of 1000TPS, sustained 250 TPS) and that requires bandwidth, though with binary data, 250 TPS should be around what 100TPS will be now. So 100kbps would be enough to handle bursts of 1000TPS and sustained 250TPS
But where does all that data go?
Actually, I say just throw it away! Why can't we use a FIFO that stores the most recent 1441 blocks for all the blockchains (when we go parallel). Since we synced the the full blockchain with the last days checkpoint, then as long as we are properly updating the "entire state of NXT network" at any given block, we are able to forge a valid block.
Unless I totally missed some reason why we need to locally store more than the last 1441 blocks, this should work. We can then specify that any NXT VM (Turing scripts) will need to be designed to use only data from prior 1440 blocks. This I do not see as a big limitation as if it is really important the client issuing script can just get data it needs and put it into AM. So, the NXTcore would need to implement the FIFO, preferably based on a web.xml parameter. That would allow people who run NXT VM generating clients to have access to exactly the window of time they need.
I hope this puts useful forging back on the table for raspis. 100kbps to fully support 1000TPS (peak output) NXT network
James
I think, snapshots were mentioned or spoken about a couple of times.
Yes, snapshots were, but not using a blockchain FIFO (other than xyzz's post that was ignored). That is what allows the raspi to keep up. Even without snapshots, a raspi would be able to keep up, as long as it ever could catch up. That is why I combined it with snapshots to make sure all raspis could catch up to the current block.
Using snapshots alone would require much more frequent snapshots to be made and created all sorts of issues with delays, etc. If the blockchain FIFO is implemented, we can get by with weekly snapshots. This is because the blockchain is not stored locally other than the most recent day.
Also, we dont need parallel blockchains because of HDD usage if we had blockchain FIFO. There could be other reasons like partitioning workload, etc., but it is a quick way to get to 1000TPS and beyond without any software magic.
I saw the post about this, but nobody else seemed to recognize the significance. I just felt that it solved the 1000TPS on raspis issue and so wanted feedback on it to make sure it would work. It just seems so simple an idea, I wonder why it wasnt already done
James