Topic: Block-Size Proposal: "Pruning Blocks" (Read 1516 times)

I'm rethinking the need to include the actual UTXO list in the blockchain.

Similar functionality can be achieved by including just the hash of all UTXO's in the block header.  The actual UTXO's would then be stored and published off-chain as desired by 3rd parties.  This reduces the amount of storage needed on the blockchain, but still provides checks and balances for new nodes to get the head of the blockchain with the correct UTXO set.

Here would be a typical use case:

  1) A new node downloads a UTXO list from a third party for a block that meets their desired safety standards (older block = more safety, newer block = more risk).
  2) The blockchain is downloaded from that block on.
  3) The UTXO list is hashed and compared to the UTXO hash of the block. 
    a) If the hash does not match, goto 1. 
          Note:  This forces third parties to come up with a balance list that has the exact same hash as the actual UTXO set balance.  Hash collisions allow for other potential fake balances to match at this point.
  4) The remaining blocks are downloaded.  For each block:
    a) The UTXO list is updated for the new block, hashed, and compared against the UTXO hash of that block.
    b) If the hashes do not match or the block is not valid, goto 1.
          Note:  This will virtually guarantee a true UTXO balance with an appropriately selected hashing method.  They may be able to fake the first block's UTXO balance, but the odds of being able to apply the transactions on top of that and get the same UTXO hash as the actual UTXO list for the next block will be astronomically small, and astronomically smaller with each successive block.
  5) Verify the head of the chain matches the head of the desired blockchain.
    a) If not, goto 1.
  6) The new node now has a dependable UTXO set and the desired blockchain.



My main concern with this is the affects on miners generating the hash of UTXO's for each block.  For example, when a pool receives a new block, they must first validate the UTXO hash for that block before they accept it, and then create the UTXO hash for the next block before they can begin work on the next one.  This means that the miner that successfully mined the last block will have a head start on everyone else (since they already validated their UTXO hash), and it will also mean downtime for pool miners while they wait for their pools to get the new UTXO hash and block for them to do work on.  Also, when a transaction is added to a block, the UTXO hash must be re-generated causing further delays, which will decrease the incentive to include more transactions as they come. 

I believe those concerns can be alleviated by choosing an appropriate UTXO hashing algorithm where additions and subtractions to the UTXO list will not require a full scan to create the new UTXO hash.  Furthermore, because of step 4b above, the hashing algorithm must also be chosen such that two UTXO lists that share the same hash but have different contents should not (except for rare, unpredictable collisions) produce the same new hash after the same transaction list is applied to both UTXO lists. 

That isn't a good assumption to make. Sybil attacks on the bitcoin network are still possible.

What constitutes a too low difficulty? How would a new node determine that?


While that could work, a malicious miner could generate blocks with timestamps several months in advance and pregenerate them.

That could work too prevent 2b from happening but what if the miner pregenerates transactions? The transactions in the blocks are pregenerated and broadcasted to the victim node at the appropriate times?

IIRC bitcoin core already does that check for new blocks anyways. Lookup the timejacking attack.


That is always a safe backup.


Well also having those UTXOs will inflate that block size because it has to include all previous UTXOs.

Yes but there will always be a limit so the blockchain will always grow linearly once the maximum is reached. Also there are other scaling solutions than just the block size limit such as lightning and sidechains.

A sybil attack is still possible, you cant just assume it away.

Very good points.  As a core assumption of Bitcoin, a single miner can not replicate the difficulty of the entire network.  Using that knowledge, here is a first-take at such a procedure to download and verify a pruned blockchain:

1) A new miner downloads all unique pruned blockchains visible to them.  It is assumed that one of them will be correct because the Bitcoin network is the most powerful network in the world.
2) If there are no competing chains, great!  If there are competing chains, they must be compared to find out which is the true blockchain and which is/are the counterfeit(s).  As you pointed out it is easier to counterfeit back to a Pruning Block than the blockchain because you do not need to generate all blocks back to the genesis block, you just need to lie with the pruning block you start with.  Thus, we can take additional precautions:
  a) Check the block difficulty level of the blocks.  If it is too low throw the blocks out.  This will limit the people who could take advantage of this attack to only very powerful miners, as you can't fake the difficulty of a block.
  b) Download back to the last 2 pruning blocks instead of just the last 6 regular blocks.  At 1008 blocks between pruning blocks plus 6 confirmations for the pruning block, this means the malicious miner would need to generate at least 1014 near the current difficulty of the actual network instead of 6.  This will limit the risk to extremely powerful miners, as even a miner/pool with 10% of the network would take 2.5 months to do this.
  c) Check that current transactions are getting through in a timely fashion (e.g. monitor for 6+ blocks for anything fishy).  Your node can monitor all published transactions and make sure they are included.  If they are not being included in one chain they you know something is fishy with that.  This will prevent against pre-generating 2000 blocks and then re-playing the blocks back.
  d) Validate that times of the blocks make sense.  This will prevent re-playing a pre-generated block sequence more than once since the dates/times will not line up.
  e) As a final fallback for those not willing to take any risk, download the whole chain.

Further Note:  Because there will always be full nodes (or even partial nodes running against the true chain), they can always warn people and miners can manually switch back to the correct chain if someone does try this attack.  Once on the correct chain, they will stay there, so power will not be moved over to fake chains.

Thus, to do this attack, you must give up your mining reward for 1000+ blocks, and at the end you will be caught by full-nodes and your chain abandoned as news of the fake chain spreads across the world with proof from the full nodes, which will be a very big loss for you.  Thus, it is pretty clear that such an attack would not be in your own self-interest.


The pruning block doesn't replace the UTXOs, it only contains copies of them.  They should not be considered separate transactions. 


True, but the amount of transactions people want to make can grow exponentially.  If the limits aren't raised accordingly, exponentially more people will be boxed out, fees will skyrocket, and the Bitcoin adoption rate will decrease.

Confirmations you mean, right? So in this case 6 blocks deep. So what if the miner produces a pruning block for me nodes that has a higher block height than the actual? Then the miner expands on that block? Currently the best blockchain is chosen by having the greatest sum of difficulty, but this breaks that because it cannot be calculated with a partial blockchain. How would the node be able to know that the blockcthat it knows of is incorrect? If it receives the actual blockchain how will it know that is the correct one versus the incorrect one given by a malicious node/miner?

So then what would happen to the original UTXO that is replaced/removed by the pruning block. What if someone wanted to spend from that? Someone who has synced the bickering and then took it offline prior to the pruning block being mined?

The blockchain cannot grow exponentially forever because the block size limit forces a maximum growth rate. It can only grow at 144 Mb per day right now.

If a miner submits a pruning block with modified balances, they will be rejected by the rest of the mining community as an invalid block because the submitted balances do not match what they have, and so it will never make it into the actual blockchain to stay.  Thus, similar to accepting payments you should not accept a parsing block until it is 6 or more transactions old (and more is probably preferable) if you are downloading from scratch.


Yes, the pruning block would essentially include a copy of all UTXOs as of the previous block.  However, any pruning block submitted with extra, incorrect, or missing UTXO's should automatically be rejected as an invalid pruning block, so it would not give miners the ability to spend other people's bitcoin (again those would be rejected by other miners).


I surely hope technology keeps up.  However, if the Blockchain Size continues doubling every year as it has been, it will quickly catch up to Moore's law as that only doubles every 2 years.  Hopefully the economy reaches user-saturation before then at which point blockchain growth will be linear, but if not as you say the block-size limit prevents this from becoming an issue.  The main problem there is that exponentially more people each year will be unable to perform transactions, which is bad for the currency.

I think, you are just too early with this, since they is no urge.
Look at how the scaling or the decentralization issue is treated, or is it an issue....hm.

That doesn't answer the question. If a new client joins and they start syncing from the latest pruning block, how does the node know that the balances in a pruning block are correct? A miner could mine an invalid pruning block  with incorrect balances but then how would a new client know that that pruning block is invalid? How can it determine this without trusting anyone like how currently full nodes do not trust anyone.

Also, bitcoin doesn't work on balances but rather transaction outputs. The way this would work with less complex changes would be to create new UTXOs with the balances. However, this essentially gives miners the ability to spend other people's bitcoin.

Additionally disk space is becoming a lot cheaper and network bandwidth is also expected to increase. We won't be having to manage a full blockchain with current technology, technology will advance and the blockchain should not outpace technology growth as long as there i is a block size limit.

Fair.

Okay - well when (even) one of the devs says that your idea is a good one then I'll admit that I should have not rejected your idea.

Fair?

I reread all your posts here and you have given exactly 0 technical reasons to back up any of the points you have made thus far (go ahead, read your own posts tell me where I'm wrong).

If you think my points are irrelevant, then please stop being a hypocrite and have a TECHNICAL discussion with me, as I'm starting to get the feeling I'm just being trolled here...

So you refuse to say about anything relevant and want to carry on with some silly stuff that no dev has replied to or will.

Good luck!

I have studied it, I see major problems in the future if nothing is done, and that is why I'm posting about the topic to begin with.  Here are the relevant facts:

1) Without trust, you must currently download the entire blockchain in order to get a pruned one.  (Note:  I know a lot more about pruning than this and the rest is not relevant. I'm willing to discuss my knowledge in another thread but here is not the place.)
2) As the blockchain grows in total size and pruning software improves, more nodes will switch from full mode to pruning mode.
3) The full blockchain will only ever grow, and will grow proportionately to the number of users using it.

Points 2 & 3 combined are the most worrisome, as less nodes to deliver more data may mean that 1 will become virtually impossible. 

For example, we currently are at a 60GB blockchain with 500k addresses.  Imagine 500 million addresses.  Since the blockchain can only grow, with those levels we can expect very quickly a 60TB blockchain.  If you have a 10MB/s (that's 80Mbps) internet speed, it will take nearly 69.4 days to download the full chain as you prune it along the way, and that's not even counting the 10,000 additional blocks that have been added along the way.  Furthermore, that assumes the full nodes have the bandwidth to support those speeds and that you were able to keep them up 24/7.  More realistically you'll be competing with 10's of other downloading clients per full node since the number will increase proportionately to the total blockchain download time multiplied by the number of users downloading a partial/full node.  In that case, most if not all will cancel the download part of the way through, meaning no one EVER gets the full chain.

Introducing Pruning Blocks into the blockchain will eliminate this problem entirely, and in the process improve download performance for the full chain because only those users desiring to run a full node will actually need to download it.

If you mean any Bitcoin devs then I'm sorry - none of them have (or likely will) respond to this topic.

Please go and actually find out about the pruning that Bitcoin uses already before proposing something inferior to replace it with.

Try and understand that naive (and sometimes just plain stupid) ideas to "improve Bitcoin" are suggested every day on this forum by people with no software development skills whatsoever (and you have failed to show that you have any).

If the developers were to waste their time having to read and respond to every suggestion such as yours then they would never have time to actually do the work that they are doing (which is why you are not going to get a response from them).

So again - if you really have anything serious to offer then firstly go and study how Bitcoin pruning works before trying to tell others how "it should work" (when you've not even displayed that you can code).

I'm sorry but them saying pruning already exists does not make this idea silly, and I've already responded why.  If you have a reason why my response to that was inadequate then I am all ears.

No - you have a silly idea and it was pointed out to you why that is so - yet you ignore that and still think the "community" should pay attention to you.

Please read more about the actual blockchain technology before "trying to improve it" (you'll notice that no Bitcoin dev has taken the slightest bit of interest in your idea although maybe next you'll say that is some sort of conspiracy).

This is a rough proposal of an idea that I find promising and that I have not heard before.  I am admittedly not an full-time expert in the blockchain, but I am a software engineer by trade and have looked into the technology a great deal.  Although this is my first development contribution to the community I hope that it will be judged based on merit and not by my perceived inexperience.

Here are the merits I see

Merits:
1) Simplify and globalize the pruning process
2) Eliminate the need to download the full blockchain to get an trustworthy & accurate ledger

You provide no technical details and have clearly very little understanding about the technology (from what you have posted).

Yet you think you have come up with some grand idea that others should look at?

No-one is going to waste their time with your ideas unless you actually show that they have the slightest bit of merit (which so far they do not).

Understand how the blockchain works before trying to improve it.

It removes the need to trust a 3rd party and replaces it with the need to trust the blockchain: 

Currently, if you download only a partial chain you must trust a third party for the balance information of the blocks you did not download.

With this, if you download only a partial chain you must trust blockchain technology for the balance information of the blocks you did not download.

Lets assume I have just installed my client. How can I trust the "balances" (I assume you want to store the UTXO set, but details...) when I can not calculate them myself from history data and compare it to the data in the "pruning block"? If I need to do this, someone will have to keep all history data for me to download and verify. Once I verified the data I can discard it. This however is already implemented and its called - you guessed it - pruning.