Topic: Can't we avoid reorgs once and for all? - page 2. (Read 260 times)

You're all correct. I'm still thinking though: isn't there a way to avoid reorgs without incentivizing miners to be selfish? Couldn't there be a rule (that perhaps makes system more subjective in terms of consensus) which would make nodes reject those blocks, and decide objectively between 2 blocks (without waiting for the next one to be built on top)?

My thought was: low block intervals with zero cost. The cost of reorgs isn't high in bitcoin, because of 10 minutes block interval, but if you were to drop it to 1 minute interval, you'd notice lots of orphan blocks.

Isn't a soft fork enough? I'm thinking of it as limiting the protocol rules further. Not violating the old rules.

It would not be a case of an attacker still working on 700,001 when the rest of the network has moved on to 700,002, but rather an attacker already mining 700,001 and keeping it secret.

To build on the selfish mining idea in achow101's third paragraph above, imagine a miner finds block 700,001, and it has an exceptionally low hash, which would be enough to overcome two or even three blocks which have an "average" hash. This miner also includes, in every block they mine, a private transaction moving a large amount of bitcoin from one address they own to another address they own, which was never broadcast to the wider network. The miner decides not to broadcast their block 700,001, but instead keep it secret and instead broadcast a transaction which spends the aforementioned large amount of bitcoin with a merchant, exchange, whatever. They let that transaction confirm in block 700,001 on the main chain, get 2 or maybe even 3 confirmations, and then successfully reverse it by broadcasting their replacement block 700,001, which has a sufficiently low hash to reverse three existing blocks.

There is nothing to stop every miner from including such a self transfer in every block they mine, and any time someone is lucky enough to find a very low hash they can keep their block secret and attempt a large double spend.

This idea isn't new and this same question has been asked by a lot of people over the years. Conceptually, it doesn't make sense. It would also incentivize selfish mining which could make reorgs more frequent, not less.

Conceptually, your idea doesn't actually measure the amount of work that went into the chain. The chainwork is a measure of how many hashes, on average, were required to create the chain. Because it is tied to the target, each block using the same target adds the same amount of work to the chain, regardless of the actual value of the hash. If this were changed to be based on the actual hash, the chainwork no longer represents the actual work done. A miner that gets lucky and finds a hash that is less than the target did not actually perform the number of hashes that would be needed to find a hash of similar value. They did not perform those hashes, so it would be misleading to include the apparent work of that block. Of course a miner could still mine a block less than the target with fewer than the expected number of hashes. But because all of the blocks with the same target add the same amount of work, with a sufficiently large number of blocks, it all averages out. Conveniently, the difficulty period ensures that there will be a large number of blocks at a given target for that entire period to achieve that average. This means that over the entire blockchain, the chainwork calculated will be a close estimate to the actual number of hashes that were required to produce that chain. If it used the hash's apparent work, then the chainwork would be an overestimate of the amount of work that was done.

This idea also further incentivizes selfish mining where a miner doesn't broadcast their block and instead extends it in private and waiting to broadcast it later. If a miner gets lucky and finds a block with a significantly lower hash, they could not broadcast the block and mine on it in private. Then they could broadcast it when the public chain's work gets close to the work of their private chain thereby reorging the chain and getting more rewards for themselves. This is easier to do when using the apparent work because a miner just needs to get lucky once, whereas with the actual chainwork system a miner needs to get lucky multiple times and produce multiple blocks faster than the rest of the network in order to perform such an attack. Depending on the thresholds miners choose for doing this kind of thing, it could actually make reorgs worse.

If you want to be convinced, which system is better, you can write some simple tests. For example, you can use a single SHA-256 on some numbers, to get sample hashes, just to have some representation of hashes submitted by miners.
Then, you can start from the simplest test: finding the strongest hash. You can run your code for a while, and see, what will happen, when you will accept the new hash only if it would be lower than the lowest one.
You can run such code for some time if you want to estimate your hashrate. Then, you can use a division to calculate the chainwork.
Then, you can calculate chainwork for each hash you need:
Next, you need some way to create a history. You can for example use two hashes: one for the previous hash, and another for the hash of the current number you just computed. Then you will have any linked chain you want.
And then you can compute the total chainwork for any chain. For example, if you assume that all numbers are hashed and chained in order, you will get this:
At this point, you can trigger any rules related to chain reorganization. If there is only one miner, then there is nothing to compare with. So, you can assume that there are N miners present in the network, and each miner generates its own hashes. First, let's assume that each miner has similar computing power. You can just generate numbers as before, and take them modulo N, then the remainder equal to zero means the first miner, the remainder equal to one is for the second miner, and so on.

For two miners, it would mean the first miner will mine SHA-256("0"), and the second miner will mine SHA-256("1"), and each miner will try to extend the chain. First, you can assume that the previous block hash is zero, then calculate all hashes for all miners (here: two), and keep only the lowest hash. This is what would happen:
However, to complete the whole picture, you need more than that. You need to compute the total chainwork, because it is not only about picking the best hash from one of the N miners. So, you start from the zero hash, and the zero chainwork, and always pick the hash that results in the highest total chainwork after adding such block to the chain.
Here, things start to get interesting, because the total chainwork seems to be the same for both miners. In the current system, we would keep both branches, and wait for solving that by next blocks. However, in your proposal, the lower hash will be always picked in that case. In your version, it will be solved in this way:
After writing more code, we can see, what would happen on the alternative branch:
As you can see, after mining 16 blocks, you have chainwork equal to 60 on one branch, and 84 on another branch. After testing longer chains and different scenarios, where there would be some attacker, trying to endlessly mine the Genesis Block (or the earliest possible block after that), you would notice more unexpectedly low hashes. Because at first, it seems that picking the lowest hash is an obvious solution. But when you write more tests, then you will notice more attacks, and there is no reason to introduce them to the current system.

For example, even here, on some smaller numbers, you can see that it is possible to go from 25 to 84 chainwork. Every sometimes, some miner may hit such block, that will be the lowest block hash ever found. Statistically, it is inevitable, that if many miners are trying, then one of them may hit it, just because of how many hashes were checked by all miners.

Also, it is a good moment to open the whitepaper again, and read chapter 11 named "calculations". Because then, you can compare your attacking scenarios with those probabilities described in the whitepaper, and see, if the probability matches with reorgs you will see during your testing.

Probably mostly theoretical question; I'd agree, it's not something necessary right now or in the foreseeable future.

Please don't derail the topic; look at https://github.com/bitcoin/bitcoin/blob/master/doc/bips.md and complain somewhere else.

Only 1 coin network has no reorgs.
That coin is algorand , 4 second block speed and transaction finality.
You might want to study how they did it and if that design can be ported over.

FYI:  https://www.algorand.com/technology/immediate-transaction-finality


But be aware, btc has done no real improvements to its onchain network for years,
doubtful any would get thru now.

 

But why? Unless I am missing something the system works the way it is. Doing this would require a hard fork for something that is not that frequent an occurrence.
Having an occasional reorg / orphan / lost block is just part of mining.

-Dave

So the counter argument is: if the block hash of the last block is too close to the target, the miners become incentivized to reverse it (since it's easier), especially if it contains a lot of fees. But is it true, though?

The current system works like this: to reverse a block, find two hashes below the target. My proposed system works like this: to reverse a block, find one hash below the previous hash (which is below the target); unless of course it's difficulty adjustment between the blocks. In that case, the new block hash should be greater than just the target.

Indeed, good point. Reversing the previous block requires mining one more block in the current system.

So you're asking: what happens if Miner C mines block 700,001 and 700,002 whose total work would be more than Miner A's 700,001 and 700,002 combined? They'd be reversed. Quite unusual though to mine 700,001 when we're already in 700,002, unless you want to attack the network.

Yes. Good point.

No, it is not, I also thought about solving it in that way, and my conclusion is that the current chainwork is better. For example, you can look at the lowest block hashes in the history, for example block 634842 with hash 000000000000000000000003681c2df35533c9578fb6aace040b0dfe0d446413. Sometimes, a miner can accidentally find a very low block hash, lower than expected. Then, in the current system, such miner will have no advantage over other participants. However, in your system, it will suddenly gain a lot of chainwork, sometimes enough to reorganize many blocks in a row.

Also, that last thing seems to be the most dangerous. For example, the total chainwork could be based only on block hashes. What then? It turns out that in such case, it is possible to mine the first block after the Genesis Block, and overwrite a lot of early blocks in that way. And in this way, you don't have to start from the lowest difficulty, and go through all difficulty adjustments, to build some stronger chain. All you need is constantly trying to mine a single block, and betting that it will be stronger than other blocks.

So, in theory, you can calculate chainwork in a different way. But in practice, some miners will hit some lower blocks by accident, and then they will have an unfair advantage, because it can turn out, that their blocks will be stronger than N blocks in a row.

Just to be sure because I might this wrong, but in this case what will happen if
- Miners A and B mine block 700,002 (one of them does)
- Miner C mines his own blocks 700,001 and 02 with 01 having a lower hash than the previous block mined by miner B?

There has actually been a few questions about this on Stackexchange; most notably, an answer from our very own theymos!


The initial question in that post also proposes a similar solution like you; theymos says about that:

There's some discussion about it and myself, I'm not sure whether it would make sense to start handling reorgs like that.
Personally, I mostly resonate with this statement by theymos; it's not a big issue for Bitcoin in the first place so I wouldn't worry too much about it.

This is my understanding of reorgs. Please correct me if I'm somewhere wrong.

• Assume we're at block height 700,000.
• Miner A mines block 700,001.
• Miner B mines block 700,001.
• Both miners broadcast their success at the same time.
• Some nodes receive coinbase transaction A, and some coinbase transaction B (let's call one group A, and the other B).
• Miners from group A mine on top of coinbase transaction A, and miners from group B mine on top of coinbase transaction B.
• Group A mines the candidate block successfully, it adds it on top of 700,001 and shares it with everyone.
• Now block height is 700,002, and the coinbase transaction from block 700,001 is A, meaning that both miner B and miners from group B wasted their computational power.

The correct chain is the one with the most work. The group's A chain is correct, because their chainwork is greater. As far as I can tell, chainwork is calculated in amounts of blocks. If difficulty is 1, and you add a block on top of the chain, 0x100010001 of chainwork is added.

Now my question is: isn't a block hash based chainwork more accurate than a block amount based one? If, instead of difficulty, we used block hash as a meter for chainwork, there would be no reorgs, because either miner A or miner B would find a hash with more work than the other.

Example:

• Assume we're at block height 700,000.
• Miner A mines block 700,001: 0x00000000000000000002f39baabb00ffeb47dbdb425d5077baa62c47482b7e92
• Miner B mines block 700,001: 0x000000000000000000010e76862af418f16ddb538f6f03ef7a7052b751d79b829 (hash is lower than A's, and therefore B has probably worked more)
• Both miners broadcast their success at the same time.
• All nodes will discard Miner A's block, because Miner B's block's hash is lower than A's.
• Everyone works on top of Miner B's block.

Doesn't that drop the cost of reorgs to 0?