I was surprised by the latest announcement of Pieter that a SegWit implementation which changes pretty much everything in bitcoin can be implemented via a soft fork, where it does not require all the nodes to upgrade to be compatible
After a bit research, this is possible because you can always give old data new meaning in a new implementation, while the old nodes simply do not know how to parse that data
In principle, with this method, you can move all the transaction data out of the block, not only signature data, so that old nodes always see empty blocks
As a result, a new block will be accepted by the old nodes because they appear to be valid blocks, but in the new implementation, they are just a small subset of the new data structure, all the transaction data is in another related block
More importantly, after further analysis, it shows that although normal nodes can still run old client, all the mining nodes will have to upgrade, otherwise the blocks mined by them will simply be orphaned, because the new nodes do not accept old blocks, and new nodes have majority of hash power
As a result, such a soft fork is to force 100% of the mining nodes move to the new implementation so that there is no slightest chance of the old mining nodes forking into their own chain: Their fork will always be orphaned since they accept the longest chain
The same happens with hard forks as well. Deployment requires consensus.
What this means in practical?
If one type of nodes, Classic for example, controls over 51% of hash power, they can implement whatever change they like by simply use this soft fork trick: Let all the old Core nodes accept their new format blocks while Classic nodes will reject Core blocks, so that Core blocks all get orphaned due to less hash power
Again, deployment requires consensus, as it does with hard forks.
The most deadly part is that Classic nodes do not need to ask for permission of Core nodes, they simply enforce it by orphaning blocks mined by those Core nodes. And Core nodes have no way to fork into another chain by rejecting Classic blocks, since they can not tell which block is Core block and which block is Classic block
Anyone can do that now. You can orphan blocks mined by other miners now, it makes no difference. Again, consensus. Same thing happens with hard forks.
As a result, a miner running core will simply lose money, he either trash his miners and quit the game, or bite the bullet and upgrade to Classic so that his hash power can still mine him some coins. I guess majority of the miners will upgrade. And they might start another round of similar soft fork to regain control when they accumulated more than 51% of hash power
They will probably upgrade. As I said before, and I will say it again, consensus.
This is very bad, since it totally removed the freedom of choice for those old mining nodes, you can call it a forceful take over of bitcoin network rules by 51% hash power. It totally break the major consensus rule and the spirit of bitcoin, so that with only 51% of hash power, you can implement whatever you want and force it upon rest of the miners and their hash power will be captured
So really what is the difference in the forking between a hard fork and a soft fork? The Core nodes can proceed as they did because their chain is orphaned. Any blocks produced by the core nodes are ignored by the Classic nodes. The core nodes, if they are able to build extend their own chain, or at least build a block on top of a core block, results in their chain completely separating from the classic chain since they have a different block history. It splits, much like a hard fork.
Sure, Core nodes can change to another PoW fork, but since the value of the coin is always roughly the same as the mining cost due to arbitraging, that coin will not get more value than litecoin and will be forgotten quickly
Unfortunately, it is almost impossible to prevent such thing from happening by current design, so it is very important in mining community to widely spread this information so that miners are fully aware of the deadly effect of a soft fork, and reject such attempt as much as possible. At mean time, trying to find a solution to fix this security vulnerability
And how would you propose to fix this? It isn't a security vulnerability, it is an issue with forking that is not able to be resolved due to the nature of forking. It isn't really any worse than hard forks are, but with the backwards compatible nature of soft forks, not all nodes are required to upgrade, only mining nodes. Hard forks require that all nodes upgrade.
Again, the way that both hard forks and soft forks are deployed are with consensus. Neither one is deployed unless a supermajority of the blocks indicate support for the fork. The deployment is the same with both forks. I don't see how a soft fork is worse than a hard fork.
tl;dr All forks are dangerous and all forks must be deployed with consensus otherwise problems happen. Applies for both hard and soft forks.