What you want is a rule that is a Nash equilibrium. It is in the best interests of a miner to stick to their current strategy as long as nobody else changes theirs.
Well... What if there is some sort of collusion among majority of miners (i.e. they have more than 50% of hash power combined): they will agree to decide block to mine on top of using some set of rules and some sort of a practical Byzantine fault tolerant algorithm for synchronization?
A Nash equilibrium doesn't require any collusion at all. It is something that just happens.
For example, lets say you play a game. No communication is allowed between players and their moves are secret.
Each player picks a number from 1 to 10. The results are announced and the players that picked the most popular number (say 7) are given $5.
The game is repeated, what number do you think will win?
Next, the referee says "For the next round, I strongly recommend 4. You can still vote for any of the 10, but really, 4 is a great number". Which one do you think would win? 7 might still win, but 4 has a pretty good chance.
They can simply drop blocks of miners which do not agree to participate in this PBFT-synced collusion, and as far as I can tell it is stable as long as colluding miners have a majority. (I'm not quite sure about that, but I guess it will be 'stable enough' for practical purposes.)
You don't need a majority. Say there is one pool which has 10% of the power and they say that they will queue all blocks that have the 2nd (or later) of a double spent transaction in them for 5 minuted before building on them.
Other miners now know that if they add those txs into their blocks, only 90% of the hashing power of the network will build on them for the first 5 minutes.
Within this collusion pretty much arbitrary rules can be enforced.
There are limits before the Nash equilibrium breaks down.
Simple Nash equilibrium are the best. That is why build on the highest POW chain is so strong.
However, as I show in my previous post, with large fees, that Nash equilibrium can break down.