Interesting attack indeed but h4xx0r who did you quote with the idea of giving to the next miner a share of your coinbase tx? It's trivial to give to the next miner outside the coinbase transaction by sending the reward as a transaction using one of the current inputs. Sure, then you have to scrape the bitcoins from elsewhere.
To recap the attack in laymen's terms:
If somebody paid 10,000BTC in transaction fees, miners would not care about block rewards for the next 10,000/25=400 blocks. Any miner that thinks it could outrun the biggest other miner would try to do so. If there is a draw between the top miners, such a battle could take a long time. If the top miners hold 10% of the mining power, they might try even when the other had a head start and was slowly building a chain that's growing faster than the own chain as they could still call their friends of other pools to team up catch that guy, essentially to the point where all miners took one side or the other and weaker group gives up.
During such an episode, massive re-orgs would happen, clients would act strangely, Finney attacks would be slightly easier etc. and we would have a slightly higher level of drama. And we will never know who sponsored that drama
but it would not be a cheap endeavor.
It's an interesting math problem.
If you have 51% of the global hash power, then it is generally in your favor financially to mine only on top of your own blocks, ignoring all other blocks.
As your hash power drops below 51% it begins to become beneficial to build on to of the most recent block, due to the cost associated with the orphan risk. However, with 49% of the hash power, it would probably be beneficial ignore any block that has a larger than average block reward.
The lower your percentage of the hash power, the higher value a block reward must be to overcome the cost associated with the orphan risk.
Assuming for the sake of calculation that all solved blocks are instantaneously known by all miners, and that there are never any splits caused by two or more miners simultaneously solving the current block...
Is there a formula that can indicate for every percentage of global hash power how much higher than the average block reward would be necessary to result in a positive expectation for ignoring the most recently solved block and continuing to attempt to mine it one's self?
Anything over 50% requires a no increase above average reward to result in positive expectation.
As you approach 0% the required increase above average reward approaches infinity.
What does the curve look like between those two extremes?
The assumption of selfish mining and total transparency of mining power among all the competition is essential here.
If I have 1/n-th of the network, just like all the competing, evil miners I would find every n-th block, gaining every n-th block reward if we play nicely. If we don't I might not get it in the end, so I have an incentive to help punishing others that don't play nicely, but without crazy fees, there is little incentive to build on top of one block rather than another, so it works.
With a big treasure trove setup for the winner, those evil miners can try and if all factors are known, you can't win unless you have 51%.
Imagine you have 49% and set out to try this. The others know they have 51% united but less than 49% each for themselves. They would go with the first block regardless where it came from as they don't go for cheater mode. They would further assume that any fork mining the treasure trove to another address to originate from the only candidate that might try foul play and would stick to the old chain, knowing that they can win.
Unfortunately in the real world they can't know if some other miner supports Mr. 49% just this one time, so they might be in for some long orphan chain, so I doubt there is a formula as the biggest selfish miner always wins against selfish miners, while non-selfish miners are meant to protect the network.
I guess if this ever would be a problem once, in the future miners would just put a bait into the tx pool in form of some 20% of the received fees and nobody would attempt to pull this off.
We could just as well discuss the problem of somebody paying a pool for not mining as it would have the same slowing effects as this attack if the economics are right.