I meant pass the rest to OP_TRUE. This is an output that anyone is allowed to spend. It is effectively a gift to the miner of the next block, since why would the miners include someone else's transaction spending it, when they can spend it themselves.
If the average fee per block was around 1BTC, but you include a transaction with 10BTC fee, then your coinbase gets 11BTC in fees.
If you assign the entire 11BTC to yourself, then other miners might reject your block and create an alternative block.
Another miner might create a block with 11BTC in fees, but pay 3 BTC to himself and 8 BTC to OP_TRUE.
This means that anyone can spend the 8 BTC.
The other miners have a choice, they can build on your block and just get the average of 1BTC or they can build on the other block and get part of the 8 BTC paid to OP_TRUE.
If they decide to build on the 8 BTC OP_TRUE block they have to decide how much to take. They might decide to take only 1BTC, so they get 1BTC in fees plus 8 BTC from the OP_TRUE output and then pay 7 BTC to OP_TRUE.
Some could also create a third block since they think that 2BTC out of the 10BTC was to much. Knowing how other miners will react is key to figuring out how much you should take of the payment to OP_TRUE.
Thanks for the details. My blind spot was in not realising that a miner could process a transaction but then pass its fee onto the next miner. I assumed that a miner would only leave value in the mempool by abstaining from processing some transactions. Danny also caught me on this.
I appreciate that the problem is difficult but would appreciate any links you might have to attempts to take this further.
If your example strategy:
I'm not convinced this strategy would be stable because it seems to rely on a widespread tie-breaker of "favor blocks which follow this rule" just as we currently rely on the widespread tie-breaker "favour blocks first seen". If the more profitable tie-breaker of "favour blocks which leave more on the table" becomes widespread then miners would have occasion to profit by undermining blocks built by small miners (basically no risk) while respecting similar blocks built by large miners. I'm sure we'd settle into some equilibrium but perhaps that equilibrium will only have one miner.