DeathAndTaxes: I'm normally impressed with your posts, but I think you've got some details wrong.
Thanks and I am happy to be corrected.
First, RE: the orphan cost of transactions:
Decker/Wattenhofer measured 80ms for a 1K bigger block.
...
So 3.3 millies per kilobyte is the orphan cost.
I didn't know about this source. I recall (but can't seem to find a cite) with a much lower estimated cost although it may have simply been an error on my part ~0.4 mBTC vs ~4 mBTC.
It should be fairly easy to get about another factor of about 10-20 reduction in orphan costs.
Correct me if I am wrong but a rather simple block broadcast improvement would be to not include the tx just include their hashes. Since a well connected miner aware of a given tx X should already have relayed that tx X to his peers there is little need to include the full tx in the block message. Instead block message could just consist of the header and a list of tx hashes. If average tx size is 400 bytes and the SHA-256 hash is 32 bytes that alone could cut the orphan cost of a tx by 90% to ~0.3 mBTC. In reality since the tx hash is just being compared to the UXTO a more involved modification would be for the block message to include a truncated hash of the tx. This wouldn't represent a security risk as the actual merkle tree would involve the full SHA-256 hash. Using say the first 64 bits of the SHA-256 hash would still make collisions essentially highly unlikely events and would reduce the orphan cost by another 4x to <0.1 mBTC.
And as I said in another thread, if EVERYBODY produces larger blocks then EVERYBODY bears the increased orphan cost, and the result is better for everybody . There is a fixed number of new bitcoins to be earned, regardless of the orphan rate; everybody's share of that fixed number will be the same if everybody has a slightly higher orphan block rate. But everybody will earn more fees, and their bitcoins will be worth more because bitcoins will be more useful.
Agreed. Hopefully pools and major solo operators can see that their long term profitability is based MORE on the growth of Bitcoin than the short term orphan cost. Hopefully large pools are aware they don't need universal support. If 70% of hashpower agrees to produce larger blocks (say 500KB avg) for the good of the network. It cuts the orphan cost by 70%.
Still I think the orphan cost does highlight the reality that miners are going to be increasingly reluctant to devote more space to free tx. This is something users will have to come to grips with. Including a fee and waiting hours or days is unacceptable (although it is often for non-fee reasons) however no fee tx should be considered charity by users and if someone does you an act of charity in an hour, or day, or week well you got what you paid for. The default behavior of most clients should probably be changed to include the "min fee" on ALL txs not just low priority ones. If users want to they could change this but they should be warned "Including no tx fee may result in delayed confirmation times". Enforcement for relaying at node level should still only be on low priority.
It is somewhat ironic that this is more of an issue due to the higher exchange rate. The min fee on low priority tx was lowered due to rising exchange rate. Today 3.3 mBTC is ~$1.50. Ouch. However if Bitcoins were worth less it would be less of a cost. Since Bitcoin is often used as a proxy for USD a 5 mBTC fee (which more than covers orphan costs) is more viable at a lower exchange rate.