Edit:I should acknowledge
https://bitcointalksearch.org/topic/suggestion-introduce-penalty-for-attempted-double-spend-3168 where jav advocates something amounting to shunning.
An effective way to discourage double spending attempts would be to propagate both attempted transactions and then "shun" the inputs.
"Shunning" is where the network mutually agrees that it will never accept as valid any transactions or blocks which contain a shunned address or shunned TxOut. If shunning is not unanimous then any transactions by non-participants, which transfer money from a shunned address or use a shunned TxOut as an input to a transaction, are themselves shunned.
In the shunning community, the value of these shunned transactions or addresses can be redistributed among themselves as they wish or (preferably) considered forever gone in an even more irrevocable fashion than if the private key had merely been lost.
If a merchant has accepted a transaction and has handed over the goods before a transaction has been incorporated into a block then it's cold comfort that the double spender's money is shunned after the double spend is detected. The merchant is still owed payment for the goods. The solution is as follows:
Firstly, the merchant should require that the transaction paying him for the goods should be from a TxOut or an address that holds many more bitcoins than the value of the transaction. That is, the "change" portion of the transaction that returns the "unspent" portion of the TxOuts referenced back to the payer should be much larger than the value transferred to the payee. If the payer attempts a double spend then the payer runs the risk that all the bitcoins will be shunned even though the value of the attempted fraud is much lower. This scheme has the advantage that it requires little change to the existing infrastructure.
Optionally, in the event of a double spend, the network could elect to honour the payment part of the transaction and shun the "change". This means that the merchant is not out of pocket while the fraudster is punished. In order to signal which is payment and which is "change", the payer structures the transaction such that the "change" is paid to an address that was also used to sign the crediting transaction. Care needs to be taken with such schemes to make sure that double spending remains uneconomical when the double spender acts in undetectable collusion with one or more of the double-spent merchants.
A supplemental scheme would give the sender feedback about the propagation of the transaction across the network but it involves much more infrastructure change and network traffic. We observe that a minority of miners generate the majority of blocks. If the merchant can see that the transaction has been accepted by these miners then he is confident it will not be double spent. The proof of a miner's power is the number of blocks it generates in a given time. If the generated blocks always credited the same address for a particular miner then we would know who the biggest miners were (at the cost of a big loss of pseudonymity for the miner). The merchant would require that the payer construct a transaction which also requires some proportion of the largest miners to sign that they have received the transaction and will include it in their next block, in return for some fee. When the merchant sees the miner's receipts arrive across the network then he is comfortable handing over the goods.
Note that the last supplemental scheme is tentatively included merely to spark off ideas as it is ugly. I would also warn against shunning as, although simple and powerful it easily results in fragmentation and I feel that the schisms it facilitates could weaken or even destroy Bitcoin.
ByteCoin