There definitely needs to be new protocols in place. Once is an accident. Twice gets suspicious.
As with everything else, I am open to suggestions.
The issue as I see it is that I need a manual step in the processing of large withdrawals, so I can eyeball things before large amounts of coins are sent out. I need to decouple the wallet which holds the large amounts from the live system, or it may as well be on the live system.
Currently I've been doing a 4-step process:
1) query balance on site
2) if enough, debit balance on site
3) query balance on site again the make sure user didn't somehow reduce balance during step 2 leaving negative balance
4) send coins
When I made the mistake with celeste's withdrawal I missed steps 2 and 3.
I will automate the 4 steps with a simple script which runs locally and queries the remote server, and only use that script for cashing out large withdrawals.
There's nothing that can physically stop me from typing "bitcoind sendtoaddress 1abc... 1300" without doing steps 1 through 3 first, but if I have a script that does the 4 steps for me, I won't feel the need to be typing bitcoind commands like that.