Unconfirmed transactions
We experienced some delayed transaction confirmations over the past week.
I just wanted to take a moment to update you on the status of this issue and apologize for any trouble it may have caused you. Our goal is to have all bitcoin transactions arrive within a few seconds (about the speed of an email) and to be confirmed within the hour. In this case we fell far short of that goal (some transactions were delayed for a number of days), and for that I am truly sorry. I know it caused undue hassle and concern for our customers, and that is something we never want to happen.
The majority of these delays should now be resolved. I’d like to address the solution we now have in place and some additional notes on how we think about transaction confirmations.
The root of the issue stemmed from how bitcoin transactions tend to chain together. If you have a group of 20 bitcoins and you want to send 1, your transaction actually uses all 20 bitcoins and sends you 19 bitcoins as “change”. This is much the same as using a $20 bill and getting $19 change for your purchase.
Your 19 bitcoins can then be used for the next transaction, and the change for that for the next, and so on. This creates a chain of transactions where you are spending the change from the previous.
One challenge of the bitcoin protocol is that transactions can often take 10 minutes or longer to be “confirmed” (or make it into a block). So if you are rapidly sending transactions you may end up using coins that haven’t been confirmed. And if one of those transactions happens to never confirm (or takes a long time to do so) it hold up every other transaction in the chain.
Coinbase has a number of processes to counteract this, including using a hot wallet, breaking outputs into smaller chunks (sending a hundred 1 BTC payments from a 100 change will create a longer chain than sending a hundred 1 BTC payments from ten groups of 10 BTC), and paying miners fees on our customer’s behalf to make sure transactions are mined quickly. But as we continue to scale and move beyond several thousand transactions per day, our basic systems were not cutting it any longer and there were a few edges we weren’t accounting for.
In particular, what caused the issues this week was getting transactions into the hot wallet which very long confirmation times. Luke Jr and the the community were very helpful in assisting us in getting these mined, and a big thank you goes out to them. We also re-generated/broadcast a number of transactions to get them confirmed.
We have now rewritten how our coin selection works, and pushed out a big code update today. In particular, confirmed coins will be required in several cases, including:
when sending to the hot wallet
well selling coins through Coinbase
In these situations the number of blocks (or confirmations) that are required varies with the amount of the output.
There are other transactions (for example those coming out of the hot wallet) where we require fewer confirmations, or in some cases a maximum “depth” to the chain of 3 transactions. If no coins are in the user wallet or hot wallet that meet this criteria, then you may be required to slow down generating new transactions for a period of time. We will automatically try regenerating pending transactions at increasing intervals until there are sufficient confirmed coins to complete the transaction.
There is a trade off to requiring confirmed coins for every transaction. The pros are that it assures all new transactions are confirmed quickly, and unconfirmed chains can’t build up. The cons are that it requires the user to wait if there are currently no confirmed coins available. As Coinbase continues to go, we will keep working on improving this situation and work to make sure it has the right balance between confirmation times and sending rate. There is a lot more we could be doing around instant confirmations down the road as well.
Hopefully this helps explain the issue we faced and let’s you know that we are taking it seriously. As always, we are open to your feedback in the support forum.
Sincerely,
Brian Armstrong
CEO, Coinbase