Meh, had an account handy so here it is...
March 11th Payout Errors Detail
This was posted on the official forum thread at Bitcointalk previously. Those who have already heard about it, this is just a summary/recap, not a repeat event.
On March 11th, an upgrade was being done to move the Merged Mining server to bitcoind 0.8. Previously, upgrading the bitcoind software on a pool server was safe to do live. Simply stopping the old binary, copying over the new one, and restarting it, would keep the pool up and running, and prevent miners from disconnecting.
These updates were done live in the past because the ~30 seconds of "downtime" (a block can't be solved during this brief downtime) was worth the cost of miners disconnecting and possibly being stuck offline/on a backup pool. However, something went very wrong with the upgrade to bitcoind 0.8. The main reason to upgrade to 0.8 was the drastically improved speed of the bitcoin client, which reduces stales and reduced the chance of orphan blocks. This was accomplished with a new database backend, LevelDB. Unfortunately, when upgrading to 0.8, it does not automatically update your old blockchain information. It resync's the blockchain from scratch.
The problem on March 11th was this brief period where the server was resync'ing the blockchain. It was not caught instantly, it took about 45 seconds to notice. During this time, the pool server recorded work at the network difficulty from very early blocks, a difficulty of 1. Users were being awarded 25 BTC per share.
These increased rewards being logged were not caught immediately, and a total of 16 users were able to withdraw 1,254 BTC (the balance of the hot wallet) that they should not have earned. All rewards from this period have since been revoked, but only one of the 16 users has returned any funds (a total of 47 BTC).
This news post was made to keep the incident transparent. The pool suffered approximately 1,400 BTC in losses this week. 1,254 from the error in payouts, and somewhere between 150 and 250 coins from the hard fork event. My plan is to simply absorb these losses, and continue moving forward.
The pool is accepting donations from users as a result of these events. Miners can direct a spare worker or load balance backup to the worker: btcguild_donate . Direct donations are also accepted through the address: 1GuiLdvH3YjJyEExyTmcd7X2nwX7pZvV7a
March 15th, 2013
