Greetings Eligius miners!
So, let me go over a few things.
First, I've caught up the payout queue completely with
txn 3b529c9505fedbe31323f98963ae7bb38b88bb66b95bee7a1c8ce2ad2c58ae86.
Pool has been doing pretty well! CPPSRB has been serving everyone quite well. 100% PPS has been paid for all work for over the past month!
Unfortunately, there was a minor mishap last evening. In the interest of continued openness with the community, and to make a long story short, last night the pool reward system code did not pick up on a block the pool found, 245236. So, the reward system did not calculate anything based on the payouts in that block, and then block 245239 was found without it having knowledge of 245236, thus mostly doubling up on the payouts in the block. (See
245236 and
245239.)
Quick summary: The pool overpaid some miners. Fortunately, over 90% of the overpayment was recovered. The remaining ~2.3 BTC I have covered out of pocket. The error causing this has been corrected. Overall, there is no cause for panic and business continues as usual.
Here are the details.
The cause was an extremely rare race condition between the pool server->database->reward system made possible by our recent growth and the need for multiple eloipool instances. Code has been put in place to prevent this from happening in the future, as well as a few other potential problem conditions. (Technical details: Eligius uses a postgresql back end. One table is used to store every incoming share in a row, using a automatically incremented sequence number. Eligius is running 15+ instances of the eloipool open source pool software, all of which have a connection to the database and are inserting rows in a BEGIN INSERT COMMIT style. We average somewhere around 1000 inserts per second. As it turns out, there can be an initial collision of sequence numbers. This is always corrected. However, there is a rare case where two inserts try to use the same sequence number, they both fail because of it, and get issued new ones, skipping the original. This causes a brief delay as everything recovers from the collision, and the original insert gets processed several hundred rows after it was supposed to. In our case, the reward system had already read back the chunk of rows where the block-winning-share was eventually inserted, but before it was completed, resulting in data returned omitting this particular row in the eyes of the real time reward system. I could find very little documentation on this "feature", but, it is expected that there will be
gaps in postgres serial sequence numbers For the correction, the reward system code now checks to make sure that any blocks that contain Eligius payouts that appear on the network are flagged and verified as being reconciled with the reward system. It also attempts to wait on any gaps in the postgresql sequence numbers and rechecks data near any missing ones. The combination of the two fixes will prevent this situation in the future. As an added bonus, the pool software now provides it's current best previous block hash to the reward system and the payouts are verified to match before being valid.).
The reward system missing the block/payout resulted in an over payment of most of the miners included in block 245236's payouts. When this was noticed, I shutdown the reward system to investigate. With the payout system disabled, this allowed time for some of the miners who were included in this over payment to continue to mine and slowly correct the over payment. Luckily, we found five blocks in that time frame.
As of the time the reward system was brought back online and all was reconciled, the total over payment amount came to 6.995 TBC (2.30075557 BTC). I have personally absorbed this final over payment amount out of my own funds. Last night, however, I had braced for the worst (all miners affected leaving the pool and not offsetting the over payment) and semi-expected a full 25 BTC shortage, which I would have needed to pay to cover miner payments.
Here is the final remaining list of miners that were overpaid. Some portion of the over payment was able to be applied to these miners' shelved shares so that the over payment did not result in a huge over payment with regard to work-completed. However, the payments of those shelved shares is not part of the normal block reward and thus is included in the total over payment amount above.
0.01373157 BTC - 1DkXMqzxJ4mrZhT25ET48wzxLK7Z5dLDTC
0.02435758 BTC - 1PzV9kHvB2vU3np2Ei1YxvPAS8HZppJ5UX (0.02435758 BTC applied to shelved)
0.04976031 BTC - 15oQaoMxK9d6DGVFSSnJUYEjPsd77X1KPB (0.04976031 BTC applied to shelved)
0.07145660 BTC - 12EDsoSowtFgcZPAdhB74AwPkcu9mhy37P (0.07145660 BTC applied to shelved)
0.09235259 BTC - 1Duder7vFJVmvq3SrixDCY1knShZ7LEF8R (0.00980343 BTC applied to shelved)
0.09556698 BTC - 19d6exSEyAPDXLLe3W3ucexpVXUABFeFsy (0.09556698 BTC applied to shelved)
0.10163246 BTC - 12tKVT4KupVY2sVzSX4jnBWC9ZjzFHbtQg (0.01093716 BTC applied to shelved)
0.11408797 BTC - 1CLr2aR8DrjRxSgQbhim3BNeeZHY2Ug26B (0.11408797 BTC applied to shelved)
0.12780791 BTC - 1ALkgvSRuAGuVtyUVGfncuRxmHa7gUVQQt (0.12337554 BTC applied to shelved)
0.12923071 BTC - 1Mcc4H1MRVWxGNGSZ7RiYsKFboeV2jU2Kt (0.00711126 BTC applied to shelved)
0.15235391 BTC - 1ELFX9sovv4LbsmEV3ivbVZS7s5Dr5Sj8f (0.00241371 BTC applied to shelved)
0.15360348 BTC - 1Q8eAfxs5fuTWJE2hLoyYspzU5TrRWQeFf (0.01646416 BTC applied to shelved)
0.50090432 BTC - 1Foh62T2GcMNj9wh3nqc7bSD1EhsQ5BKpe
0.67390918 BTC - 1Q56RJLDotSBGD2386juC4ffefxG3Zx3KV
Total Overpaid: 2.30075557 BTC
If any of these miners are kind enough to come forward and return any portion of this over payment, please do so, and it would be greatly appreciated. I There are no negative balances on Eligius as a result of this. If anyone is kind enough to return any of these over payments, or is otherwise willing to donate to the cause, again, it would be greatly appreciated and you may send said funds here:
1oopsyR7UB3kA1cr6ka7bDLnYbFRBnHmN (yes, oopsy...)
I apologize to everyone for this issue. It is fortunate that the majority of the miners at Eligius are honest and continued to mine even after the over payment, and I thank you all for that. To those who did benefit from this mistake I do welcome you back and will not penalize you if you decide to continue mining at Eligius. While the mistake was obvious, and I would hope that no one would intentionally take advantage of the pool, it also was not the fault of any miner, so, I can not hold anyone else accountable for the error besides myself. It is also fortunate that the error was caught quickly. It is also very fortunate that the form of Eligius's payouts (payments directly from the coinbase) limits the potential impact of these over payments to at most, one block reward, where other reward systems at other pools can (and have) lost hundreds of BTC to errors.
To summarize again: Business as usual now. I have fronted the coins for the over-payment out of my own funds (not the pool's). The error was the result of a rare race condition that was not taken into account in the reward system code. The code has been corrected. There was no compromise of any pool systems in any way. No miners were negatively affected by this.
Thank you all for your continued support of the pool! I think there are great times ahead for us!
-wk