Hi all,
The couple of stats fail-safes over the past few days (none of which lasted very long) were due to orphaned blocks which required human double checking (long period of time between mining and actual network-side reorg of the block). One of several fail-safes that I'm working on writing code to eliminate the full need for a human double check.
As everyone knows by now, the pool continues to work as normal and your earnings are tracked as normal when these happen and even when the entire webserver is completely offline. Any blocks found while in fail-safe mode generally do not have automated payouts however, because the system is in a state where it is unsure if those payouts would be correct until I give it the go ahead.
Its not a hardware failure or a hack or anything, its just a security measure put in place to make sure that the pool doesn't double-pay anyone. It is much preferred that coins end up in the Eligius cold wallet then get manually paid out then for the pool double pay people and not have the funds to catch up. Since Eligius pays from the coinbase/generation transaction generally, things need to be pretty precise on this side.
On another note, the frequency of which we find blocks has increased substantially. This is causing a minor amount fail-safe blocks to pay the cold wallet without automatic payouts a bit more often due to all of the sanity checking I have in the CPPSRB code. Sometimes it is not able to compile a new payout list for the next block before a miner actually finds a block. So, the payout queue (and next manual payout) gets larger when the block pays the offline wallet instead of miners directly. This isn't really an issue, and everyone is still paid and credited normally. However I am going to work to optimize the code so that the next payout list can be ready and fully verified a bit faster so this happens less often. Again, security related code in place here. I prefer security over convenience in this case, especially when it comes to the amount of funds handled here. I'd much rather manually pay everyone a little later, but correctly, than to have something not get verified and have even one person paid incorrectly.
Anyway, aside from these minor items, all is well on this end!
Happy mining,
-wk