Just checked and all my miners are still happily hashing away on the pool.
So I can relax and go make that second cup of Sunday morning coffee.
Yeah sorry about that - but I'm gonna ruin your coffee again.
I have to restart KDB once more.
The initial problem that started the whole set of restarts happened as a shift was being processed.
Unfortunately the shift state said it needed to be fixed back on the first reboot (and I saw that on the console) but alas with the repeated failures it got skipped over after that.
I've rolled the shifts in the storage DB back to "a2nob belldandy" and the next KDB restart will reprocess everything from then on again.
You'll notice your "charlotte" shift is empty at the moment.
Sorry about that - another KDB restart about to run in the next minute.
All back up again finally - at 9:05 UTC
When KDB came back up, no stats were getting to it from ckpool - then ckpool deadlocked itself and crashed.
This of course meant a failover to your backup pools, then fail-back would have started at 9:05 UTC
This cleared up all the remaining problems since both KDB and ckpool had now restarted - and now all is going ok finally.