OK. So there are unfortunately a whole bunch of things going into the profitability for today, some of which are unrecoverable, some are essentially display issues (recoverable), some are outside of our control (some recoverable, some unrecoverable).
First, the unrecoverable:
So from what I can tell, we had about 3-4 hours early this morning of less than optimal mining (the definition of less than optimal I'm not sure on, around -25% seems likely). This was due to two factors. First, the database issues caused our switcher to make less than optimal choices. Second, the database issues caused a problem in our block propagation strategy. We typically relay blocks to remote nodes to better help with propagation time (fewer orphans). Due to the database issues, this failed as well, so we ended up having more orphaned blocks during that time (20% instead of 10%), these happened more on lower difficulty coins than high ones (as expected), so its not a hard 10%, but its definitely a factor. This is obviously mining time spent, that cannot be recovered, and we're expected to have _actual_ lower earnings today than normal because of it (guessing 10% or so).
Second, the display issues:
We track balances every few minutes. Its what gives us the running tallies of coins, earnings for the day, and overall profitability numbers. The way in which we track them is essentially snapshots of what we _think_ we have in earnings. This is coins that we've already exchanged (known), coins that we think we can exchange (confirmed blocks, waiting for deposit to the exchange to convert), and coins that we hope we can exchange (blocks that aren't confirmed yet, could still easily be orphaned). When we solve a block, we calculate what we think it is worth (coins x price), and add to our balance. If in the future the block gets orphaned, the balance in a future snapshot will be less than the one previous. What this means, is that if a block is earned in a previous day, it will be added to the earnings for that day, but if it is orphaned today, would be removed from earnings for today (yesterday is higher than it should have been, albeit slightly, today is lower than it should have been).
This compounds with the third point, issues outside of our control:
A few days ago (3-4) we earned a bunch of coins (Reddcoin, Karmacoin, Mintcoin, Smartcoin), these were all on a new exchange (coinmarket.io). We ran them in testing and everything was fine. We started mining them in rotation, everything was fine. We started mining them fully, everything was fine for a day. Then coinmarket went offline, and has been claiming they'll be coming back "today" for about 3 days straight. In that time, all of the coins that we earned while mining, have been stuck there, unable to be exchanged. These coins were essentially "earned" 3-4 days ago (higher earnings on those days), and have been dropping in expected price (other exchanges prices included) over those 3-4 days (mostly today/yesterday), leading to today's balance being lower than it should be.
The last piece is something that I expect is fully recoverable (more of a display issue):
The way we calculate price is still a relic of the way it came to be. We originally (day 1) started tracking prices as a method of determining the value of coins (imagine that!). This includes depth analysis. Essentially depth analysis can be explained with a simple example: There are 2 bids for a coin, one bid for 10 of that coin for 1.0btc each, one bid for 10 of that coin for 0.1btc each. Assume each block is worth 10 coins. The first block you mine is worth a good bit (10x1 = 10btc!), however, the 2nd block you mine is worth _significantly_ less (10x0.1 = 1btc). This is extremely important to be able to tell how much of the coins you mine you'll actually be able to sell (and their expected prices). We've always tracked prices as a "what is the next block worth", as that was always the most important piece from a switcher point of view. This is the price we use across the board for things like "expected btc value".
Normally, using this price isn't an issue, its a conservative estimate of actual prices. In the previous example, if we had mined 2 blocks (20 coins), we would show the expected value as 20x0.1 = 2btc (note the low price), when in actuality, it would be 10x1 + 10x0.1 = 11btc (actual price). This normally isn't a problem because we normally don't have that many outstanding coins, and as they convert to btc, we get the actual price they were converted at, and our stats stay very close to actual. The time that this fails, is times like today. Cryptsy's API is down (not sure why), so we haven't been able to auto-trade coins overnight. This means coins have been piling up (they've actually been traded now, but our scripts can't see that until the API is back up), and because they've been piling up, and earnings are using the conservative estimate, earnings show as lower than in actuality.
All of these things combined are essentially factors in the current earnings looking rediculously low for the day. And unfortunately, telling how much is each of them is a really challenging thing. I'm still digging through it all, just thought I'd give you guys an update to what I've found so far.
