I thought I might explain a little of the CKDB share storage for anyone curious.
I was gonna post this in the pool thread, but of course realised, here is more appropriate.
(It relates to a change/fix I'm doing at the moment and thought I'd write out the current details at the same time)
CKDB has a 2 level structure for storing shares.
ShareSummary and MarkerSummary
ShareSummary is implemented.
MarkerSummary isn't yet - this simply means that I use a LOT more ram until I do implement it - and fast reporting at the 'Marker' level of course isn't possible until it's done.
CKPool creates work per WorkInfo which is effectively each block template we get from bitcoin.
We do this by default every 30s (faster than 'certain' other pools) since that is both good for Bitcoin and good for the pool.
For Bitcoin it means faster expected average transaction confirmation.
For the pool it means higher expected average transaction fees.
An obvious Win-Win for both.
CKDB groups shares, per WorkInfoID (the internal ID of the WorkInfo) and creates per worker ShareSummary records of these in RAM and in the DB.
You could compare this to what some other pools call "Shifts", with a ShareSummary record being a shift of work for a worker, of usually 30 seconds.
ShareSummary, per worker, per 30s, means a lot of ShareSummaries
The PPLNS code currently uses the ShareSummary information to calculate payouts, so the N value actually used for the payout will include the full 30s of work for the starting WorkInfoID calculated to be the start of the payout i.e. the N value used is expanded to the full first and last WorkInfoID - so, in concept, it is similar to a PPLNS "Shift" payout structure.
The MarkerSummary design allows any arbitrary range of WorkInfoIDs
The design is: to later to be able to determine the necessary WorkInfoID ranges to ensure all summary calculations required are possible, and then summarise the ShareSummaries into the MarkerSummaries and then be able to delete the ShareSummaries.
The original idea was that the Markers would be the block WorkInfoIDs and the PPLNS WorkInfoIDs and mean a much smaller amount of data required in RAM and in the database.
However, the timing of doing the summarisation to the Markers isn't possible until the end Marker is older than what is needed by other code.
For the current PPLNS used, that would mean: not until the end Marker is more than the N in PPLNS old.
i.e. all ShareSummaries would have to exist back to N share diff ago.
This actually runs pretty well on hashmine since most of the workers are high hash rate devices and have many high diff shares in each WorkInfo.
If the average number of shares per WorkInfo is low or the average difficulty of the shares is low, this of course leads to a much larger number of ShareSummaries - as is happening now on the kano.is pool.
Thus I've now decided to use the MarkerSummary to make larger "Shifts" that are both a subset of the original idea, i.e. more of them, but also at a size where it's OK to use for the PPLNS calculation - which I'll chose some value between 20 and 100 WorkInfoIDs (probably 50)
The MarkerSummary changes are pretty much next on the todo list.
