New Block Difficulty Algorithm Proposal:The Problem: We need to differentiate a "spike" from "regular" total net hash rate to keep block rate steady at 1 block every 2 minutes, or 30 blocks an hour, in order to respond to large miners jumping in and out of mining RPC.
Considerations: The block difficulty needs to adapt quickly, so it needs to evaluate blocks solved more frequently than it does. If it doesn't, it misses spikes and blocks fly by every few seconds. The second challenge, however, is how to return to baseline. We must flexibly handle spike difficulty settings vs. regular difficulty settings so that a return to lower difficulty doesn't take a long time when the spike is gone, otherwise confirmations take a long time and small miners lose interest. This second challenge requires evaluation of the block rate over a
long time to establish a non-spike period's "baseline" mining rate.
So we need two windows of time consideration, a short-term window to detect spikes and a long-term window to establish regular mining rates. The math should be as simple as possible for testing and transparency. I propose using quartiles, because it's as simple as it gets for identifying outliers:
http://en.wikipedia.org/wiki/QuartileProposal:Evaluate total net hash every 5 blocks, or, if feasible given network communication speeds and interface with MPOS, every block. Then distinguish whether we are in a spike based on the net hash rates over the past 180 blocks (=6 hours of blocks according to the desired rate of 30 blocks per hour.) If we are in a spike, adjust difficulty upwards quickly. Stay there until spike passes. When spike passes, revert difficulty to a factor based on the median block rate for the past 180 blocks.
Algorithm:Every 5 blocks (or 1 block if possible), calculate:
Quartiles for the total estimated net hash rate based on the past 180 blocks.
Are we in the upper quartile = in a spike? (Is the present net hash rate in the top 25% of net hash rates we've seen in the past 6 hours?)
Yes: Adjust difficulty immediately based on present net hash rate, to retarget to 2 mins per block.
No: Set difficulty based on the median net hash rate over past 180 blocks to target to 2 mins per block.
Advantages:Rises quickly to meet rapid increase of hashrate, preventing plundering of blocks every few seconds.
Falls back to normal quickly when the spike is gone, preventing long delays in confirmations when net hash rate returns to normal.
Distinguishes a spike from a long-term increase of hash rate if the coin gets popular. In 3 hours (90 blocks), a long-term high hash rate becomes the median normal hash rate. Interim adjustments should be smoother, based on the median, rather than the average.
Verification:Anyone have a CSV formatted list of timestamps and block numbers handy? I could grab a list from altexplorer.net, but I would need to muck around reformatting it. This could be tested in a spreadsheet first, and then on a testnet if RPC has one.
Tweaks:(a) Defining a spike: Instead of looking at the upper quartile, we could define a spike more narrowly as a net hash rate in the uppermost 10% or other percentage of the blocks we've seen over the long-term window.
(b) Length of long-term window: I picked 180 blocks / 6 hours as a safe range of time to establish the "regular" net hash rate. This window could be longer or shorter, but it should remain a high multiple of how many blocks we wait before recalculating the difficulty.
(c) Evaluating every 5 blocks would be pretty fast (10x faster than present) but the quicker the algorithm adjusts, the fewer blocks are lost to plundering.
Your thoughts??