How does this deal with variable or unknown block rewards? Already now block rewards are rarely 50 BTC and the endgame scenario is anyways random block rewards (which should just be profitable).
It doesn't. For now I recommend (for all reward systems) that the operator keeps transaction fees, and take this into account in his business plan when he decides what pool fees to take. I'll investigate this more thoroughly as it becomes a bigger problem.
EDIT: As discussed below, this turns out not to be a serious problem. Though variability in block rewards does increase operator risk.
How is maturity time affected?
You should look not at the time until all the reward is obtained, but rather at the average time it is received. So if your reward is 2 BTC, and you receive 1 BTC a million shares from now and 1 BTC two million shares from now, I count it as a maturity time of 1.5M shares. If you consider time to be fully repaid important for reasons of making a clean break from the pool, I have a different suggestion to address this which I will post about later.
When o=1, you can choose r to have whatever tradeoff you want between maturity time and variance. It turns out you get the same tradeoff as with 0-1 cutoff. A good choice is r=1+p, resulting in decay by a factor of e every D (difficulty) shares, with maturity time of 1 (expressed in multiples of D) and variance of (pB)^2/2. This is the same as with PPLNS when N = 2D. Then you can decrease o (and change r accordingly), keeping maturity time constant while decreasing variance of participants (at the cost of increased operator variance).
How exact and how "stable" are payout estimates displayed on a pool website (if I stop mining, how fast do shares decay)?
The decay rate, and the accuracy of the estimates (which is basically the inverse variance), are tunable, and as discussed above, start as good as PPLNS and can then be improved.
How far can the pool in theory go into minus to "absorb some variance"?
The operator can only lose if f is negative, and in this case, can lose up to (-f)B per block found. Compare with PPS, which is equivalent to f -> -Inf and the operator's loss per round is unbounded.
Oh, and it might be great to have a google spreadsheet with example calculations + chart porn!
I'll think about it.