Then they do pseudo experiments ignoring all the things I said above and show a better payout at some random other pool for that day and they're lost for good.
While you're technically correct, I would contend that to the average user, his "pseudo experiment" is a pretty good indicator of his profitability, delta his very short term luck. I understand the math behind this, and why this is wrong in the greater scheme of things. I know about variance, and about short term spikes (or dips!) in luck, and all the good stuff about p2pool and why it's better for a decentralized network than other alternatives.
But here's the thing - all of that is irrelevant to Joe Blow Miner. I have X GH/s of mining power. If I point them at p2pool, or at GHash.io, or at Slush's, or at any other pool, I'm still "supporting the network", so long as I'm cautious about fraudulent pools or ones approaching 40% or more of the network. The differentiating factors then become ease of use and profitability.
I'm pretty sure we can all agree that running your own p2pool node is a lot more complex than just typing in a URL into your miner. Using someone else's node is the same as any other pool.
So let's look at profitability. The common refrain when users complain about uneven payouts from p2pool is that it will all average out over the long term. This is not a useful answer to a user who's burning electricity for (what seems like) no gain. An answer of "wait a few months" is not going to sway the user. We need a better answer.
The thing is, while variance may average out over the long term, that only matters if
difficulty remains the same. If you think that I'm wrong, let me offer you a scenario. Hypothetically, let's say that based on current pool hashrate and network difficulty, we can expect one block per day. I'll offer you a choice. You can get three months of double the expected block success (two blocks per day initially), followed by three months of zero payouts. OR, you can get three months of zero payouts, followed by three months of double the expected success rate.
The naive answer is that both will average out to the same thing - and that answer is correct
so long as the pool maintains the same hashrate and block difficulty throughout the six months. In reality, even if we hold the pool hashrate constant over six months, your total payouts with the two options will be radically different due to the increasing difficulty rate. The double payouts followed by a dry spell will earn much more than a dry spell followed by double payouts, simply because you're getting paid when difficulty is lower - thus more blocks are found during a given time period.
And that doesn't even take into account the reduced pool hashrate that will result from a three-month dry spell.
So the short answer is the same it's always been: we need to find a way to make p2pool more attractive to users. That in turn will increase our chance of finding a block in a reasonable time, reduce the variance, and increase profitability for everyone. And yes, I realize the catch-22 inherent in what I just wrote.