Topic: [1500 TH] p2pool: Decentralized, DoS-resistant, Hop-Proof pool - page 711. (Read 2591919 times)

The dead shares are not relevant to the bad luck since they can actually solve for a block and result in everyone getting rewarded. P2Pool does look at them and double check if they solve a block before reporting back to cgminer as 'rejected'.

The type of problem we are looking for is exactly the opposite actually.. ie someone who is successfully submitting non-orphan, non-doa shares, but yet not finding any blocks. I can only think of far fetched scenarios under which this might happen but here are two such examples:

1) Someone modifies their p2pool code so they submit shares as normal but never submit a block. Stupid and unlikely, but theoretically possible.
2) Some obscure bug in some combination of bitcoind and p2pool, such that rarely a found block fails to be passed on to bitcoind (and yet shares and getworks continue to flow freely, the bug would have to only affect the case where it is an actual block solve)

As an example of 2), if anyone finds this line in your logs, congratulations.. you are the problem
"Error while processing potential block"

How does the p2pool know the # of dead shares?
Aren't DOA shares detected at the node level, rejected, and then never forwarded to peers?
Do the p2pool.info charts include dead shares?  If so they are slanting the luck measurements.

Note: the questions refer to dead, aka DOA, aka dead on arrival, aka "rejected" (in cgminer) not orphaned shares.

Binomial distribution - couldn't work out an average # for share difficulty throughout the life of p2pool, so took the values for shares submitted vs expected shares required for our current number of blocks from p2pool.info and assumed avg difficulty of 1.5 million. So N=somehugenumber, p=0.00000066667, k=actual shares found/p.

Not the best, but shouldn't be far off.

Well, remember that a variance of 8 times is not all that rare when it comes to block times and share times.
So when your talking about a base multiplier of around whatever the difficulty is now (around 650?) your going to be waiting quite a long time before you'll expect to see some convergence on your expected hashrate.

Yesterday I ran an Icarus for over 3 hours on a standard pool and got 105% of my expected 1 difficulty share rate with 380MH/s
Multiply that 3 hours by 650 ...
For the first 5 minutes it was over 120% - not that uncommon also to have it vary a lot early on - again multiply that 5 minutes by 650 ... and you get over 2.25 days

The point of a pool is to reduce variance - and although p2pool does that like every other pool, it's base figures are around 650 times higher due to being 650 difficulty shares. So expect a lot longer to see your expected share rate (i.e. your average can be higher or lower for a lot longer)

(Edit: corrected that hash rate above )

It was never 2% when @ 91% luck over 90 days.  I will run the probability numbers for current luck.

I like the direction he's going. Maybe it's not a pool issue (maybe it is), but it could just be an issue with the graphs projecting incorrect values. Afterall, p2pool.info is an estimate of hashing speeds and such. It's using the estimated speed (based upon shares) to determine the overall speed of the pool and from there calculating the projected block time. I don't know if you have seen your miner's speeds according to p2pool.info, but my miner is 200mhash/sec faster than my cards are actually mining.

Nobody suggested anything suspect in it, just that something may still be wrong.

I have to agree. I'm no statistician, and maybe it would be worth having someone better versed in statistical analysis to pipe in, but if that one line is indeed a projection of what should be expected, and the other line, what we're actually getting, with variance added, we should expect those lines to cross and re-cross in several different places and continue to do so in the future.

The actual results line, when smoothed out, should be pretty close to the projections, but it is spending too much time below the projection, which would imply an issue somewhere with the actual results we're getting.

One thing the graphs do not suggest, which is correct? Is the actual results line reflecting something wrong, and the projection is correct, or is the actual results line correct, and there is something wrong with the calculation for the projection?

Is it possible to make similar graphs for other operating pools as well to compare against?

Nobody is suggesting that there is something hinky going on with the code, but just by looking at the statistics, after this long a run so far, that as long as this pattern continues, there is the increasing likelihood, of something not operating quite right somewhere.

It could just be the projection calculation too, who knows, but it begs looking into.

-- Smoov

Strange, I could see the pics in firefox, where I uploaded them with. In another browser I couldnt see them neither.
I reuploaded to another host, hope it works now.

Thanks for the info, you two.

Ente

I don't see them in your post above.

Thanks for the info.
I can see the images fine in that post..
Will check and repair.

Any one else, do you see the three pics?

Ente

Ente, I'm not able to see your images, I had to answer your message and copy and paste urls (safari and firefox on a mac).

In this answer I've removed the img tag from the quote so that the url is clickable.

spiccioli

I am far from suggesting there is any malicious intent in this. This does not make sense at all and contradicts everything I see in this thread.
Of course there won't be a "hey, I found a haxx0r backdoor in line 713 which steals our monies!!". And no, I don't plan to read the code, as I am not knowledgeable to do so.

But, obviously for me, there is a general and (now) constant problem nevertheless. I don't write this to hurt p2pool, the opposite is the case. There are enough outcries and "have to go back to xypool" here to justify a closer look, even without the problem being visible in the luck chart for weeks now.

Ente

How are you calculating that?

All right.

This is what the "luck" graph looks like:

(as can be seen here: http://u.forre.st/p2pool/luck.png)


This is what "variability and luck" would make me expect it look instead:



But, in fact, it looks much more like this, which is not "variability and luck", but a inherent problem besides variability:


The red line is like the average block-finding rate. Do you see the angle between the black "hashes calculated" graph and the red "blocks found" graph? THAT is what I am talking about, not the stochastic ripple in the original blue line, which I crudely smoothed out with the red line.

It does not seem to be caused by block orphans, as people stated before that these are in the low % range. Neither has it anything to do with local orphans or local deads, as these don't add to the (black) "hashes calculated" neither.


I am not confident enough to calculate the "we are losing x % hashingpower" from this graphs, maybe someone else is. To me it seems clear to be a constant "loss", therefore the two graphs resemble two lines with an angle.
My wild guess would be we are effectively losing 15% hashingpower, in the sense that we have 15% less blocks, since 2 months.

Ente

edit: fixed images

Yes, what I'm suggesting is that we may be having issues with orphaned blocks or some such issue.
Just saying, the odds of luck as bad as p2pool now displays is less than 1%

Please stop spreading FUD  P2pool is open source and there is nothing suspect in the code. Do you know what variability and luck is?

I'm beginning to suspect not.
back when the pool was at 91% luck over 90 days, the odds of it having such a poor reward after so much hashing was ~2%.
It's obviously lower than that now. I'm giving it another 48 hours then I'm going back to abcpool 

Yes.

No, of course not "compensated" as a reaction to the bad luck streak. But "evened out" in the sense that this single bad luck streak has less and less influence as the time or number of draws rise.

I am pretty sure we all are more or less talking about the same thing, and actually agreeing. How about we say "there are bad times, there are good times, everything is fine", and leave that page-long non-p2pool-related statistics/probability stuff?

The still unanswered question remains:
Is p2pool in fact statistically "working", in the sense that 100 Th will yield the [statistically] same amount of blocks as 100 Th on a single bitcoind or centralized pool?

Ente

IMHO Kano is correct, and I don't see the relevance of the Central Limit Theorem to the question of whether bad luck in the past will be compensated for in the future.  The idea that tossing a fair coin and getting, say, six heads in a row will be "evened out" by disproportionally more tails in future tosses is known as the Gambler's Fallacy.  If you've had bad [or good] luck in the past, then your lifetime expectation for luck is bad [or good]; you cannot change the past -- it is the baseline for your future.  More generally, probability theory deals with unknown (often future) events, not known past events.