Topic: [1500 TH] p2pool: Decentralized, DoS-resistant, Hop-Proof pool - page 710. (Read 2591919 times)

Thanks for all that.  If there is an error my guess is it is in the "average hashrate".  It might be better to simply count total shares (and dificulty) received by your node.  

I will take a closer look at how the avg hashrate is calculated.  An error there would mean you are starting with "dirty data".

Note: I am not saying there IS an error just that given the 15% divergence we should take a closer look.  Your computation "downstream" from the avg hashrate computation looks valid.

On edit:
A clarification
You are calculating this by polling http://localhost:9332/rate periodically? how often?
then getting avg hashrate over the block?
then duration * avg hashrate = # of hashes?
then # of hashes / 2^32 = # of shares?

Either way, both 'actual' and 'estimated' shares should be geometrically distributed, right?

I would say the "difficulty 1 shares" are close enough as an estimate of hashing power. They are easy enough to find and other pools use them for calculating the payout (and surely the pool's hashpower) too.
So, if we can be sure the "number of diff-1-shares submitted between the last share and this share" is correct, I would use it as the base of all later calculations.

I don't understand that. Sounds contradictious to me?

Ente[/list]

Raw data is here:  http://p2pool.info/blocks

For each block, you can see:

• The actual number of "difficulty 1" shares submitted before the block was found.  Note, that since we don't actually know how many "difficulty 1" shares were submitted, this is an estimate of how many difficulty 1 shares should have been submitted based on the average hashrate at the time and the duration of the round.  And so if the hashrate published by http://localhost:9332/rate is wrong, then this number of shares will also be wrong.
• The estimated number of "difficulty 1" shares theoretically needed based on the bitcoin difficulty at the time

% expected for a single block is:  actual shares / expected shares

% luck over a 30 day window is (sum of expected shares for all blocks found within 30 days) / (sum of all actual shares for all same blocks)

About 1):
Every miner working on Bitcoin, no matter if pool, solo or p2pool, has to decide what he wants to deliver before he finds the hash. That is because he has first to decide *what* to hash, then try a billion variances of that data, then, if/when he finds a valid hash, he can not change the data to still have a valid hash.
Exactly that discussion came up some weeks ago about the share-difficulty:

1) decide which share difficulty you work on, i.e. 600 or 2000
2) hash a million nonces, where you are allowed to change a "non-important" part of the data for every try, so you get a different hash result
3) eventually, a hash reaches the (in 1) decided upon) target. that means the first x letters of the hash are zero
4) submit exactly the block you solved (including the exact transactions and random nonce data) to the pool, or bitcoin, or p2pool network
5) profit

what you try:

3) you found a block+nonce combination with a valid hash. lets say you worked on "600 difficulty" data, which you decided in 1). You now luckily and randomly found a hash with much more zeros than needed for "600", in fact it would pass a difficulty of "5000"!
4a) thinking "yay, I want to be paied out for those "5000" now, not just the stinky "600"!
4b) you take that exact block+nonce and edit that "this is a 600 block" to "this is a 5000 block lol"
4c) you hash that block+nonce from 4b)
4d) you get an entirely different hash, which surely won't even pass a "1" target. crash and burn.

I wrote that example about p2pool share difficulty because I find it easier to understand. The point is, you have no chance to change the block+nonce to still have the same or a similar hash. The hash, and the hash only decides upon "jackpot" or "totally worthless".  And, of course, all important data is included in the block. Actually that stuff is called coinbase, and, for example, includes the hash of the block before too, among other stuff.
Now when you work on p2pool shares, it includes both the "p2pool share data" as well as the "bitcoin block data". Any manipulation will change the hash, so if you strip the p2pool part of the data you found a valid hash for, the hash does not fit any more.
(I didnt read about the details yet, the combination of both p2pool- and bitcoindata is quite clever, else we couldn't ever solve a bitcoinblock at all.. merge mining is on the same topic)

The only thing you, as a p2pool miner, can do, is drop the valid bitcoin block hash you found. No profit for the p2pool gang, but obviously no profit for you neither. The only effect this has is it will make the "luck" graph look bad. Which is exactly what we are talking about, of course..

Hope that helps, and correct anything I got wrong!

Ente

@D&T:

The '% of expected' round 'lengths' published on p2pool.info are uniformly distributed, and have a very regular mean of about 160% (p-value for the linear model is less than 10^-12). If the '% of expected' works the way I think it does (normalises round length to D) then I'd expect '% of expected' to be geometrically distributed and the average to be 100%.

I'm think I'm not getting something about the way the '%luck' is calculated?

Thank you two for the explanation. Makes perfect sense.

Negative binomial?

Because it is transferring around the network to every p2pool every share (not just every block like the normal block chain)

So shares must be rare enough to only happen at a rate that can be handled - 10s is quite fast

To make it 10s on average per share means that the difficulty must go up to match the network (350GH/s)

350GH/s * 10s / 1 diff = 815 diff (hmm is 815 correct?)
(1 diff = 2^32 Hash)

p2pool.info relies entirely on the number reported by p2pool, itself, at http://localhost:9332/rate   If that is wrong, then the luck calculation will be wrong.  Fixing it is not something I can do.  The fix, if there is one to be made, needs to come from forrestv.

No it is perfectly on topic.

p2pool tries to maintain a 10s LP windows (10s between shares).  This is a compromise.  Shorter LP window = higher orphans.  Longer LP window = higher difficulty.

One way to look at it is p2pool is ~350 GH/s.  At diff 1 shares that is ~81 shares per second or <0.01s between shares.  Work would become stale before it even finishes propogating the network.

So the difficulty is simply a compromise.  It takes difficulty of ~800 to keep 350 GH/s producing 1 share per 10 seconds.

I agree completely with this summary.. the percents I'm not sure about. I'm pretty sure the leads are worth exploring though. If we are mistakenly including too many dead hashes in the luck calculations then that is no good for the pools image.

This is a bit off topic, and is probably something major that I should know, but why does p2pool use such high share difficulties?

Your numbers don't make sense.

You agree conventional pools have 1% to 2% stales due to bitcoin LP and invalid hashes.  Let's say 1.5%.   If 98% of p2pool DOA were NOT due to those reasons then 2% of them are.  Thus 1.5%/2% = 75% it would have an overall DOA rate of 75%.  For 98% of p2pool DOA to be due to 10 sec LP then it would need to have a DOA rate of 75%.  Obviously the % due to 10s LP is much lower.

If p2pool has a 4% DOA rate (??) and 1.5% of that is due to Bitcoin LP and invalid hashes then they make up ~40% of all DOAs.

Still I think I was confused.  p2pool doesn't track global DOA only global "stales" which if forrest can verify I believe is orphans only.


One correction:
You are forgetting invalid hashes.  No GPU produces 100% flawless hashes.  Some % (0.1% to 1% depending on overclock, temps, ASIC quality, etc) are just bad.  Nonsense hashes that solve no block.

TL/DR version

Conventional pool rejected shares = invalid hashes + stale hashes (after Bitcoin LP)
p2pool DOA shares = invalid hashes + stale hashes (after Bitcoin LP) + "pre-orphaned" hashes (hash that would be 100% orphaned it submitted)
Only the last one is valid work which can solve a block.

My understanding is that > 98% of P2pool's DOA's have the potential to solve a block. The only ones that would not be useful are those DOA's that came right after a real block chain LP, ie once every 10 minutes. This also matches up well with the 1-2% of shares that regular pools reject, and I agree, 100% of these regular pool rejects are useless and have no chance of solving a block.

DOA simply means the share is invalid by the time the local node gets it.  Don't make any assumptions beyond that.  Conventional pools reject a sizable number of shares and they have no 10 sec LP.  "Some" DOA may still be valid but not all of them are.

The "horrible" miner is just an example.  IF (and I am not sure they are) DOA are included in p2pool.info luck calculations they will skew them.

That is a good question.. the code that would do this would start at around line 577 in main.py.. and in looking it may not in fact recover from this, it seems to just log an error. But I'm going to shutup and let forrestv answer this one.

I believe most DOA CAN solve a block in P2Pool's case. They are only DOA with regard to the 10 second shares, but they are probably hashing away on a perfectly valid 10 minute real block. I'm not 100% on this though. And I see your point.. in the extreme case a terribly connected miner, who will never solve a real block, could be hashing away and getting say 50% stales and 50% DOA. He is not getting any shares and so is not hurting the rest of us, but his hashes may be skewing the luck. Again I'm not 100% on this either.

Regarding #2 doesn't the local p2pool node pass the block to p2pool peers also.  So a failure on one node's bitcoind would prevent block propagation.  Now it may result in slower block propagation so we should see that in a higher block orphan rate (which we don't) rather than just lower luck.

If I am mistaken and p2pool node doesn't do that this may be a way to make the network "stronger" so that the failure of one bitcoind can't lower block finding rate.

Of course they are relevent.  Most DOA can't solve a block.  They are stale (due to bitcoind LP) or invalid.  All pools have these (commonly called rejected).  They are never included in luck calculations because they weren't good work to begin with.

Say the average DOA rate is 3%.  If those 3% of shares (unpaid, worthless work) are included in luck calculations that the expected # of blocks will be inflated by 3% making the pool appear to be 3% unluckier.  No conventional pool includes rejected shares in their luck calculations.

As an example of how invalid it is to include DOA shares I could point my miners towards a normal pool and have the found shares also submitted to p2pool.  Obviously the shares wouldn't be valid for p2pool so it would be a 100% DOA rate.  If the DOA rate is/was included in luck calculations then I made the pool appear x% unluckier.

So I will ask again.
1) HOW does p2pool network know the DOA rate?
2) Are they included in p2pool.info luck calculations?