Topic: [1500 TH] p2pool: Decentralized, DoS-resistant, Hop-Proof pool - page 709. (Read 2591919 times)

If a share is DOA or orphan it will still count as a real block. If you look at the source in main.got_response(header, request) the first thing it does is check if it is a real block. Only after that does it bother to check for DOA. CGminer will submit dead shares if you use --submit_stale, or if you are running a fairly recent version it will do this automatically. If you are using an old version of cgminer, and not using --submit_stale, then you could discard a valid block before it even reaches P2Pool.

Yes, it is always the ping from your current computer to the node. Everyone will get different numbers and if your node is on your machine or on the same network your ping will be very low. In effect it shows you the node that will be best for you and you alone to connect to.

The version number isn't in the data below so I don't display it.
http://p2pool.hopto.org:9332/local_stats

Meh - I somehow managed to post this in the BFL thread when I mean to post it here ...
OK fixing that, here it is here where it should be.

... and now I'll ask an evil question coz I'm curious about it ...

I added the code that did network block detection in cgminer.
It reports "BLOCK!" on BOTH Accepted and Rejected shares.
I've wondered in the past if anyone has ever had a Rejected "BLOCK!"
i.e. you found a block, but you were JUST unlucky enough to find it in the time frame where it would be Rejected
( of course people have, but they'd need to keep their logs and check them ... I do keep all my logs and sometimes check them Cheesy )

They should be rare but also easy to calculate how often you should get them ...
The Rejection rate of your shares is a percentage.
That Rejection rate is also the % of blocks you find that should also be a Rejected "BLOCK!"


... now for a code change suggestion ...

So if you get 10% of your shares rejected then 10% of the blocks you find should also be rejected ....................
However ........... of that 10% that get rejected, 98.3% of them would be accepted by the bitcoin block chain but are invalid for the share chain and are thus thrown away .............

(I also wonder if this is a factor in the luck discussion - but I've no idea if it is)

Yeah I think there should be an exception in the code to allow rejected shares, that match this circumstance, and enable support for the exception in the share chain
i.e. if you get a Rejected BLOCK! it should make it all the way to the bitcoind and then it's up to bitcoind to reject it if it was a real block chain LP (not a P2Pool share chain LP)
Of course if this was implemented: it would also mean that everyone mining with cgminer on P2Pool should enable submit stale
(and also P2Pool should tell all miners to submit stale shares)

I'm starting to think those are ping times from the client to the p2pool nodes, not the node to the other nodes. Loading the page from the miner would be most accurate.

is it possible to show what version of p2pool the node is running, or is that not available?

also... woohoo! lowest ping time on your list!

138ms!

-- Smoov

Cool thanks.

Seems like hopto's ping time to my node is consistently less than the ping time to hopto. I guess I have a good network?

You can save the html page to your own web-static/ folder and then you'll have it too

The ping times will be unique to where you are and the pools are hard coded from ones people wanted to be public. It is possible to grab a list of all your connected peers but maybe not everyone would appreciate that.

Pages like this record tons of info about each share, and it is recording atleast some deads:
http://p2pool.hopto.org:9332/static/share.html#0000000000555d58d25a50651e47ec138edd63f4d7dfa868a9b679e160c79524

I hadn't thought about the fact that blocks can be longer than sharechain. I see a couple work arounds
a) quick & dirty for analysis I could simply cron job to archive the sharechain at quicker interval than shares fall off the end and recombine it to get a continual history

b) slightly better would be looking into the code to make p2poool keep shares around longer and/or make it a command line param

As far as orphaned (but not DOA) shares I need to look at the code on propogation.  If they aren't fully distributed then how does a node calculate hashrate & stale (better called orphan) rate?  


So following that train of thought how does an arbitrary node determined the stale (orphan) rate for the network?  My node right now says orphan rate is 8.9%.  How does it know?  It must be notified of all orphaned shares (either by receiving shares and incrementing its local orphan count or by some orphan share message).  My understanding is things like hashrate, orphan %, etc all all calculated locally.  We just need to tap into this.

BTW I am not expecting you to do any of this based on my hunch.  This weekend I will dive into the code and see if we can find a method to count shares (well technically sum of share difficulty).  If I can and it diverges from p2pool.info stats then we can look at using that to improve accuracy of the stats.  If it doesn't well then I was the only one who wasted his time.

When do the rest of us get this page? This looks nice. http://p2pool.hopto.org:9332/static/nodes.html

It would be an interesting experiment (and much easier because you can ignore the first problem by picking blocks that are shorter than 18 or so hours) to spot check.  If you want to calculate how many hashes you think went into a few blocks, I can tell you how many hashes p2pool.info thinks went into it and we can see how much variation there is in the two methods.

I don't fully understand the internals of the share chain distribution, but I think there are a couple problems that would have to be solved:

• We have a significant number of blocks where the duration was longer than the length of the share chain.  So in order to add up shares after a block is found, I would have to invent my own persistence of the the share chain that never got cleaned up.  Not trivial or fun (and fun = motivation to do it)
• The "hashes done by the pool" include hashes contained in stale shares that are not fully distributed through the p2p network.  Because those stale shares mostly could have been valid blocks, they need to be included in the luck calculation.  In fact a lot of our blocks actually were stale shares (as determined by the fact that they often aren't announced in IRC).  The current p2pool hashrate calculation handles this by counting the valid shares and then increasing the count based on the pool-wide stale rate.

By psueodshare I assume you mean the low diff share cgminer thinks it is mining?
Which DOA stat is for the diff 1 shares and which is for "full sharechain difficult shares?

DOA is reported in 3 ? places
1) In graphs both total and per miner
2) On the /static page
3) In the console window (which I find myself using less and less).

It may have just been a timing issue.  I think this weekend I will restart p2pool w/ no historical stats and restart all miner instances to get a "baseline".

I am also trying to figure out an easy way to convert JSON file into relational db tables so I can do some deeper digging.

My hunch (and I would like to stress completely unfounded) is that there is some error in the p2pool.info stats.

I think this error comes from the double estimations.

The raw data is shares seen by p2pool node.
p2pool abstracts that into global hashrate averaged over 10? minutes
p2pool poools that hashrate every 5 min?
p2pool uses that to construct avg hashrate over the block
p2pool use that and block time to determine # of hashes in block
p2pool compares # of hashes in block vs expected # of hashes in block to get "luck"

I am wondering if there is a better way.  simply look at all the shares since last block.
diff 1 shares for block = sum of difficulty of shares seen by node since last block
right? 

You do mean the dead on arrival pseudoshare rate, not the dead share rate, right? The dead on arrival rate should match cgminer's, but the one displayed on the console is only averaged over 10 minutes. Did you look at the graphs to get a more accurate rate? (area of local dead/area of local total)

That is exactly how pool hashrate is calculated, as far as I know.  p2pool does the calculation and publishes the hashrate at http://localhost:9332.  I just pull from that value and calculate an average over the duration of a block.


Agreed.  I have said basically that same thing several times in this thread.  If the hashrate reported by p2pool is too high, our luck will appear artificially lower than it actually is.  Forrest has several times reassured me that the hashrate calculation being done by p2pool is correct, but I have not validated that myself by reviewing code.



Yes.  I collect the pool's hashrate from the http://localhost:9332/rate API every 5 minutes.  Then calculate the average hashrate during the time between when blocks are found.  Then calculated # of shares exactly as you indicated (hashrate * duration / 2^32).  The only reason I use diff 1 shares instead of hashes directly is the convenience of using smaller numbers.

Raw hashrate data is here:  http://p2pool.info/stats

Note, hashrate stats are only 1 per hour for the first few months of p2pool's life because that is what forrest had available to backfill my database.  From the time p2pool.info was created forward (sometime in early Feb), hashrate data is in 5 minute increments.  The code that calculates the average hashrate actually calculates a weighted average so as to deal with mixed frequency of hashrate stats within a single block.

Ah yes. I see now.

less than target or greater than difficulty one or the other.

I think you meant less than, not greater than. I don't know the answer to your question though

So I have been taking a close look at p2pool lately (for my own peace of mind if nothing else).

Shouldn't p2pool node report all DOA as rejected back to the miner.

Across my entire farm cgminer shows ~ 1.5% reject rate ("R" in cgminer).  p2pool shows ~2.5% DOA rate.

My assumptions is the logic flow is something like this
1) miner requests work (getwork).
2) p2pool provides work.
3) miner returns low diff shares (I have mine set to a static diff 1).
4) p2pool verifies share
   a) if share is DOA it increments DOA count by one, and sends reject notification to iminer
   b) if share is > share diff and not dead it submits it to all peers for inclusion in share chain and sends accept notification to miner
   c) if share is > block diff it submits it to bitcoin network and sends accept notification to miner

So why doesn't DOA % match reject % in cgminer?

p2pool doesn't record the # of 1 diff shares anywhere.  twmz is estimating the # of 1 diff shares by looking at avg hashrate during the block.   If there is some inaccuracy in the avg hashrate then it will reflect as inaccurate # of "actual shares".

"actual" is more an estimate.  He is using the word actual to mean actual vs expected.