Topic: [ATTN: POOL OPERATORS] PoolServerJ - scalable java mining pool backend - page 4. (Read 31153 times)

Thanks for the feedback... keep it coming...

ok I've just uploaded the latest revision .08.  Please note the base distribution was updated at .07 to apply .08 you will need to start with .07 and apply the contents of .08 mini-binary over the top.

This contains a number of changes that I hope will address most of the 4 issues I highlighted above...

- support for partially stale shares.  Where a single chain has advanced to the next block old work may still be valid for the other blocks.  This patch checks that the work is valid for at least one block giving three possible outcomes.  accepted, accepted partial-stale, stale

- longpoll passthru.  To address the double longpoll issue. There is a fundamental design clash between longpolling and merged mining.  Basically it works like this:  When there is a double block change (i.e. btc and nmc solved by one solution) the daemons won't see the new block at exactly the same time.  From what I've seen a delay of around a 1-2 seconds is not unusual.  What happens then is that one chain advances to the next block.  PSJ checks the other chains to see if they've updated, sees they haven't so starts sending out LPs.  Before the miner receives the LP and establishes a new LP connection the second chain advances so PSJ starts sending out another batch of LP's.  Those miners who haven't got their new LP connection registered in time miss out so continue working on the old block for as long as they would normally (probably about a minute).

This patch addresses this by setting a longpoll passthru period.  Where two block changes happen within a specified period (10 seconds).  After the 2nd block change and until 10 seconds after the 1st block change has passed, any longpolls received will have a short expiry of 1 second after which they'll return new work to the worker.  This will give any slow miners a few seconds to get their longpoll in and learn that there's a 2nd new block to work on. The reason for the 1 second delay is to prevent longpoll spam.  Most (but not all) miners will immediately send another LP request as soon as they get a response.  With 0 delay this sets up an LP spam loop which thrashes the server and causes high CPU usage and low response times.  They will still spam 1/sec until the passthru period is over which is why this is called a workaround rather than a fix.  There's really no way to fix this issue completely except to ditch either longpolling or merged mining alltogether.   

On yourbtc.net my stales rate was around .15% before merged mining.

what sort of rates were you seeing previously?

I am running my equipment on yourbtc.net which is now running psj with merged mining and the stales are higher on phoenix 1.6.2 also. I am running at about 2% stales.

I was just about to ask about this.  Is  anyone else seeing higher stale rates?  From what I've heard it's only cgminer clients that are getting it.  Any info people can report back will be helpful.

There's a numbers of issues at play here I think...

1/ There is a fundamental design clash between longpolling and merged mining.  Basically it works like this:  When there is a double block change (i.e. btc and nmc solved by one solution) the daemons won't see the new block at exactly the same time.  From what I've seen a delay of around a 1-2 seconds is not unusual.  What happens then is that one chain advances to the next block.  PSJ checks the other chains to see if they've updated, sees they haven't so starts sending out LPs.  Before the miner receives the LP and established a new LP connection the second chain advances so PSJ starts sending out another batch of LP's.  Those miners who haven't got their new LP connection registered in time miss out. 

2/ cgminer may be a bit too clever for it's own good.  It does it's own checks on whether work is valid and I presume it uses prev_block_hash to check or maybe the X-Blocknum header.  If an NMC block is found but it isn't a BTC solution then the pseudo block number will advance but the X-Blocknum header (which is for BTC block in psj) and the prev_block_hash won't change.  So cgminer may think it's the same block and carry on using it's cached work.

3/ The rules for accepting a 'partially stale' share are not defined.  If NMC block advances the previous work can still produce a valid BTC block although it will be rejected by the NMC daemon.  The default behaviour is if it's not the current block (block in psj is now defined as a pseudoblocknum which is sum of all chain's blocknums) reject it as stale.  This may have to change but I need some consensus from pool ops on the what the correct behaviour should be.  I'm not going to code every option under the sun coz I'm little bit sick of merged mining TBH.

4/ The code for detecting and switching to new blocks is still pretty raw.  It's possible the 'fireBlockChange' sequence is happening more often that it should which would put valid shares into the stale list.  I don't know if this is the case but the solution 3/ contains the fix for this issue if it exists...

thx 4 info.


mm works really smooth so far, unfortunately the stale rate has dramatically increased from 0.3% to 3.3%. Do you only deem shares as valid when BTC + NMC results are up2date results? Only main chain would be enough imo if true.

Tenebrix, Fairbrix, and Litecoin are the only ones as far as I know.

sorry, but what is ScryptBrix? Google does not enlighten me

If I'd realised how little had to be changed I would have done this weeks ago but that's how it goes...

The most recent version of psj poolserverj-0.3.0.07.MM.tar.gz now has Scrypt support.  You'll have to remove all the merged mining config options from the properties file and set POWalgorithm=scrypt

Fair warning, this is completely untested as I don't have a scryptBrix/coin client.

nope I just pasted everything to /bin derp. Working now.

You're missing a class that is in the 'core' snapshot file.  Did you copy the snapshot files into lib/lib_non-maven?

Its not quite working for me but I got a whole slew of errors - http://pastebin.com/eXtfLMRx

MM edition .06 is now on the downloads page.  This is first version of MM that I'm reasonably happy with.

It should now be sending longpolls if either block changes and the server is now actively monitoring the state of all block chains..  Issues with high CPU load caused by PSJ rejecting valid work should also be resolved.

If you are going to try it I'd appreciate if you could set the following:
debug=true
trace=true
traceTargets=blockmon

If anything odd happens this should give me some useful info to play with.

well it works I found 2 namecoin blocks.

Is this namecoind or bitcoind? I ask because the vinced merged mining bitcoind crashes on me like hourly.

yay, I'll give it a test.

ok posted as .05.  You'll need the full .02 package then overlay the files in .05

No I haven't posted it up yet.  Give me a bit. I'll designate it .05 so look out for that one

So http://poolserverj.org/dist/mm-mini-binary.04.tar.gz is the magic peace of software?

Great work!

should have checked before I posted.... it works!