Pages:
Author

Topic: [ANN] Eloipool - FAST Python3 pool server software - GBT/stratum/dyntarget/proxy - page 15. (Read 123152 times)

mrb
legendary
Activity: 1512
Merit: 1028
I confirm the bug. One of my tests shows:

Hash computed by hashtest2():
0000000039d5e400932fef0b1d7cdec1f2833dec594e30ee03252113f03be9b4

Hash computed by regeneratehash():
0000000000e4d5390bef2f93c1de7c1dec3d83f2ee304e5913212503b4e93bf0

regeneratehash() is the one that does not compute it correctly.

But there is a 2nd endianness bug in share_diff() which cancels the bug in regeneratehash (which is why the "best share" is always updated correctly when a new share is found)! This 2nd endianness bug in share_diff() is what causes TNR_HIGH shares to sometimes spuriously increase the "best share" statistic without a corresponding share submitted to the pool.

I have a patch fixing both endianness bugs. I am currently testing it...
legendary
Activity: 1596
Merit: 1100
Yeah that sounds like a bug everybody inherited from cpuminer, which definitely had rough edges when it came to any target besides the standard difficulty-1.

mrb
legendary
Activity: 1512
Merit: 1028
I am reassured the bug seems to be in bfgminer. I may take a stab at a fix this weekend.
legendary
Activity: 2576
Merit: 1186
Upon further investigation, I am 99% certain we're looking at a BFGMiner bug here. There are two functions which both recalculate the share/block hash and compare it to a target: fulltest (util.c) and regeneratehash (miner.c). It seems that at least on little-endian, these two functions produce opposite endians for each 32-bit chunk of the hash; in cases where the target is presumably not met, the shares are fed through the share_diff function (responsible for updating the best block) with the opposite endian than it expects. This causes your best block to report something erroneous (or at least it seems this is where the error is). What needs to be done now, is to audit the endian handling in these two functions to determine which one is correct and ideally unify the code into a single function. Hopefully I'll get to that within the next week, but if you want to take a stab at it, feel free... Wink
mrb
legendary
Activity: 1512
Merit: 1028
Well the highest share I found in share-logfile is 2.3M which was the value displayed by bfgminer yesterday. But then again, the code path between the moment receiveShare() is called and when it is logged in share-logfile, is quite complex and it is possible to imagine a bug that would cause the shared to be received, but not logged.

I am surprised about these reports of "Best share" possibly displaying incorrect information. I would think this is a dead simple feature to implement, so why would this be buggy?
legendary
Activity: 2576
Merit: 1186
By the way, what is the use of GotWorkURI? Shares found from getmemorypool work cannot be submitted via getwork, or can they?
GotWorkURI is for merged mining. gotwork != getwork
mrb
legendary
Activity: 1512
Merit: 1028
By the way, what is the use of GotWorkURI? Shares found from getmemorypool work cannot be submitted via getwork, or can they?
mrb
legendary
Activity: 1512
Merit: 1028
I was not running bfgminer with --debuglog.

I restarted eloipool to change some of its config, so it is too late to look at the RBFs variables.

The only trace of data I have on eloipool is the share-logfile. But I need to write some custom code to re-hash the share data there, to find out if the share hash with a 19.5M difficulty is in there. That's what I am doing now...
legendary
Activity: 2576
Merit: 1186
I solved a block today. I know it because bfgminer currently shows "Best share: 19.5M". However eloipool silently ignored the solved block. There goes 25 BTC...
I doubt Eloipool was the problem in this case. The "best share" code is much newer, and there have been reports of similar behaviour from altcoin solo mining; unfortunately, I don't know any good way to reproduce it. Did you happen to be running BFGMiner with a debug log by any chance? There might be some useful info in there...

Furthermore, it is almost impossible to do a post-mortem analysis. I found out that checkShare() in eloipool.py has buggy logging code, so pretty much no useful information has been logged. Indeed, the checkShare.logger object has a default level of 'WARNING', so logfunc() below will never log INFO and DEBUG messages:
Eloipool's code only configures loggers for DEBUG or INFO... For post-mortem analysis on the pool side, check out the RBFs variable (only kept in memory!) and bitcoind's debug.log. Of course, that's assuming the share ever got to Eloipool, which I suspect isn't the case since you apparently didn't see the info logging from it.
mrb
legendary
Activity: 1512
Merit: 1028
I have been solo mining against an eloipool instance (revision 475de39) for the last few days, using bfgminer 2.9.3 and 'getwork'. eloipool is configured with a ShareTarget set to 0x0000000000fff... to emulate difficulty 256. I wanted to give this setup a shot before trying the newer Stratum protocol. I thought it would be a robust way to mine, using mature code paths (both at the bfgminer side, and at the eloipool side).

I was wrong.

I solved a block today. I know it because bfgminer currently shows "Best share: 19.5M". However eloipool silently ignored the solved block. There goes 25 BTC...

Furthermore, it is almost impossible to do a post-mortem analysis. I found out that checkShare() in eloipool.py has buggy logging code, so pretty much no useful information has been logged. Indeed, the checkShare.logger object has a default level of 'WARNING', so logfunc() below will never log INFO and DEBUG messages:
Code:
        logfunc = getattr(checkShare.logger, 'info' if blkhashn <= networkTarget else 'debug')
        logfunc('BLKHASH: %64x' % (blkhashn,))
        logfunc(' TARGET: %64x' % (networkTarget,))

Either eloipool silently ignored the block due to an internal bug. Or it crafted an invalid block which was sent to my bitcoind, who in turns ignored it because it was invalid. I am quite disappointed.

In case you care to investigate, Luke, here is my config.py:

Code:

### Settings relating to server identity

# Name of the server
ServerName = 'mrb eloipool'

### Settings relating to server scaling/load

# Share hashes must be below this to be valid shares
# If dynamic targetting is enabled, this is a minimum
ShareTarget = 0x0000000000ffffffffffffffffffffffffffffffffffffffffffffffffffffff

# Automatically adjust targets per username
# 0 = disabled
# 1 = arbitrary targets
# 2 = power of two difficulties (zero bit counts)
DynamicTargetting = 0

# How many shares per minute to try to achieve on average
DynamicTargetGoal = 8

# Number of seconds hashrate is measured over
DynamicTargetWindow = 120

# Minimum and maximum of merkle roots to keep queued
WorkQueueSizeRegular = (0x100, 0x1000)

# Minimum and maximum of BLANK merkle roots to keep queued
# (used if we run out of populated ones)
WorkQueueSizeClear = (0x1000, 0x2000)

# Minimum and maximum of BLANK merkle roots to keep queued, one height up
# (used for longpolls)
WorkQueueSizeLongpoll = (0x1000, 0x2000)

# How long to wait between getmemorypool updates normally
MinimumTxnUpdateWait = 5

# How long to wait between retries if getmemorypool fails
TxnUpdateRetryWait = 1

# How long to sleep in idle loops (temporary!)
IdleSleepTime = 0.1

### Settings relating to reward generation

# Address to generate rewards to
TrackerAddr = '1xxx'

# Coinbaser command to control reward delegation
# NOTE: This example donates 1% of block rewards to Luke-Jr for Eloipool development
#CoinbaserCmd = 'echo -e "1\\n$((%d / 100))\\n1579aXhdwvKZEMrAKoCZhzGuqMa8EonuXU"'

### Settings relating to upstream data providers

# JSON-RPC server for getmemorypool
UpstreamURI = 'http://:@localhost:8332'

# Set to True if you want shares meeting the upstream target to wait for a
# response from the upstream server before logging them. Otherwise, for such
# shares, upstreamResult will always be True and upstreamRejectReason will
# always be None. Note that enabling this may cause shares to be logged out of
# order, or with the wrong timestamp (if your share logger uses the log-time
# rather than share-time).
DelayLogForUpstream = False

# Bitcoin p2p server for announcing blocks found
UpstreamBitcoindNode = ('127.0.0.1', 8333)

# Network ID for the primary blockchain
#UpstreamNetworkId = b'\xFA\xBF\xB5\xDA'  # testnet
UpstreamNetworkId = b'\xF9\xBE\xB4\xD9'  # mainnet

# Secret username allowed to use setworkaux
#SecretUser = ""

# URI to send gotwork with info for every share submission
#GotWorkURI = ''

# Share hashes must be below this to be submitted to gotwork
GotWorkTarget = 0x0000000000ffffffffffffffffffffffffffffffffffffffffffffffffffffff

# Aim to produce blocks with transaction counts that are a power of two
# This helps avoid any chance of someone abusing CVE-2012-2459 with them
# 1 = cut out feeless transactions; 2 = cut out even fee-included transactions (if possible)
POT = 2

# Avoid mining feeless transactions except to satisfy POT
# Note this only works if POT is in fact enabled in the first place
Greedy = False

### Settings relating to network services

# Addresses to listen on for JSON-RPC getwork server
# Note that Eloipool only supports IPv6 sockets, and if you want to bind to an
# IPv4 address you will need to prepend it with ::ffff: eg ::ffff:192.168.1.2
JSONRPCAddresses = (
('', 28332),
)

# Addresses to listen on for Bitcoin node
# Note that Eloipool only supports IPv6 sockets, and if you want to bind to an
# IPv4 address you will need to prepend it with ::ffff: eg ::ffff:192.168.1.2
BitcoinNodeAddresses = (
('', 28333),
)

# Addresses that are allowed to "spoof" from address with the X-Forwarded-For header
TrustedForwarders = ('::ffff:127.0.0.1',)


# Logging of shares:
ShareLogging = (
{
'type': 'logfile',
'filename': 'share-logfile',
'format': "{time} {Q(remoteHost)} {username} {YN(not(rejectReason))} {dash(YN(upstreamResult))} {dash(rejectReason)} {solution}\n",
},
)
legendary
Activity: 2576
Merit: 1186
Hey,

I am not the best in git and linux, can you tell me, how to revert the last commit?
I have just clone the git from github, but it is very old I think.
Eloipool is in Gitorious, not GitHub...

To simply pull the top commit off:
Code:
git reset --hard HEAD^
full member
Activity: 222
Merit: 100
Hey,

I am not the best in git and linux, can you tell me, how to revert the last commit?
I have just clone the git from github, but it is very old I think.
legendary
Activity: 1064
Merit: 1000
TAiS46,

I have it running, and with a couple of Alt coins.

You need to completely revert the last commit, That involved taking a section out and putting code back in.

I also disabled some of the more extraneous options in the config file and am running it as a basic GBT pool. The GBT long-polls are right on time, and I haven't see any of my rigs complain yet.

Luke,

I have to admit Luke, I am impressed. After all the crappy and abandoned code that usually involved ancient oddball dependencies I have have spent way to much time on over last few months. Eloipool is a breath of fresh air.

I only lost half my hair going through this code.  Grin

Anyway, Hope you are feeling better and I look forward to further development on Eloipool. I think my next project will be a pool and it looks like it will be based on Eloipool.
full member
Activity: 222
Merit: 100
no fix and this moment, does not work Sad
full member
Activity: 222
Merit: 100
Ok, keep me updated.
Maybe I will use your software for a bitcoin-24.com pool Wink
get well soon
legendary
Activity: 2576
Merit: 1186
Are the Transaction-Longpoll warnings also a failure from you?
I search the whole time for the fix Cheesy
No, that was (if it's what I think you're referring to) a bugfix; it wasn't actually doing what it needed to be doing.
full member
Activity: 222
Merit: 100
Oh ok just to let you know what I changed:

Code:
        def _makeOne(self, putf, merkleTree, checkBlock, height):
                MT = self.currentMerkleTree
                myblock = self.currentBlock
                MR = self.makeMerkleRoot(MT, height=height)
                if checkBlock:
                        # Only add it if the block hasn't changed in the meantime, to avoid a race
                        if self.currentBlock != myblock:
                                return
                else:
                        # Only add it if the height hasn't changed in the meantime, to avoid a race
                        if self.currentBlock[1] != height:
                                return
                putf(MR)

        def makeClear(self):
                self._doing('clear merkle roots')
                self._makeOne(self.clearMerkleRoots.put, self.curClearMerkleTree, False, height=self.currentBlock[1])

        def makeNext(self):
                self._doing('longpoll merkle roots')
                self._makeOne(self.nextMerkleRoots.put, self.nextMerkleTree, False, height=self.currentBlock[1] + 1)

        def makeRegular(self):
                self._doing('regular merkle roots')
                self._makeOne(self.merkleRoots.append, self.currentMerkleTree, True, height=self.currentBlock[1])

switches the args: height=self.currentBlock[1], False


Are the Transaction-Longpoll warnings also a failure from you?
I search the whole time for the fix Cheesy
legendary
Activity: 2576
Merit: 1186
Yeah, that's what happens when I try to fix a rare edge race condition when I have a fever over 39 C/103 F. Sad

Just ignore that top commit for now and I'll fix it when I get better
full member
Activity: 222
Merit: 100
Help with
Code:
# Network ID for the primary blockchain
UpstreamNetworkId = b'\xF9\xBE\xB4\xD9'  # testnet
is that correct?

and another failure
Code:
Traceback (most recent call last):
  File "eloipool.py", line 149, in
    from merklemaker import merkleMaker
  File "/tmp/eloipool/merklemaker.py", line 448
    self._makeOne(self.clearMerkleRoots.put, self.curClearMerkleTree, height=self.currentBlock[1], False)
SyntaxError: non-keyword arg after keyword arg

Edit: Fixed the error, changed some thing in the File.

Next error:
Code:
2012-11-29 12:44:43,877 merkleMaker     WARNING Transaction-longpoll requested 100 seconds ago, and still not ready. Is your server fast enough to keep up with your configured WorkQueueSizeRegular maximum? (doing longpoll merkle roots)
2012-11-29 12:44:43,878 merkleMaker     WARNING Haven't updated the merkle tree in at least 100 seconds! Is your server fast enough to keep up with your configured work queue minimums? (doing longpoll merkle roots)
legendary
Activity: 2576
Merit: 1186
Been a while since I first announced Eloipool, so I'm updating the first post with new features Smiley

Most notable is support for dynamic share targetting in various modes (fractional, power-of-two (zero bit count), and bdiff rounded).
Combined with GBT, this is what pools need to support ASICs.
Additionally, Eloipool now supports the (pre-BIP draft) stratum mining protocol.
These new features were sponsored by Inaba/EclipseMC.

Also, a little background on the "loop" branch that's been sitting in the repository for a while:
This branch is designed to increase Eloipool's support of the GBT protocol on the upstream side.
Basically, you can run Eloipool as a "sub-pool" to any other GBT-enabled pool.
As of tonight, it is up to date with the final GBT protocol specifications, and works somewhat with both Eligius and BitMinter upstreams.
The main problems left to address are upstream longpolling support and properly dealing with upstreams when sending out the initial new block notifications (currently these produce and accept invalid work).
If you'd like to try this branch out, create a fresh clone of Eloipool and use git checkout loop[/url] to get the loop branch.
Customize your configuration as follows:
  • CoinbasePrefix must be reasonably long to avoid collisions with upstream coinbase data.
  • WorkQueueSizeLongpoll must be (0, 0) for now.
  • It is a good idea to enable DelayLogForUpstream and use {upstreamResult} in your share log.
  • Set ExpectedUpstreamLatency to a number of seconds of expected delay between your pool server and the upstream server.
I've been developing this on my own spare time. Donations, bounties, or outright hiring me to improve it is welcome.
General "loop"-related donations can be sent to 14VUob11S77JmrnShEKXPWttg8s6FvW12W.
Pages:
Jump to: