bitHopper: Python Pool Hopper Proxy - page 203.

burp

member

Activity: 98

Merit: 10

Here is how to make twisted less noisy:

http://stackoverflow.com/questions/5078980/twisted-starting-stopping-factory-protocol-less-noisy-log-messages

EDIT: I just noticed you fixed a couple of things, nice

c00w

full member

Activity: 196

Merit: 100

1) Server side LP is broken?
Yeah. It is. I'm disabling it until I can figure out why twisted is ending the calls early.

2) Screwy logic with regards to switching?
Yeah. I missed that. I'm switching it over.

3) Better location of pool update function?
Well thats the wrong location because it gets called before we get new information. Where it is now if the logic wasn't broken it would work a lot better.

4) How often does bitHopper check pool API's?
EDIT: Changed to every 117 seconds.

5) BTCguild is broken?
I was waiting for their api to come up so that I could add the update function.

OCedHrt

member

Activity: 111

Merit: 10

Quote from: ahitman on July 09, 2011, 11:24:09 AM

I think your right, should it be

Code:

if servers[current_server]['shares'] - min_shares > .50 * servers[current_server]['shares']:

?

I think I would prefer my miners to switch as soon as there is anything lower, to maximize effectiveness. Unless the switching lowers your hasrate too much, but I dont see how it would.

Actually, I changed it to:

Code:

            if servers[current_server]['shares'] / 2 > min_shares:

Which really achieves the same thing as using >. But this means it only switches if a pool has less than half the shares of the current pool. Though I think to prevent pool hopping backing back and forth you only need to require a 1-5% difference in shares.

The bigger issue I'm seeing is that it's doing an API stats pull every share accepted and that's going to raise some eyes from the pool operator side. I'm not sure if server side LP is even working.

ahitman

sr. member

Activity: 302

Merit: 250

Quote from: OCedHrt on July 09, 2011, 11:13:20 AM

Quote from: ahitman on July 09, 2011, 10:24:38 AM

Quote from: OCedHrt on July 09, 2011, 09:16:03 AM

And, what's the reasoning for waiting until the server with the least shares to have more than half of the current server before switching?

Code:

        if min_shares < servers[current_server]['shares']:
            if servers[current_server]['shares'] - min_shares < .50 * servers[current_server]['shares']:
                select_best_server()
                return

Check out http://forum.bitcoin.org/index.php?topic=26866.msg342833#msg342833 its to avoid hopping back and fourth between two pools.
How much overhead is there when hopping between pools? Would it be limited to how many work units a miner had queued up? Or is there some other losses Im missing?

No I don't think so. As written it wouldn't do that. Rather it actually prevents a switch until the lowest share server has at least 50% of the current server, not the difficulty. The other problem is that it's doing an update_server every share submit, not just on each LP.

I think your right, should it be

Code:

if servers[current_server]['shares'] - min_shares > .50 * servers[current_server]['shares']:

?

I think I would prefer my miners to switch as soon as there is anything lower, to maximize effectiveness. Unless the switching lowers your hasrate too much, but I dont see how it would.

OCedHrt

member

Activity: 111

Merit: 10

Quote from: ahitman on July 09, 2011, 10:24:38 AM

Quote from: OCedHrt on July 09, 2011, 09:16:03 AM

And, what's the reasoning for waiting until the server with the least shares to have more than half of the current server before switching?

Code:

        if min_shares < servers[current_server]['shares']:
            if servers[current_server]['shares'] - min_shares < .50 * servers[current_server]['shares']:
                select_best_server()
                return

Check out http://forum.bitcoin.org/index.php?topic=26866.msg342833#msg342833 its to avoid hopping back and fourth between two pools.
How much overhead is there when hopping between pools? Would it be limited to how many work units a miner had queued up? Or is there some other losses Im missing?

No I don't think so. As written it wouldn't do that. Rather it actually prevents a switch until the lowest share server has at least 50% of the current server, not the difficulty. The other problem is that it's doing an update_server every share submit, not just on each LP.

ahitman

sr. member

Activity: 302

Merit: 250

Sorry to take spam this thread with so many posts but I found this useful and maybe someone else will as well.

Quote from: organofcorti on July 09, 2011, 06:39:45 AM

Just using new bithopper - how can you tell which pool you're connected to?

I added a line to my pool.py file that will display the chosen server name with **** around it so one can pick it out of the torrent of information flowwing through the log file. Please let me know if there is a better spot for this as Im not good at python (yet) so I just stuck it in to see what happens.

Code:

    for server in servers:
        info = servers[server]
        if info['shares']< min_shares and info['lag'] == False:
            log.msg( '***min_shares :' + str(min_shares))
            min_shares = servers[server]['shares']
            server_name = server

    if server_name == None:
        server_name = 'eligious'

    """This line added to show current server in a more visible way"""
    log.msg( '******New server **** :' + str(server_name))

    global new_server
    global lp_set

ahitman

sr. member

Activity: 302

Merit: 250

Here is some logic that popped in my head while looking through the code (I've never programmed in python but I have done some programming at school). Please let me know if this makes sense:

1. Check the api of each pool for share_count
2. If share_count[of the pool you are checking] < 1 then share_count[of the pool you are checking] = 10**10
3. Pick the pool with the lowest share count

Step 2 will catch any misbehaving api or a pool that is down and is returning a 0 for the share count.

Let me know if the code is doing this or something like this already, I have a hard time following some of the if statements, but you've inspired me to go and learn some python syntax!

ahitman

sr. member

Activity: 302

Merit: 250

Quote from: OCedHrt on July 09, 2011, 09:16:03 AM

And, what's the reasoning for waiting until the server with the least shares to have more than half of the current server before switching?

Code:

        if min_shares < servers[current_server]['shares']:
            if servers[current_server]['shares'] - min_shares < .50 * servers[current_server]['shares']:
                select_best_server()
                return

Check out http://forum.bitcoin.org/index.php?topic=26866.msg342833#msg342833 its to avoid hopping back and fourth between two pools.
How much overhead is there when hopping between pools? Would it be limited to how many work units a miner had queued up? Or is there some other losses Im missing?

Quote from: organofcorti on July 08, 2011, 09:57:13 PM

Quote

Well I almost have Long polling support enabled on the server side so it'll switch everytime a block is found.

looking forward to LP support, but how will you know where to switch from LP? Won't it just tell you that a block was found somewhere in the network? I suppose you could use that info to determine when bithopper checks to see which pool has the least shares to see if one of them was the one that found the block? Or maybe I'm just misunderstanding LP.

Again, thanks for such a simple and easy to use/modify proxy.

I think you understand correctly, and this is a good idea in theory, but if you are using the api of the pool to figure out which pool it is and it takes a while for some pools to update the stats, then your probably best off staying in the current pool until bithopper picks up a low block count from the winning pool through its regular checks, and switches anyways.

How often does bithopper check each pools apis?

OCedHrt

member

Activity: 111

Merit: 10

Quote

[7/9/11 6:11:07 AM] Added Barts (#1) (14 CU, local work size of 128)
Waiting...Exception in thread "DiabloMiner LongPollAsync" java.lang.NullPointerE
Waiting...
at com.diablominer.DiabloMiner.DiabloMiner.forceUpdate(DiabloMiner.java:
Waiting...
at com.diablominer.DiabloMiner.DiabloMiner$LongPollAsync.run(DiabloMiner
Waiting...)
Waiting... java.lang.Thread.run(Unknown Source)

I get this exception on the DiabloMiner side when connecting to BitHopper. But it still seems to run after that.

Also, it seems btcguild doesn't have a get shares function? The user api json is broken when there are no workers, but luckily the site-wide api is functioning. And, server_update() should not be called in each get response actually, and should be done at the end of update_servers because each server_update does a comparison against the rest of the server and calls select_best_server() - logging is also much quieter this way Tongue

Code:

def btcguild_sharesResponse(response):
    global servers
    info = json.loads(response)
    round_shares = int(info['round_shares'])
    servers['btcg']['shares'] = round_shares
    log.msg( 'btcguild :' + str(round_shares))

def btcguild_getshares():
    d = getPage('https://www.btcguild.com/pool_stats.php')
    d.addCallback(btcguild_sharesResponse)
    d.addErrback(log.err)

def update_servers():
    global servers
    bitclockers_getshares()
    mineco_getshares()
    btcguild_getshares()

    server_update()

And, what's the reasoning for waiting until the server with the least shares to have more than half of the current server before switching?

Code:

        if min_shares < servers[current_server]['shares']:
            if servers[current_server]['shares'] - min_shares < .50 * servers[current_server]['shares']:
                select_best_server()
                return

burp

member

Activity: 98

Merit: 10

Are the pool stats updated after each new share now, or am I misinterpreting the log spam? At least when I run the client without long-polling, or disable the long-polling header at bitHopper.

c00w

full member

Activity: 196

Merit: 100

Hmmm. I see the problem and It has something to do with cgminer managing to call bitHopper quickly enough to make it make two LP connections one of which gets returned. I'll figure out why my code to stop this doesn't work tommorow. For now use poclbm or phoenix.

burp

member

Activity: 98

Merit: 10

Quote from: c00w on July 09, 2011, 06:45:35 AM

2) Burf:
Its going to sound horrible but can you run git pull again? I just rewrote the client LP system to have a lot less bugs. It appears that miners implement LP in all sorts of ways because its actually not clearly specified anywhere.

Also what miner are you using? That might have a disturbingly significant effect. The logs look like nothing is wrong.

Nope, no effect. I'm running cgminer (http://forum.bitcoin.org/index.php?topic=21275.0).

organofcorti

donator

Activity: 2058

Merit: 1007

Poor impulse control.

Quote from: c00w on July 09, 2011, 06:45:35 AM

1) How can you tell what pool you are connected to?
It should say rpc request [] from blah.blah.
That blah.blah is the server it is pulling from.

Sorry, I can only see an RPC request to mtred which hasn't yet recorded any shares from me. SHould I just dl and try the lastest?

Code:

011-07-09 10:47:37+0000 [HTTP11ClientProtocol,client] 127.0.0.1 - - [09/Jul/2011:10:47:36 +0000] "POST / HTTP/1.1" 200 608 "-" "poclbm/2011.beta4"
2011-07-09 10:47:38+0000 [HTTP11ClientProtocol,client] LP triggered 
2011-07-09 10:47:38+0000 [HTTP11ClientProtocol,client] Starting factory 
2011-07-09 10:47:38+0000 [HTTP11ClientProtocol,client] Starting factory 
2011-07-09 10:47:38+0000 [HTTP11ClientProtocol,client] Starting factory 
2011-07-09 10:47:38+0000 [HTTP11ClientProtocol,client] Starting factory 
2011-07-09 10:47:38+0000 [HTTPChannel,1,127.0.0.1] RPC request [] From mtred
2011-07-09 10:47:38+0000 [HTTPChannel,6,127.0.0.1] RPC request [] From mtred
2011-07-09 10:47:38+0000 [HTTPPageGetter,client] mtred :0
2011-07-09 10:47:38+0000 [HTTPPageGetter,client] Stopping factory 
2011-07-09 10:47:39+0000 [HTTP11ClientProtocol,client] LP Call mtred.com:8337/LP
2011-07-09 10:47:39+0000 [HTTP11ClientProtocol,client] 127.0.0.1 - - [09/Jul/2011:10:47:38 +0000] "POST / HTTP/1.1" 200 608 "-" "poclbm/2011.beta4"
2011-07-09 10:47:39+0000 [HTTP11ClientProtocol,client] LP Call mtred.com:8337/LP
2011-07-09 10:47:39+0000 [HTTP11ClientProtocol,client] 127.0.0.1 - - [09/Jul/2011:10:47:38 +0000] "POST / HTTP/1.1" 200 608 "-" "poclbm/2011.beta4"
2011-07-09 10:47:39+0000 [HTTPPageGetter,client] mineco :1415190
2011-07-09 10:47:39+0000 [HTTPPageGetter,client] Stopping factory 
2011-07-09 10:47:39+0000 [HTTPPageGetter,client] bitclockers :1087070
2011-07-09 10:47:39+0000 [HTTPPageGetter,client] Stopping factory 
2011-07-09 10:47:39+0000 [HTTP11ClientProtocol,client] LP triggered 
2011-07-09 10:47:39+0000 [HTTP11ClientProtocol,client] Starting factory 
2011-07-09 10:47:39+0000 [HTTP11ClientProtocol,client] Starting factory 
2011-07-09 10:47:39+0000 [HTTP11ClientProtocol,client] Starting factory 
2011-07-09 10:47:39+0000 [HTTP11ClientProtocol,client] Starting factory 
2011-07-09 10:47:39+0000 [HTTP11ClientProtocol,client] LP triggered 
2011-07-09 10:47:39+0000 [HTTP11ClientProtocol,client] Starting factory 
2011-07-09 10:47:39+0000 [HTTP11ClientProtocol,client] Starting factory 
2011-07-09 10:47:39+0000 [HTTP11ClientProtocol,client] Starting factory 
2011-07-09 10:47:39+0000 [HTTP11ClientProtocol,client] Starting factory 
2011-07-09 10:47:39+0000 [HTTPPageGetter,client] bitcoin.lc :1485982
2011-07-09 10:47:39+0000 [HTTPPageGetter,client] Stopping factory 
2011-07-09 10:47:40+0000 [HTTPPageGetter,client] mtred :0
2011-07-09 10:47:40+0000 [HTTPPageGetter,client] Stopping factory 
2011-07-09 10:47:40+0000 [HTTPPageGetter,client] mtred :0
2011-07-09 10:47:40+0000 [HTTPPageGetter,client] Stopping factory 
2011-07-09 10:47:40+0000 [HTTPPageGetter,client] bitclockers :1087134
2011-07-09 10:47:40+0000 [HTTPPageGetter,client] Stopping factory 
2011-07-09 10:47:41+0000 [HTTPPageGetter,client] bitclockers :1087143
2011-07-09 10:47:41+0000 [HTTPPageGetter,client] Stopping factory 
2011-07-09 10:47:41+0000 [HTTPPageGetter,client] bitcoin.lc :1485982
2011-07-09 10:47:41+0000 [HTTPPageGetter,client] Stopping factory 
2011-07-09 10:47:41+0000 [HTTPPageGetter,client] bitcoin.lc :1485982
2011-07-09 10:47:41+0000 [HTTPPageGetter,client] Stopping factory 
2011-07-09 10:47:41+0000 [HTTPChannel,1,127.0.0.1] RPC request [u'00000001cce6f8a2cf82f32896d9cd75aaa1e86d626c29045094b6bd0000097e00000000ab84c24b1c41ebde8fef201eb19ba0b460acc681849e6d21d21e2a47892e05eb4e1831c81a0abbcfb83a58d6000000800000000000000000000000000000000000000000000000000000000000000000000000000000000080020000'] From mtred

c00w

full member

Activity: 196

Merit: 100

1) How can you tell what pool you are connected to?
It should say rpc request [] from blah.blah.
That blah.blah is the server it is pulling from.

2) Burf:
Its going to sound horrible but can you run git pull again? I just rewrote the client LP system to have a lot less bugs. It appears that miners implement LP in all sorts of ways because its actually not clearly specified anywhere.

Also what miner are you using? That might have a disturbingly significant effect. The logs look like nothing is wrong.

organofcorti

donator

Activity: 2058

Merit: 1007

Poor impulse control.

Just using new bithopper - how can you tell which pool you're connected to?

burp

member

Activity: 98

Merit: 10

Doesn't make a difference, here is the complete logfile: http://pastebin.com/1iY1pS3M

PS: It's no internet connectivity problem

c00w

full member

Activity: 196

Merit: 100

1) Stopping and starting messsages are annoying. Help?
Currently I started using the twisted logging system because it made finding bugs easier with the timestamps. The twisted logging system has no simple way that I can find to stop those messages from appearing. It hasn't gotten a lot of love. I'm planning on adding an actual debug option and using print otherwise for purely informational messages.

If you want to write me a subclass of httpclient with those messages turned off though I'll happily accept it.

2) Server side Long Polling is working?
It does now. I had an issue with longpoll handing because I inverted a logic statement. Its pushed to the repo in the correct form and running on my servers.

3)What happened to Burps machine?
From the log it looks like the server started cycling endlessly because there was no internet connection? I'm guessing my changes making the server swap if another server has less than 25% of its shares is at fault since if there are multiple 0 share connections it would try each, lag them out, issue a LP to the client because it swapped server, try the next and rinse and repeat through the whole list. Which certainly looks like what happened. I'll redo that code to instead first check that the share > 10% of difficulty.

ahitman

sr. member

Activity: 302

Merit: 250

This thing is great! I would like to be able to see better what its doing, is there any way to hide the Starting factory and Stopping factory messages?

burp

member

Activity: 98

Merit: 10

Does the current version work?

[2011-07-09 11:00:46] LONGPOLL detected new block, flushing work queue
[2011-07-09 11:00:46] LONGPOLL detected new block, flushing work queue
[2011-07-09 11:00:47] LONGPOLL detected new block, flushing work queue
[2011-07-09 11:00:48] LONGPOLL detected new block, flushing work queue

It doesn't seem to choose a pool.

c00w

full member

Activity: 196

Merit: 100

1) How/why does LP switching help?
Well LP means a server found a block. And it may be one of ours. So we run a server switch to the server with the lowest share-count because it may be a new round and either way all the current get-works are invalid so the overhead is minor.

Topic: bitHopper: Python Pool Hopper Proxy - page 203. (Read 355689 times)