Topic: bitHopper: Python Pool Hopper Proxy - page 163. (Read 355816 times)
They changed the format again, new code for triple:
I don't think this can be done with the new config, so I am still at yesterday's version.
Code:
def triple_sharesResponse(self, response):
output = re.search(r"-->([0-9\s]+) if output != None:
match = output.group(1)
match = match.replace(' ','')
round_shares = int(match)
self.UpdateShares('triple',round_shares)
I don't think this can be done with the new config, so I am still at yesterday's version.
Anyone else have triplemining broken in the new bitHopper? Getting 'error in api'.
Also I love polmine, great service and good efficiency so far. But the new config broke the regex that ryouiki did. Anyone fix it? The old version (with what I assume is a commented out practice go) is:
worked fine, had some error at the start of bitHopper which I paid no attention to but worked.
So can anyone fix Triplemining and Polmine for me?
Cheers!
Also I love polmine, great service and good efficiency so far. But the new config broke the regex that ryouiki did. Anyone fix it? The old version (with what I assume is a commented out practice go) is:
Code:
'polmine':{'shares':default_shares, 'name':'polmine',
'mine_address':'polmine.pl:8347', 'user':polmine_user, 'pass':polmine_pass,
'lag':False, 'LP':None,'api_address':'http://polmine.pl/?action=statistics',
'role':'mine' },
def polmine_sharesResponse(self, response):
#info = re.search(r"stkich: [0-9]* ", response).group(0)[4:-5]
round_shares = int(re.search(r"stkich:[ 0-9]* ", response).group(0)[16:-5].replace(' ',''))
self.UpdateShares('polmine',round_shares)
So can anyone fix Triplemining and Polmine for me?
Cheers!
very nice idea!
what about just placing a command line in pools.conf which gets executed every time round-shares are updated?
that way we could completly decouple bithopper from pools - and even change them without an restart.
shell command could return sth like "poolhashrate,roundshares,false|true" (false|true: is the api lagging)
which could easily be parsed by python
what about just placing a command line in pools.conf which gets executed every time round-shares are updated?
that way we could completly decouple bithopper from pools - and even change them without an restart.
shell command could return sth like "poolhashrate,roundshares,false|true" (false|true: is the api lagging)
which could easily be parsed by python
My changes:
I don't think regex api_index is necessary if you write proper regular expressions to match, as in my example.
I also added api_method cmd which runs a command in the shell where you can use arbitrary pipe, curl,grep,sed,awk magic.
api_method:cmd
api_cmd:curl -s -k https://pool.bitp.it/leaderboard | … more shell magic that returns a number
api_method:cmd
api_cmd:echo 1234
I don't think regex api_index is necessary if you write proper regular expressions to match, as in my example.
I also added api_method cmd which runs a command in the shell where you can use arbitrary pipe, curl,grep,sed,awk magic.
api_method:cmd
api_cmd:curl -s -k https://pool.bitp.it/leaderboard | … more shell magic that returns a number
api_method:cmd
api_cmd:echo 1234
Code:
diff -rN -u old-bitHopper_darcs/pool.cfg.default new-bitHopper_darcs/pool.cfg.default
--- old-bitHopper_darcs/pool.cfg.default 2011-07-23 13:34:45.675261168 +0200
+++ new-bitHopper_darcs/pool.cfg.default 2011-07-23 13:34:45.679261217 +0200
@@ -51,7 +51,7 @@
mine_address: eu1.triplemining.com:8344
api_address:https://www.triplemining.com/stats
api_method:re
-api_key:-->\d+
+api_key:.*?(\d+)
api_index:3,-5
role:mine
#CHANGE THIS
diff -rN -u old-bitHopper_darcs/pool.py new-bitHopper_darcs/pool.py
--- old-bitHopper_darcs/pool.py 2011-07-23 13:34:45.675261168 +0200
+++ new-bitHopper_darcs/pool.py 2011-07-23 13:34:45.679261217 +0200
@@ -107,15 +107,15 @@
elif server['api_method'] == 're':
output = re.search(server['api_key'],response)
- s,e = server['api_index'].split(',')
- output = output.group(0)
- s = int(s)
- if e == '0' or e =='':
- output = output[s:]
- else:
- output = output[s:int(e)]
- round_shares = int(output)
+ round_shares = int(output.group(1))
self.UpdateShares(args,round_shares)
+
+ elif server['api_method'] == 'cmd':
+ import subprocess
+ round_shares = int(subprocess.Popen(server['api_cmd'], shell=True,
+ stdout=subprocess.PIPE).communicate()[0])
+ self.UpdateShares(args,round_shares)
+
else:
self.bitHopper.log_msg('Unrecognized api method: ' + str(server))
Wouldnt it be possible to allow the config for "less knowledable" users and leaving the pure python code for those who want to tinker more in depth?
Seems bitHopper loses more and more flexibility. With this config file even more is gone, that is the possibility to write more complicated functionality to get the share counts, which I relied on for some pools I added myself. I will stick to older versions.
It currently has to have the payout entered manually. with a bitcoind interface it could be automatic but I haven't written it yet.
@c00w: How does the efficiency/payout work? Is it something you have to do manually, or is it supposed to be automatic?
Explanation of the new pool system / how to add pools.
Pool.cfg is a config file.
Pools have the format
[internal name]
name: value
mine_address: address:port
role:disable/backup/mine/mineppln(Once I add it)
user: username
pass: Password
If you have a mining pool you have to specify
api_address:address
and
api_method: json/re
For json add
api_key: json_accesor1, json_accesor2, etc...
So if a pool has its roundshares at
{round_shares:1234}
then
api_key:round_share
if it is
{pool:{round_shares:1234}}
then api_key:pool,round_shares
For re (Regular expressions),
api_key is the expression to match.
api_index is how the resulting match should be sliced to get the value.
See triplemining for an example.
If you get a pool working then well put the code in an issue so it can be added.
Or the forums but I keep forgetting...
Pool.cfg is a config file.
Pools have the format
[internal name]
name: value
mine_address: address:port
role:disable/backup/mine/mineppln(Once I add it)
user: username
pass: Password
If you have a mining pool you have to specify
api_address:address
and
api_method: json/re
For json add
api_key: json_accesor1, json_accesor2, etc...
So if a pool has its roundshares at
{round_shares:1234}
then
api_key:round_share
if it is
{pool:{round_shares:1234}}
then api_key:pool,round_shares
For re (Regular expressions),
api_key is the expression to match.
api_index is how the resulting match should be sliced to get the value.
See triplemining for an example.
If you get a pool working then well put the code in an issue so it can be added.
Or the forums but I keep forgetting...
Amazing streblo. Really great stuff.
The thing I'm still unclear on though (and maybe I'm just dense) is:
Based on your analysis, given the situation of hopping two pools having different speeds, should I stay with the faster pool even though it is further from share 1 and then hop to slower later. Or do I hop to pool 2 immediately after it solves a block. It seems to me the downside is getting stuck in a slow pool and missing the spread of shares to other available pools ( even backup)
The thing I'm still unclear on though (and maybe I'm just dense) is:
Based on your analysis, given the situation of hopping two pools having different speeds, should I stay with the faster pool even though it is further from share 1 and then hop to slower later. Or do I hop to pool 2 immediately after it solves a block. It seems to me the downside is getting stuck in a slow pool and missing the spread of shares to other available pools ( even backup)
No.
Quote
. Hashrate, timeslicing (as I understand it), number of shares you vs other other people submitted, etc. ALL DON'T MATTER.
Different hashrate pools only affect variance, not speedup (=efficiency).
Amazing streblo. Really great stuff.
The thing I'm still unclear on though (and maybe I'm just dense) is:
Based on your analysis, given the situation of hopping two pools having different speeds, should I stay with the faster pool even though it is further from share 1 and then hop to slower later. Or do I hop to pool 2 immediately after it solves a block. It seems to me the downside is getting stuck in a slow pool and missing the spread of shares to other available pools ( even backup)
The thing I'm still unclear on though (and maybe I'm just dense) is:
Based on your analysis, given the situation of hopping two pools having different speeds, should I stay with the faster pool even though it is further from share 1 and then hop to slower later. Or do I hop to pool 2 immediately after it solves a block. It seems to me the downside is getting stuck in a slow pool and missing the spread of shares to other available pools ( even backup)
1) Flowers code?
I understand the idea of slicing between pools and will implement it. However I haven't had the time to look at his code and figure it out.
2)New pool config file?
I decided nobody should have to touch python source code. And its easier. For example fixing triple mining required only two lines of code to change.
And I am going to add a mine_ppln mode so we can hop ppln pools.
3) Where were you?
Canada.
4) Adding new pools?
I counted three at last count. Assuming they have json its pretty quick to add them. I'll try and do it tommorow...
edit: Streblo!
I love the explanation. And I'm adding ppln.
I understand the idea of slicing between pools and will implement it. However I haven't had the time to look at his code and figure it out.
2)New pool config file?
I decided nobody should have to touch python source code. And its easier. For example fixing triple mining required only two lines of code to change.
And I am going to add a mine_ppln mode so we can hop ppln pools.
3) Where were you?
Canada.
4) Adding new pools?
I counted three at last count. Assuming they have json its pretty quick to add them. I'll try and do it tommorow...
edit: Streblo!
I love the explanation. And I'm adding ppln.
Very nice work, Streblo! Great explanation, and the first clear explanation of multiple pool hopping that I've seen.
As a big favour, could you show me how you derived the PPLNS function?
<\ttfn>
As a big favour, could you show me how you derived the PPLNS function?
<\ttfn>
I've calculated some statistics for everyone to help clear up some confusion. Poisson processes can be very non-intuitive. These numbers are the result of considering every possibility (and the probability of that possibility). Any situation which you can think of has been accounted for and (hopefully) accounted for. This analysis assumes one has an accurate share count for pools. That is to say, none of these numbers include things like delayed or falsified JSON stats
Proportional pools
Firstly, in a proportional pool the expectation value of a submitted share is dependent on one and only one variable: the number of shares submitted previously. Hashrate, timeslicing (as I understand it), number of shares you vs other other people submitted, etc. ALL DON'T MATTER. In fact, one can calculate the speedup. Define the normalized shares submitted, x, as the number of shares submitted divided by the current difficulty. Then the expectation value (average value) of a share (normalized to solo mining) is exp(x)*E1(x). E1 is a nonanalytic integral, but wolfram alpha will compute it for you. Alternatively, to a decent approximation, one can use speedup = (.428/x)^(0.65). This approximate formula can be handy for quickly calculating expectation value without resorting to numerical methods. The accurate speedup is shown in Figure 1. Notice the speedup is greater than one before x~0.428 and less than one after 0.428. This is where the magical 42.8% comes from: shares submitted to the pool when x<0.428 have an expectation value greater than one. This is the essence of pool-hopping.
Figure 1. Speedup vs normalized shares submitted (see above text)
Multiple proportional pools
As shown above when considering proportional pools, shares should always be submitted to the pool with the least number of shares or should be solo-mined if all pools are above x~.43. How much is to be gained from multiple pools? A lot If you go to the aforementioned Wolfram Alpha link, calculate the speedup for x=1/difficulty=1/1690000. It equals 13.7x! This means if every share you submitted was the first to a proportional pool, you would have over a 13x speedup! Hypotheticals aside, what about in practice? FIgure 2 shows the results of Monte Carlo calculations for the speedup vs number of proportional pools hopped which roughly follows the trend: speedup = ln(2.5+pools). Here we can see it is always QUITE advantageous to add more pools. The marginal return, although diminishing, decrease only slowly! Hence, a hopper should have as many pools as possible.
Figure 2. Speedup vs number of proportional pools. Note the famous ~28% increase for hopping one pool. Due to the steep slope of speedup(x) for small x, it is quite advantageous to add more pools.
Other scoring methods [Edit: I understood PPLNS incorrectly; the math is completely incorrect]
Many scoring systems are not statistically analyzed and hence still offer opportunity for hopping. For example, Pay-Per-Last-N-Shares is still quite hoppable (with typical N), as follows. Let's renormalize N by dividing by the difficulty, to give us the normalized last N shares, M. Using the same analysis as above, the speedup can be computed as shown below in Figure 3. PPLNS strongly deviates from proportional only during rounds which pay (per share) below the statistical average. Because the hopper "suffers" from PPLNS only during sub-par rounds, the absolute difference between proportional and PPLNS is small. One can calculated the expectation values of PPLNS shares (as a function of x and M), with the following formula (Wolfram link): exp(x) ( Gamma(0,x) - Gamma(0,M+x)) which is valid for x < M. A maximum speedup (at current difficulty) is still 13x with M=0.5. A quick'n'dirty way to implement PPNLS pools in hopping software is to penalize the PPNLS pool by multiplying round_shares by .43/.21. Slush's pool is also hoppable under the same mechanism, although the curves depend on hashing rate as well. Almost any proportional-based pool will be hoppable at x<.10
Figure 3. Comparison of expectation value from share submission from proportional and PPNLS (M=0.5, see text) payout schemes. Notice the intersection with y=1, where share expectation value is equal to solo mining. These intersections are respectively equal to ~0.21 and ~0.428 for PPNLS and proportional.
Conclusion
I hope this clears up some confusion about expectation values of shares and pool hopping. If anything is unclear feel free to reply or send me a PM.
Cheers
Donations greatly appreciated!
1FQoC3zsos22QxZC35fMfP8JteMK9nEQPU
Proportional pools
Firstly, in a proportional pool the expectation value of a submitted share is dependent on one and only one variable: the number of shares submitted previously. Hashrate, timeslicing (as I understand it), number of shares you vs other other people submitted, etc. ALL DON'T MATTER. In fact, one can calculate the speedup. Define the normalized shares submitted, x, as the number of shares submitted divided by the current difficulty. Then the expectation value (average value) of a share (normalized to solo mining) is exp(x)*E1(x). E1 is a nonanalytic integral, but wolfram alpha will compute it for you. Alternatively, to a decent approximation, one can use speedup = (.428/x)^(0.65). This approximate formula can be handy for quickly calculating expectation value without resorting to numerical methods. The accurate speedup is shown in Figure 1. Notice the speedup is greater than one before x~0.428 and less than one after 0.428. This is where the magical 42.8% comes from: shares submitted to the pool when x<0.428 have an expectation value greater than one. This is the essence of pool-hopping.
Figure 1. Speedup vs normalized shares submitted (see above text)
Multiple proportional pools
As shown above when considering proportional pools, shares should always be submitted to the pool with the least number of shares or should be solo-mined if all pools are above x~.43. How much is to be gained from multiple pools? A lot If you go to the aforementioned Wolfram Alpha link, calculate the speedup for x=1/difficulty=1/1690000. It equals 13.7x! This means if every share you submitted was the first to a proportional pool, you would have over a 13x speedup! Hypotheticals aside, what about in practice? FIgure 2 shows the results of Monte Carlo calculations for the speedup vs number of proportional pools hopped which roughly follows the trend: speedup = ln(2.5+pools). Here we can see it is always QUITE advantageous to add more pools. The marginal return, although diminishing, decrease only slowly! Hence, a hopper should have as many pools as possible.
Figure 2. Speedup vs number of proportional pools. Note the famous ~28% increase for hopping one pool. Due to the steep slope of speedup(x) for small x, it is quite advantageous to add more pools.
Other scoring methods [Edit: I understood PPLNS incorrectly; the math is completely incorrect]
Many scoring systems are not statistically analyzed and hence still offer opportunity for hopping. For example, Pay-Per-Last-N-Shares is still quite hoppable (with typical N), as follows. Let's renormalize N by dividing by the difficulty, to give us the normalized last N shares, M. Using the same analysis as above, the speedup can be computed as shown below in Figure 3. PPLNS strongly deviates from proportional only during rounds which pay (per share) below the statistical average. Because the hopper "suffers" from PPLNS only during sub-par rounds, the absolute difference between proportional and PPLNS is small. One can calculated the expectation values of PPLNS shares (as a function of x and M), with the following formula (Wolfram link): exp(x) ( Gamma(0,x) - Gamma(0,M+x)) which is valid for x < M. A maximum speedup (at current difficulty) is still 13x with M=0.5. A quick'n'dirty way to implement PPNLS pools in hopping software is to penalize the PPNLS pool by multiplying round_shares by .43/.21. Slush's pool is also hoppable under the same mechanism, although the curves depend on hashing rate as well. Almost any proportional-based pool will be hoppable at x<.10
Figure 3. Comparison of expectation value from share submission from proportional and PPNLS (M=0.5, see text) payout schemes. Notice the intersection with y=1, where share expectation value is equal to solo mining. These intersections are respectively equal to ~0.21 and ~0.428 for PPNLS and proportional.
Conclusion
I hope this clears up some confusion about expectation values of shares and pool hopping. If anything is unclear feel free to reply or send me a PM.
Cheers
Donations greatly appreciated!
1FQoC3zsos22QxZC35fMfP8JteMK9nEQPU
Anyone still have invites to hoppersden?
dunno, ask around
(just joking I feel fine today)Thanks for the invite, I'll try to poke around and absorb/contribute whatever I can this weekend.
as i think he speaks about my mod:
no pplns is not hoppable. the window ist just to big.
atm i am trying to hop slush @300% difficulty and stays until block is found; but i know already that this won't work (due to bitcoin poison process)
^^ i just want to learn it mysql the hard way
no pplns is not hoppable. the window ist just to big.
atm i am trying to hop slush @300% difficulty and stays until block is found; but i know already that this won't work (due to bitcoin poison process)
^^ i just want to learn it mysql the hard way
With time slicing and penalty around 3.5, hopping PPLNS can still be effective...
proof? either a screen shot or a mathematical proof will be fine.
Is it just me, or is updates on pool shares on triple mining a little excessive?
I would think once every 60 seconds would be plenty fine.
Using c00w's hopper, latest version
Code:
[22:27:35] triple: 921,102
[22:27:44] triple: 921,232
[22:27:54] triple: 921,377
[22:28:14] triple: 921,680
[22:28:24] triple: 921,837
I would think once every 60 seconds would be plenty fine.
Using c00w's hopper, latest version
X8s.de now swapped to PPLNS
"Found block for Round 30 18 minutes ago · reply · retweet · favorite"
"After finding the next block we'll switch to PPLNS 5 hours ago · reply · retweet · favorite"
"Found block for Round 30 18 minutes ago · reply · retweet · favorite"
"After finding the next block we'll switch to PPLNS 5 hours ago · reply · retweet · favorite"
I removed them. Personally I hate PPLNS, once you stop mining those pools your shares starts instantly falling no matter how much you mined. When they then hit a block during your downtime you'll get only pennies.. Would be interesting to know efficiency of eg. x8s for 24/7 mining because there will always be people who start mining earlier in round and then drop off. Their contribution helps finding block faster and that benefits 24/7 miners.
Jump to: