[1500 TH] p2pool: Decentralized, DoS-resistant, Hop-Proof pool - page 538.

K1773R

legendary

Activity: 1792

Merit: 1008

/dev/null

Quote from: lenny_ on May 26, 2013, 04:15:56 PM

Thanks.

How to apply it to my p2pool? I have only p2pool from git and stratum-mining-proxy from git.

save the text above in a file and use patch(1)
basic linux stuff u should know Wink

iongchun

member

Activity: 75

Merit: 10

Quote from: lenny_ on May 26, 2013, 04:15:56 PM

Thanks.

How to apply it to my p2pool? I have only p2pool from git and stratum-mining-proxy from git.

Before forrestv's patch to stratum-mining-proxy is merged by slush,
maybe you can try forrestv's branch temporarily, ex.:

Code:

git clone https://github.com/forrestv/stratum.git stratum-forrestv
cd stratum-forrestv
git checkout maxlengthfix

lenny_

legendary

Activity: 1036

Merit: 1000

DARKNETMARKETS.COM

Thanks.

How to apply it to my p2pool? I have only p2pool from git and stratum-mining-proxy from git.

forrestv

hero member

Activity: 516

Merit: 643

Quote from: lenny_ on May 26, 2013, 02:49:30 PM

Why stratum mining proxy does not work with p2pool? Embarrassed

Code:

$ python mining_proxy.py -o localhost -p 9332 -gp 5550 -sp 5551
2013-05-26 20:51:22,381 INFO proxy jobs. # Using C extension for midstate speedup. Good!
2013-05-26 20:51:22,398 ERROR proxy mining_proxy.main # Stratum host/port autodetection failed
Traceback (most recent call last):
  File "mining_proxy.py", line 178, in main
    new_host = (yield utils.detect_stratum(args.host, args.port))
  File "/usr/local/lib/python2.7/dist-packages/Twisted-13.0.0-py2.7-linux-x86_64.egg/twisted/internet/defer.py", line 1070, in _inlineCallbacks
    result = g.send(result)
  File "/home/pioruns/stratum-mining-proxy/mining_libs/utils.py", line 69, in detect_stratum
    header = f.response_headers.get('x-stratum', None)[0]
TypeError: 'NoneType' object has no attribute '__getitem__'
2013-05-26 20:51:22,401 WARNING proxy mining_proxy.main # Stratum proxy version: 1.5.2
2013-05-26 20:51:22,408 WARNING proxy mining_proxy.test_update # Checking for updates...
2013-05-26 20:51:22,709 WARNING proxy mining_proxy.main # Trying to connect to Stratum pool at localhost:9332
2013-05-26 20:51:22,711 INFO stats stats.print_stats # 1 peers connected, state changed 1 times
2013-05-26 20:51:22,712 INFO proxy mining_proxy.on_connect # Connected to Stratum pool at localhost:9332
2013-05-26 20:51:22,712 INFO proxy mining_proxy.on_connect # Subscribing for mining jobs
2013-05-26 20:51:22,771 WARNING proxy mining_proxy.main # -----------------------------------------------------------------------
2013-05-26 20:51:22,771 WARNING proxy mining_proxy.main # PROXY IS LISTENING ON ALL IPs ON PORT 5551 (stratum) AND 5550 (getwork)
2013-05-26 20:51:22,771 WARNING proxy mining_proxy.main # -----------------------------------------------------------------------
2013-05-26 20:51:22,772 INFO proxy client_service.handle_event # Setting new difficulty: 0.999984741211
2013-05-26 20:51:22,773 INFO proxy mining_proxy.on_disconnect # Disconnected from Stratum pool at localhost:9332
2013-05-26 20:51:22,773 INFO stats stats.print_stats # 0 peers connected, state changed 1 times
2013-05-26 20:51:25,809 INFO stats stats.print_stats # 1 peers connected, state changed 1 times
2013-05-26 20:51:25,809 INFO proxy mining_proxy.on_connect # Connected to Stratum pool at localhost:9332
2013-05-26 20:51:25,809 INFO proxy mining_proxy.on_connect # Subscribing for mining jobs
2013-05-26 20:51:25,812 INFO proxy client_service.handle_event # Setting new difficulty: 0.999984741211
2013-05-26 20:51:25,813 INFO proxy mining_proxy.on_disconnect # Disconnected from Stratum pool at localhost:9332
2013-05-26 20:51:25,813 INFO stats stats.print_stats # 0 peers connected, state changed 1 times
2013-05-26 20:51:28,183 INFO stats stats.print_stats # 1 peers connected, state changed 1 times
2013-05-26 20:51:28,183 INFO proxy mining_proxy.on_connect # Connected to Stratum pool at localhost:9332
2013-05-26 20:51:28,184 INFO proxy mining_proxy.on_connect # Subscribing for mining jobs
2013-05-26 20:51:28,186 INFO proxy client_service.handle_event # Setting new difficulty: 0.999984741211
2013-05-26 20:51:28,187 INFO proxy mining_proxy.on_disconnect # Disconnected from Stratum pool at localhost:9332
2013-05-26 20:51:28,188 INFO stats stats.print_stats # 0 peers connected, state changed 1 times
2013-05-26 20:51:30,894 INFO stats stats.print_stats # 1 peers connected, state changed 1 times
2013-05-26 20:51:30,894 INFO proxy mining_proxy.on_connect # Connected to Stratum pool at localhost:9332
2013-05-26 20:51:30,894 INFO proxy mining_proxy.on_connect # Subscribing for mining jobs
2013-05-26 20:51:30,897 INFO proxy client_service.handle_event # Setting new difficulty: 0.999984741211
2013-05-26 20:51:30,898 INFO proxy mining_proxy.on_disconnect # Disconnected from Stratum pool at localhost:9332
2013-05-26 20:51:30,898 INFO stats stats.print_stats # 0 peers connected, state changed 1 times

Keep disconnecting, mining not possible.

Lenny, I was looking at stratum-mining-proxy just this morning and found the cause of this. You need to apply this patch to stratum to get it to work:

Code:

diff --git a/stratum/protocol.py b/stratum/protocol.py
index 02d9954..f64fce8 100644
--- a/stratum/protocol.py
+++ b/stratum/protocol.py
@@ -36,6 +36,7 @@ class RequestCounter(object):
                 
 class Protocol(LineOnlyReceiver):
     delimiter = '\n'
+    MAX_LENGTH = 16384000
     
     def _get_id(self):
         self.request_id += 1

I'll get this in a pull request later.

EDIT: Pull request is here: https://github.com/slush0/stratum/pull/8

lenny_

legendary

Activity: 1036

Merit: 1000

DARKNETMARKETS.COM

Why stratum mining proxy does not work with p2pool? Embarrassed

Code:

$ python mining_proxy.py -o localhost -p 9332 -gp 5550 -sp 5551
2013-05-26 20:51:22,381 INFO proxy jobs. # Using C extension for midstate speedup. Good!
2013-05-26 20:51:22,398 ERROR proxy mining_proxy.main # Stratum host/port autodetection failed
Traceback (most recent call last):
  File "mining_proxy.py", line 178, in main
    new_host = (yield utils.detect_stratum(args.host, args.port))
  File "/usr/local/lib/python2.7/dist-packages/Twisted-13.0.0-py2.7-linux-x86_64.egg/twisted/internet/defer.py", line 1070, in _inlineCallbacks
    result = g.send(result)
  File "/home/pioruns/stratum-mining-proxy/mining_libs/utils.py", line 69, in detect_stratum
    header = f.response_headers.get('x-stratum', None)[0]
TypeError: 'NoneType' object has no attribute '__getitem__'
2013-05-26 20:51:22,401 WARNING proxy mining_proxy.main # Stratum proxy version: 1.5.2
2013-05-26 20:51:22,408 WARNING proxy mining_proxy.test_update # Checking for updates...
2013-05-26 20:51:22,709 WARNING proxy mining_proxy.main # Trying to connect to Stratum pool at localhost:9332
2013-05-26 20:51:22,711 INFO stats stats.print_stats # 1 peers connected, state changed 1 times
2013-05-26 20:51:22,712 INFO proxy mining_proxy.on_connect # Connected to Stratum pool at localhost:9332
2013-05-26 20:51:22,712 INFO proxy mining_proxy.on_connect # Subscribing for mining jobs
2013-05-26 20:51:22,771 WARNING proxy mining_proxy.main # -----------------------------------------------------------------------
2013-05-26 20:51:22,771 WARNING proxy mining_proxy.main # PROXY IS LISTENING ON ALL IPs ON PORT 5551 (stratum) AND 5550 (getwork)
2013-05-26 20:51:22,771 WARNING proxy mining_proxy.main # -----------------------------------------------------------------------
2013-05-26 20:51:22,772 INFO proxy client_service.handle_event # Setting new difficulty: 0.999984741211
2013-05-26 20:51:22,773 INFO proxy mining_proxy.on_disconnect # Disconnected from Stratum pool at localhost:9332
2013-05-26 20:51:22,773 INFO stats stats.print_stats # 0 peers connected, state changed 1 times
2013-05-26 20:51:25,809 INFO stats stats.print_stats # 1 peers connected, state changed 1 times
2013-05-26 20:51:25,809 INFO proxy mining_proxy.on_connect # Connected to Stratum pool at localhost:9332
2013-05-26 20:51:25,809 INFO proxy mining_proxy.on_connect # Subscribing for mining jobs
2013-05-26 20:51:25,812 INFO proxy client_service.handle_event # Setting new difficulty: 0.999984741211
2013-05-26 20:51:25,813 INFO proxy mining_proxy.on_disconnect # Disconnected from Stratum pool at localhost:9332
2013-05-26 20:51:25,813 INFO stats stats.print_stats # 0 peers connected, state changed 1 times
2013-05-26 20:51:28,183 INFO stats stats.print_stats # 1 peers connected, state changed 1 times
2013-05-26 20:51:28,183 INFO proxy mining_proxy.on_connect # Connected to Stratum pool at localhost:9332
2013-05-26 20:51:28,184 INFO proxy mining_proxy.on_connect # Subscribing for mining jobs
2013-05-26 20:51:28,186 INFO proxy client_service.handle_event # Setting new difficulty: 0.999984741211
2013-05-26 20:51:28,187 INFO proxy mining_proxy.on_disconnect # Disconnected from Stratum pool at localhost:9332
2013-05-26 20:51:28,188 INFO stats stats.print_stats # 0 peers connected, state changed 1 times
2013-05-26 20:51:30,894 INFO stats stats.print_stats # 1 peers connected, state changed 1 times
2013-05-26 20:51:30,894 INFO proxy mining_proxy.on_connect # Connected to Stratum pool at localhost:9332
2013-05-26 20:51:30,894 INFO proxy mining_proxy.on_connect # Subscribing for mining jobs
2013-05-26 20:51:30,897 INFO proxy client_service.handle_event # Setting new difficulty: 0.999984741211
2013-05-26 20:51:30,898 INFO proxy mining_proxy.on_disconnect # Disconnected from Stratum pool at localhost:9332
2013-05-26 20:51:30,898 INFO stats stats.print_stats # 0 peers connected, state changed 1 times

Keep disconnecting, mining not possible.

gnomicide

full member

Activity: 122

Merit: 100

Quote from: forrestv on May 26, 2013, 02:08:12 PM

Quote from: gnomicide on May 26, 2013, 11:33:19 AM

I'm working on a mining interface which will interact with P2Pool, and following information on Stratum here: https://www.btcguild.com/new_protocol.php

Any idea why I'm getting the response:

{"error": {"message": "Service not found", "code": -32601, "data": null}, "jsonrpc": "2.0", "id": 1, "result": null}

when I POST to my p2pool instance:

http://x.x.x.x:yyyy

{"id": 1, "method": "mining.subscribe", "params": []}

method: getwork returns what I'm expecting. Do I have to somehow POST to the "stratum+tcp://..." URL instead? Or am I missing certain headers? (only have 'Content-Type' = 'application/json')

I can see the error's coming from jsonrpc.py:

Code:

for service_name in method.split('.')[:-1]:
  provider = getattr(provider, 'svc_' + service_name, None)
  if provider is None:
    raise Error_for_code(-32601)(u'Service not found')

but it should be able to find svc_mining, and then rpc_subscribe right after, and I can't tell why provider is coming back null.

(Unfortunately I don't know Python... just Ruby and C#)

Stratum doesn't use HTTP-based JSON-RPC. Instead, it uses a line-based JSON-RPC where you just send a \n-terminated lines. Connect to the port and then just send:

Code:

{"id": 1, "method": "mining.subscribe", "params": []}

and it'll work.

Now getting back:

{"params": [13.07823061477393], "jsonrpc": "2.0", "method": "mining.set_difficulty", "id": 81758260}

That's one big hurdle passed, thanks a lot!

forrestv

hero member

Activity: 516

Merit: 643

Quote from: gnomicide on May 26, 2013, 11:33:19 AM

I'm working on a mining interface which will interact with P2Pool, and following information on Stratum here: https://www.btcguild.com/new_protocol.php

Any idea why I'm getting the response:

{"error": {"message": "Service not found", "code": -32601, "data": null}, "jsonrpc": "2.0", "id": 1, "result": null}

when I POST to my p2pool instance:

http://x.x.x.x:yyyy

{"id": 1, "method": "mining.subscribe", "params": []}

method: getwork returns what I'm expecting. Do I have to somehow POST to the "stratum+tcp://..." URL instead? Or am I missing certain headers? (only have 'Content-Type' = 'application/json')

I can see the error's coming from jsonrpc.py:

Code:

for service_name in method.split('.')[:-1]:
  provider = getattr(provider, 'svc_' + service_name, None)
  if provider is None:
    raise Error_for_code(-32601)(u'Service not found')

but it should be able to find svc_mining, and then rpc_subscribe right after, and I can't tell why provider is coming back null.

(Unfortunately I don't know Python... just Ruby and C#)

Stratum doesn't use HTTP-based JSON-RPC. Instead, it uses a line-based JSON-RPC where you just send a \n-terminated lines. Connect to the port and then just send:

Code:

{"id": 1, "method": "mining.subscribe", "params": []}

and it'll work.

gyverlb

hero member

Activity: 896

Merit: 1000

Quote from: gyverlb on May 25, 2013, 02:12:54 PM

Preliminary report of my testing of recent bitcoind builds.

As nothing seemed to bother bitcoind since sipa's patch was included, I decided to test really permissive settings.

For the record here are the modifications I used:

Code:

blockmaxsize=1000000
blockminsize=400000
mintxfee=0.00001
minrelaytxfee=0.00001

Result of "bitcoind getmininginfo":

Code:

{
    "blocks" : 238047,
    "currentblocksize" : 996888,
    "currentblocktx" : 1175,
    "difficulty" : 12153411.70977583,
    "errors" : "",
    "generate" : false,
    "genproclimit" : -1,
    "hashespersec" : 0,
    "pooledtx" : 3059,
    "testnet" : false
}

According to blockchain info, there were 4026 unconfirmed tx (~19MB) when my memory pool had 3059 of them (pooledtx above).
The total of taxes in unconfirmed TX is currently above 25BTC (if a block could include all of them it would be worth twice the usual reward).

Getblocktemplate average latency is 0.3s. My efficiency is 112.5% at the moment after 24h of running a fresh P2Pool instance and bitcoind with these settings which is the average I'm used to (I didn't have a higher efficiency with smaller maxblocksize values with 0.8.1).

I could force higher getblocktemplate latency but I don't see the point: either p2pool code changed and this latency doesn't change efficiency as much as it did or the level where it starts to make efficiency worse is far above what a normal bitcoind configuration should be able to reach currently (0.3s was obtained with 2x the default max block size and 0.1x the default minimum fee limits).

One possible cause of low efficiency: if you don't have a very fast CPU, bitcoind <0.8.1 with the default configuration is eating 100% of one CPU core on each getblocktemplate call, this could slow your p2pool node enough to lower your efficiency.

I'll update the guide.

TL;DR;
My recommendation: upgrade to 0.8.2rc3 or git HEAD and then use the values above in your bitcoin.conf: it should raise the P2Pool payouts to ~26BTC/block (that's 4% more from fees) in the current situation.

gnomicide

full member

Activity: 122

Merit: 100

I'm working on a mining interface which will interact with P2Pool, and following information on Stratum here: https://www.btcguild.com/new_protocol.php

Any idea why I'm getting the response:

{"error": {"message": "Service not found", "code": -32601, "data": null}, "jsonrpc": "2.0", "id": 1, "result": null}

when I POST to my p2pool instance:

http://x.x.x.x:yyyy

{"id": 1, "method": "mining.subscribe", "params": []}

method: getwork returns what I'm expecting. Do I have to somehow POST to the "stratum+tcp://..." URL instead? Or am I missing certain headers? (only have 'Content-Type' = 'application/json')

I can see the error's coming from jsonrpc.py:

Code:

for service_name in method.split('.')[:-1]:
  provider = getattr(provider, 'svc_' + service_name, None)
  if provider is None:
    raise Error_for_code(-32601)(u'Service not found')

but it should be able to find svc_mining, and then rpc_subscribe right after, and I can't tell why provider is coming back null.

(Unfortunately I don't know Python... just Ruby and C#)

gyverlb

hero member

Activity: 896

Merit: 1000

Preliminary report of my testing of recent bitcoind builds.

As nothing seemed to bother bitcoind since sipa's patch was included, I decided to test really permissive settings.
I allow for the maximum block size supported by the protocol (1MB), fees only 10% of the default minimum and even allow for 400kB of fee-less transactions if nothing else is available. These last ones should help the network clean itself from these unconfirmed transactions and help people with older bitcoind versions which slowed down due to the current backlog.

Here are the modifications I'm testing right now:

Code:

blockmaxsize=1000000
blockminsize=400000
mintxfee=0.00001
minrelaytxfee=0.00001

Here are the default values according to "bitcoind --help"

Code:

blockmaxsize=250000
blockminsize=0
mintxfee=0.0001
minrelaytxfee=0.0001

Note : I believe the default blockmaxsize has been raised to 500000 some time ago with 0.8 (and according to the current main.h value for MAX_BLOCK_SIZE_GEN it should be), the help may be outdated or I may have missed something in the source code.

"bitcoind getmininginfo" is useful to check how the node includes transactions:

Code:

{
    "blocks" : 237896,
    "currentblocksize" : 998727,
    "currentblocktx" : 965,
    "difficulty" : 12153411.70977583,
    "errors" : "",
    "generate" : false,
    "genproclimit" : -1,
    "hashespersec" : 0,
    "pooledtx" : 2064,
    "testnet" : false
}

The currentblocksize is almost 1MB which is expected (it takes some time to reach it though).
The node selected 965 transactions out of 2064 it knows about (there are 5109 on the network according to blockchain.info but my node didn't see all of them yet).

The getblocktemplate latency started to rise but is still around 0.2s. My efficiency is still very good at 110-115%. I just restarted my P2Pool node to get more accurate values (it's an average and I didn't have the same bitcoind settings all along). I'll keep an eye on the node traffic too: I suppose that as nodes must verify shares they need to know all transactions used in the block templates which may mean more traffic with bigger templates.

What is surprising is that there's a log of transactions with fee/kB above 0.00001 and below 0.00005 (with 0.00005 I couldn't fill a 500kB block template). I wonder who/what is generating these transactions, certainly not ordinary users.

I've yet to see how bitcoind and P2Pool behave on a 24h period to make any recommendation but it seems that sipa's patch allows us to include far more transactions in our blocks without performance/income penalties. Generating 1MB blocks may even be doable (which may mean substantially more income and some troll repellent as a bonus).

mdude77

legendary

Activity: 1540

Merit: 1001

Quote from: gyverlb on May 25, 2013, 09:56:44 AM

There are such misconceptions floating around...

OK first mdude77 is wondering why the latency is lowered by restarting bitcoind and raises again.

When you restart bitcoind it doesn't save the list of unconfirmed transactions on disk: they are only maintained in what's called the memory pool.
When you restart it, at first it doesn't have any transaction in its memory pool. So any rpc call to bitcoind involving any parsing of the list of transaction in the memory pool is fast: getblocktemplate is obviously one of them as its goal is to build a template for a block including the tx in your memory pool that your bitcoind deems worth confirming.
Then your bitcoind learns of unconfirmed transactions at the rate at which it's neighboors (the nodes it's connected to) on the bitcoind P2P network transmit them. Only new transactions are automatically broadcasted and received by a freshly started node quickly, old ones are transmitted far slower to avoid network congestion. So a freshly started node gets new transactions appearing on the network quickly and slowly catches up through the backlog of unconfirmed transactions. The rate at which a node receives old transactions varies greatly: it depends on the nodes it is connected to (not every node is configured the same, some may relay more transactions than others).
This is why getblocktemplate latency is slowly rising and is perfectly consistent with getblocktemplate latency being heavily dependent on both the number of transactions your bitcoind accepts in its memory pool and the number of transactions it selects for confirmation (these being controlled by maxblocksize, minxtxfee and minrelaytxfee among others).

Second: the getblocktemplate and P2Pool orphan rate.
First a bit of vocabulary just to avoid any misunderstanding: what daemondazz calls P2Pool blocks are P2Pool shares.

AFAIK P2Pool doesn't need a fresh getblocktemplate to build a share. It can reuse a previous one (unless a Bitcoin block has just been found, then shares based on old templates would generate orphan blocks, see one of my earlier posts on why it's bad). The only drawback of using an old getblocktemplate result is that it doesn't include new transactions that were received after it (so it lowers the fee income and slow transaction confirmation a bit).

What the problem is is probably more subtle than that: P2Pool is an event-driven program (as opposed to multi-threaded or multi-process) with most of its event being handled asynchronously (waiting for a getblocktemplate result is probably done asynchronously).
What could happen but is very difficult to analyze is that the event-driven framework doesn't cope with long running queries as well for some obscure technical reason (like polling some event result in exponentially longer intervals instead of being woken up by a select or equivalent system call) or might have a timing problem when some task that can't be done asynchronously (like pure crypto computations) happens to be done right when the geblocktemplate comes back. There's even a system in place in P2Pool to punish shares computed from old templates when a newer block is known: P2Pool nodes generating them because of high getblocktemplate latency will most probably see them orphaned.

I don't know the exact reason why getblocktemplate affected efficiency and even if it's still the case today as forrestv might have changed something that removes this problem. It was still the case very recently (like less than 2 months ago) when getblocktemplate took more than 0.2s. I don't check often how it affects p2pool but I'm doing it right now (in fact I'm studying how the block size and fee limits affect getblocktemplate in the current situation, checking the efficiency is just a bonus). If the behavior of p2pool changed I'll know it in the following days and will be able to update my guide. For now I still recommend to keep it under 0.2s to be safe.

Thanks for the detailed explanation. Now I understand, and it makes a lot more sense than it did before, and I agree with your conclusions.

M

Mogumodz

sr. member

Activity: 290

Merit: 250

Quote from: gyverlb on May 25, 2013, 10:10:40 AM

Quote from: Mogumodz on May 25, 2013, 07:08:47 AM

Quote from: mdude77 on May 25, 2013, 06:54:49 AM

Quote from: Mogumodz on May 25, 2013, 06:37:56 AM

While all you lot touch each other and fight about it.

No changes to Bitcoin.conf, simply updated to 0.8.3rc3 and I have some LOW LAT back, seems to be holding.

I still don't see what the big deal was about latency. Everything seemed to be working fine, but people were panicking over apparent high latency. Instead they'd rather use a pre-release version of bitcoind that specifics warns you not to use for mining.

M

Trouble started with 0.8.1 and 11.4 for me. Seems all back to normal now. Not had to change anything in Bitcoin.conf to affect tx and wotnot, waiting worked.

As your post could be misinterpreted: to be accurate simply waiting didn't work: you had to update bitcoind to a version which includes the fix from sipa for the getblocktemplate low performance. The P2Pool version doesn't have anything to do with it and you verified it yourself by upgrading bitcoind and keeping the same P2Pool version.

Your trouble started when bitcoind 0.8.1 had to handle the current backlog of unconfirmed transactions. 0.8.1 could handle a normal block size worth of transactions (500kB to 1MB) without problems but it simply couldn't process more than 20MB of them efficiently until sipa fixed the code.

Cheers, I was trying to convey that simply updating the Bitcoin client solved it for me and that the p2pool version is the same as when the trouble started. I see how that can be misinterpreted as I was just listing everything I was using at the time and currently.

gyverlb

hero member

Activity: 896

Merit: 1000

Quote from: bitpop on May 25, 2013, 09:39:35 AM

Sweet. What's the harm in making my node include them all.

Difficult to say. With a default configuration your node won't include them all.

I'm testing lower fee limits to find out if it affects bitcoind and P2Pool negatively but it takes time. As I explained earlier old unconfirmed transactions move slowly and on each bitcoind restart you have to download them again from your peers (currently my bitcoind still can't fill a 500kB block with what it got from the network since my last restart).

gyverlb

hero member

Activity: 896

Merit: 1000

Quote from: Mogumodz on May 25, 2013, 07:08:47 AM

Quote from: mdude77 on May 25, 2013, 06:54:49 AM

Quote from: Mogumodz on May 25, 2013, 06:37:56 AM

While all you lot touch each other and fight about it.

No changes to Bitcoin.conf, simply updated to 0.8.3rc3 and I have some LOW LAT back, seems to be holding.

I still don't see what the big deal was about latency. Everything seemed to be working fine, but people were panicking over apparent high latency. Instead they'd rather use a pre-release version of bitcoind that specifics warns you not to use for mining.

M

Trouble started with 0.8.1 and 11.4 for me. Seems all back to normal now. Not had to change anything in Bitcoin.conf to affect tx and wotnot, waiting worked.

As your post could be misinterpreted: to be accurate simply waiting didn't work: you had to update bitcoind to a version which includes the fix from sipa for the getblocktemplate low performance. The P2Pool version doesn't have anything to do with it and you verified it yourself by upgrading bitcoind and keeping the same P2Pool version.

Your trouble started when bitcoind 0.8.1 had to handle the current backlog of unconfirmed transactions. 0.8.1 could handle a normal block size worth of transactions (500kB to 1MB) without problems but it simply couldn't process more than 20MB of them efficiently until sipa fixed the code.

gyverlb

hero member

Activity: 896

Merit: 1000

Quote from: mdude77 on May 25, 2013, 08:55:52 AM

Quote from: daemondazz on May 25, 2013, 08:30:17 AM

Also, the P2Pool block time is 10 seconds, so having a latency of >10 seconds means that when a new block is detected, you're going to completely miss the next block, which would half your actual effective rate.

Finally something that is factual. Thanks!

The highest I saw mine was 3s. And as I stated multiple times, restarting bitcoind put it back down where it was supposed to be, but it did start climbing back up. I still fail to see how transactions have anything to do with it when restarting bitcoind lowers the latency.

M

There are such misconceptions floating around...

OK first mdude77 is wondering why the latency is lowered by restarting bitcoind and raises again.

When you restart bitcoind it doesn't save the list of unconfirmed transactions on disk: they are only maintained in what's called the memory pool.
When you restart it, at first it doesn't have any transaction in its memory pool. So any rpc call to bitcoind involving any parsing of the list of transaction in the memory pool is fast: getblocktemplate is obviously one of them as its goal is to build a template for a block including the tx in your memory pool that your bitcoind deems worth confirming.
Then your bitcoind learns of unconfirmed transactions at the rate at which it's neighboors (the nodes it's connected to) on the bitcoind P2P network transmit them. Only new transactions are automatically broadcasted and received by a freshly started node quickly, old ones are transmitted far slower to avoid network congestion. So a freshly started node gets new transactions appearing on the network quickly and slowly catches up through the backlog of unconfirmed transactions. The rate at which a node receives old transactions varies greatly: it depends on the nodes it is connected to (not every node is configured the same, some may relay more transactions than others).
This is why getblocktemplate latency is slowly rising and is perfectly consistent with getblocktemplate latency being heavily dependent on both the number of transactions your bitcoind accepts in its memory pool and the number of transactions it selects for confirmation (these being controlled by maxblocksize, minxtxfee and minrelaytxfee among others).

Second: the getblocktemplate and P2Pool orphan rate.
First a bit of vocabulary just to avoid any misunderstanding: what daemondazz calls P2Pool blocks are P2Pool shares.

AFAIK P2Pool doesn't need a fresh getblocktemplate to build a share. It can reuse a previous one (unless a Bitcoin block has just been found, then shares based on old templates would generate orphan blocks, see one of my earlier posts on why it's bad). The only drawback of using an old getblocktemplate result is that it doesn't include new transactions that were received after it (so it lowers the fee income and slow transaction confirmation a bit).

What the problem is is probably more subtle than that: P2Pool is an event-driven program (as opposed to multi-threaded or multi-process) with most of its event being handled asynchronously (waiting for a getblocktemplate result is probably done asynchronously).
What could happen but is very difficult to analyze is that the event-driven framework doesn't cope with long running queries as well for some obscure technical reason (like polling some event result in exponentially longer intervals instead of being woken up by a select or equivalent system call) or might have a timing problem when some task that can't be done asynchronously (like pure crypto computations) happens to be done right when the geblocktemplate comes back. There's even a system in place in P2Pool to punish shares computed from old templates when a newer block is known: P2Pool nodes generating them because of high getblocktemplate latency will most probably see them orphaned.

I don't know the exact reason why getblocktemplate affected efficiency and even if it's still the case today as forrestv might have changed something that removes this problem. It was still the case very recently (like less than 2 months ago) when getblocktemplate took more than 0.2s. I don't check often how it affects p2pool but I'm doing it right now (in fact I'm studying how the block size and fee limits affect getblocktemplate in the current situation, checking the efficiency is just a bonus). If the behavior of p2pool changed I'll know it in the following days and will be able to update my guide. For now I still recommend to keep it under 0.2s to be safe.

bitpop

legendary

Activity: 2912

Merit: 1060

Sweet. What's the harm in making my node include them all.

Subo1977

sr. member

Activity: 344

Merit: 250

Flixxo - Watch, Share, Earn!

Quote from: bitpop on May 25, 2013, 09:26:26 AM

Ps is there an explorer for unconfirmed transactions?

blockchain.info/de/unconfirmed-transactions

bitpop

legendary

Activity: 2912

Merit: 1060

Ps is there an explorer for unconfirmed transactions?

bitpop

legendary

Activity: 2912

Merit: 1060

Huge difference on mine with rc3 see sig

gyverlb

hero member

Activity: 896

Merit: 1000

Quote from: mdude77 on May 25, 2013, 07:30:01 AM

And you know for sure HOW that when the latency goes up that it affects p2pool?

Maybe because I did exactly what I described in the guide in my signature.
Mind the fact that I said it affected p2pool. forrestv didn't communicate much on the subject, he might have changed p2pool's code since the problem was detected. This is easy to check by doing the same experiment that I did and described in the guide.

Currently I'm slowly raising maxblocksize and lowering mintxfee and minrelaytxfee on my node to find out how the new bitcoind code is behaving, how the getblocktemplate latency changes and how the efficiency reacts. I'll probably have to add some tips based on my results.
For example, even with the default fees, bitcoind doesn't fill a 500kB block on my node so raising maxblocksize doesn't change much, I'm just testing fee limits below the default value.
I suspect most of the 29MB of transactions currently around is made of fee-less or below recommended fees.

Quote from: mdude77 on May 25, 2013, 07:30:01 AM

When everyone else tries saying there's a problem, you call them a troll, and it's just bad luck. Yet when you say there's a problem, it's gospel, not bad luck?

Which problem and reported by whom? If it's still this nonsense about a bug in p2pool when we have bad luck, yes there are trolls. I don't even bother arguing on the subject now, organofcorti has already made the statistical analysis of p2pool luck and shown it was perfectly normal.

And if you didn't believe me when I said getblocktemplate latency was a problem, why don't you argue with the Bitcoin devs that included a patch to lower it this very week?

Quote from: mdude77 on May 25, 2013, 07:30:01 AM

I agree, everyone having different latencies should theoretically affect their performance vs others. All the more reason to leave it alone and keep everyone the same!

Yeah right, like it would work and nobody would try to profit from the situation if there's even a mirage of an advantage to gain. There are many people motivated by pure profit or even blinded by some assumption that tuning something as much as possible is good although it's often a compromise.
Just look at what I had to say when someone gave an example of maxblocksize=5000: there's clearly work to be done to avoid these extremes (in this case just explaining that it's counterproductive should be enough).

Topic: [1500 TH] p2pool: Decentralized, DoS-resistant, Hop-Proof pool - page 538. (Read 2591920 times)