I've never seen cpu usage go higher than a 35% spike running p2pool, it usually sits around 8% - & that's merge mining 12 other alts. Maybe using XT is the problem?
What OS are you using for your nodes?
I think the biggest performance improvement could be achieved by using C instead of python, but this has been discussed many times previously in this thread to no avail. I never could get pypy to work with p2pool, could you describe how you installed/ran it - I'd like to have another go
Keep in mind that I have 350 TH/s on my node 24/7, and spikes up to 3.0 PH/s. We probably have about two hundred miners (i.e. computers, not owners) that use our node. I think that might have something to do with the CPU load.
Linux prime 3.2.0-4-amd64 #1 SMP Debian 3.2.63-2+deb7u2 x86_64 GNU/Linux
Intel(R) Core(TM) i3-2120 CPU @ 3.30GHz
I think the problem might be related to the use of the poll()-based reactor in twisted. The poll() function checks a bunch of file descriptors (e.g. network ports) to see if any have data available. When you have a lot of
http://pipeit.blogspot.com/2011/07/select-poll-and-epoll-twisted-story.htmlHere are the first few lines of my cProfile stats results for my non-XT node (slower CPU), sorted by the total time each function took (not including subroutines):
>>> p.print_stats()
Sun Sep 27 17:25:26 2015 cprofile-001.log
1042548800 function calls (1022440006 primitive calls) in 9605.998 seconds
Ordered by: internal time
ncalls tottime percall cumtime percall filename:lineno(function)
1828987 7667.676 0.004 7667.676 0.004 {built-in method poll}
133405 631.073 0.005 701.931 0.005 p2pool/p2p.py:189(update_remote_view_of_my_known_txs)
2025600 55.807 0.000 91.726 0.000 p2pool/util/math.py:64(add_dicts)
6083060/521034 42.351 0.000 204.197 0.000 p2pool/util/pack.py:292(read)
8211943/2317050 41.329 0.000 187.231 0.000 p2pool/util/pack.py:298(write)
50788863 38.770 0.000 45.809 0.000 p2pool/util/pack.py:13(read)
1748230 33.862 0.000 35.874 0.000 p2pool/util/graph.py:14(_shift)
3454845/2954699 31.683 0.000 240.778 0.000 p2pool/util/pack.py:51(_pack)
33374468 31.063 0.000 63.310 0.000 p2pool/util/pack.py:96(read)
39492368 30.537 0.000 42.449 0.000 p2pool/util/pack.py:115(write)
17576 29.261 0.002 36.366 0.002 p2pool/node.py:252(_)
2559639/2225541 27.652 0.000 1004.009 0.000 p2pool/util/variable.py:68(set)
12883 25.897 0.002 298.297 0.023 p2pool/p2p.py:396(handle_remember_tx)
925154/859232 25.618 0.000 191.821 0.000 p2pool/util/pack.py:166(read)
1748230 25.587 0.000 128.783 0.000 p2pool/util/graph.py:46(_add_datum)
7155588 23.545 0.000 32.801 0.000 p2pool/util/pack.py:221(write)
1432678/1382638 22.485 0.000 153.893 0.000 p2pool/util/pack.py:174(write)
4503389/2916784 21.317 0.000 29.991 0.000 p2pool/util/pack.py:270(__eq__)
6426883 21.155 0.000 33.183 0.000 p2pool/util/pack.py:215(read)
515095 20.821 0.000 34.674 0.000 p2pool/util/math.py:186(natural_to_string)
33493018 17.543 0.000 26.334 0.000 p2pool/util/pack.py:259(__getitem__)
101548128/101547801 17.280 0.000 18.491 0.000 {len}
2198269/2198243 16.876 0.000 101.744 0.000 {sorted}
47813834 16.131 0.000 16.131 0.000 {_struct.pack}
941 15.535 0.017 35.789 0.038 p2pool/data.py:119(generate_transaction)
5306039 14.821 0.000 14.821 0.000 {_hashlib.openssl_sha256}
5401 14.798 0.003 14.803 0.003 {method 'items' of 'dict' objects}
44406 14.483 0.000 24.064 0.001 p2pool/bitcoin/sha256.py:17(process)
49754459 14.364 0.000 15.180 0.000 {getattr}
82295913 13.969 0.000 13.969 0.000 {method 'append' of 'list' objects}
7485573 13.270 0.000 13.270 0.000 {binascii.a2b_hex}
34547735 13.172 0.000 13.172 0.000 p2pool/util/math.py:69()
5850 11.564 0.002 407.450 0.070 p2pool/node.py:246(_)
2712960 10.420 0.000 13.272 0.000 p2pool/data.py:1011(add_share)
7433713 10.328 0.000 16.661 0.000 p2pool/util/forest.py:332(get_chain)
2534168/285416 10.066 0.000 1790.124 0.006 {method 'send' of 'generator' objects}
16053 9.917 0.001 9.917 0.001 /usr/lib/python2.7/json/encoder.py:205(iterencode)
43314052 9.827 0.000 9.827 0.000 {method 'get' of 'dict' objects}
11226 9.816 0.001 9.816 0.001 /usr/lib/python2.7/json/decoder.py:371(raw_decode)
6337858 9.336 0.000 17.570 0.000 p2pool/util/pack.py:188(read)
35144317 9.291 0.000 9.291 0.000 p2pool/util/math.py:63()
2712960 9.236 0.000 11.115 0.000 p2pool/data.py:1022(add_verified_hash)
5103 8.790 0.002 175.572 0.034 p2pool/node.py:23(handle_shares)
25577856 8.662 0.000 8.662 0.000 p2pool/bitcoin/sha256.py:18(rightrotate)
5306038 8.635 0.000 8.635 0.000 {method 'digest' of '_hashlib.HASH' objects}
1748230 8.348 0.000 27.323 0.000 p2pool/util/graph.py:23(_)
185065 8.338 0.000 1097.603 0.006 p2pool/util/p2protocol.py:63(packetReceived)
2236393 8.072 0.000 10.352 0.000 p2pool/util/datachunker.py:18(get)
15805 7.840 0.000 23.109 0.001 p2pool/data.py:338(get_other_tx_hashes)
16472083 7.647 0.000 7.647 0.000 {divmod}
13992152 7.534 0.000 11.242 0.000 p2pool/util/pack.py:261(__setitem__)
1748230 7.523 0.000 45.516 0.000 p2pool/util/graph.py:32(_shift_bins_so_t_is_not_past_end)
2526120 7.308 0.000 12.750 0.000 p2pool/bitcoin/height_tracker.py:106(get_height_rel_highest)
6389979 6.975 0.000 7.048 0.000 {method 'join' of 'str' objects}
138138 6.974 0.000 75.463 0.001 p2pool/util/skiplist.py:12(__call__)
1742141 6.708 0.000 37.944 0.000 p2pool/bitcoin/data.py:10(hash256)
3859910 6.190 0.000 43.954 0.000 p2pool/util/pack.py:313(write)
8056916 5.986 0.000 9.912 0.000 p2pool/util/pack.py:192(write)
157 5.716 0.036 35.242 0.224 p2pool/main.py:199(save_shares)
6809454/4283334 5.614 0.000 29.820 0.000 {max}
6911540 5.597 0.000 5.597 0.000 {binascii.b2a_hex}
969 5.518 0.006 28.824 0.030 p2pool/bitcoin/data.py:180(calculate_merkle_link)
3556045 5.510 0.000 6.936 0.000 p2pool/util/graph.py:24()
1828987 5.438 0.000 9135.571 0.005 /usr/lib/python2.7/dist-packages/twisted/internet/pollreactor.py:147(doPoll)
3454845/2954699 5.414 0.000 245.595 0.000 p2pool/util/pack.py:73(pack)
4553432 5.247 0.000 20.618 0.000 p2pool/util/pack.py:130(read)
17540412 5.109 0.000 5.109 0.000 {method 'iteritems' of 'dict' objects}
95 5.102 0.054 5.102 0.054 {posix.fsync}
2236477/2236471 5.097 0.000 1135.022 0.001 p2pool/util/p2protocol.py:29(dataReceiver)
6729729 4.927 0.000 33.823 0.000 p2pool/util/pack.py:82(packed_size)
33380877 4.815 0.000 4.815 0.000 {ord}
1828987 4.779 0.000 459.169 0.000 /usr/lib/python2.7/dist-packages/twisted/internet/base.py:762(runUntilCurrent)
17987582 4.687 0.000 4.687 0.000 {setattr}
2807093 4.630 0.000 223.412 0.000 p2pool/util/pack.py:41(_unpack)
438995/222789 4.491 0.000 1064.895 0.005 p2pool/util/variable.py:35(happened)
3928877 4.374 0.000 9.756 0.000 p2pool/util/pack.py:134(write)
1 4.369 4.369 9605.102 9605.102 /usr/lib/python2.7/dist-packages/twisted/internet/base.py:1172(mainLoop)
336129 4.218 0.000 37.090 0.000 {method 'update' of 'dict' objects}
11691309 4.045 0.000 4.045 0.000 {isinstance}
3042144 4.038 0.000 5.191 0.000 p2pool/bitcoin/data.py:217(target_to_average_attempts)
5804151 3.976 0.000 23.636 0.000 p2pool/data.py:367()
That indicates 7667 seconds used by the poll() function inside twisted, of about 9606 seconds total for the whole program.
I think I might try a
try:
from twisted.internet import epollreactor
epollreactor.install()
except:
print "twisted.internet.epollreactor could not be loaded. "
print "Falling back to pollreactor. Performance may suffer."
from twisted.internet import reactor
or something like that. It will probably take a little more than that, as I don't know if the APIs are compatible.
The biggest performance improvements are usually algorithmic, not language-based. You shouldn't change languages until you are pretty sure that you have the algorithm down. There are some big algorithmic improvements still to be had in p2pool, so I don't think the time is yet ripe for a rewrite in C.
(However, there are quite a few modules (mostly in the utils directory, like utils/math.py and utils/pack.py) that would make for straightforward C modules that could be called by python. But more on that later, maybe.)
edit: changed the code above. It seems to be working now. I don't know if the performance is improved yet. I'll check back in a few hours. This epollreactor thing will only run on Linux >= 2.6, by the way.
