Pages:
Author

Topic: [1500 TH] p2pool: Decentralized, DoS-resistant, Hop-Proof pool - page 30. (Read 2591916 times)

hero member
Activity: 818
Merit: 1006
jtoomimnet (via the 1mb_segwit branch) follows segwit for the next two months. I am working on code that will make jtoomimnet split into a segwit chain and a segwit2x chain automatically once the fork happens. This code will also be required for supporting Bitcoin Cash, which is my immediate goal.

Supporting Bitcoin Cash is likely to require some p2pool performance improvements as well, as the current codebase struggles with 1MB blocks, much less 8MB. I hope to have Bitcoin Cash working by the end of the weekend as long as you have a high-end server. I'll try to get the needed performance improvements afterwards.

Quote
I will be forced to chose for segwit2x anyway?
It is not my goal to force anyone to do anything. As long as you are aware of the risks of mining on a minority chain, you are welcome to do so. I don't want people to mine on the minority chain due to ignorance, but as long as people know what they're doing, it's fine with me.
newbie
Activity: 27
Merit: 0
Go back and read here this might help. https://bitcointalksearch.org/topic/m.19006724

Ok, thank you. This solved the problem.

I've got another question about p2pool. I'm fairly new to mining using p2pool, so the current situation with the different forks is a bit confusing. As I understand it from the posts on this forum and the output of jtoomim's 1mb_segwit p2pool, jtoomim's fork follows segwit2x, which will be a hardfork of the bitcoin blockchain in november. I'm in favor of segwit but not of segwit2x. So I would rather not mine on a p2pool fork that supports segwit2x. As I understand it the current version of forrestv p2pool's does not support segwit. That's the version I use currently for mining. However I see that veqtrus wrote a commit that supports segwit but not segwit2x. So my question is the folowing. Is there a plan in the foreseeable future to merge forrestv's version and veqtrus's commit, so that I just need to do a git pull to have a a p2pool version that supports segwit?  Or, is forrestv's version going to merge with jtoomim's version, so that I will be forced to chose for segwit2x anyway?

Thank you.
newbie
Activity: 45
Merit: 0
Dear community! And esp MR Jtoomim!
I bet i'm not alone in willing to have a BCH (or BCC) sort of fork of P2pool. Is it real? I am not a coding person, but BCH is not far from the original BTC and maybe it is not so big work to do?
Join in the keke51. The complexity of the network BCH falls, the price increases. Ideal conditions for mining BCH. Really need P2pool BCH.
newbie
Activity: 5
Merit: 0
Dear community! And esp MR Jtoomim!
I bet i'm not alone in willing to have a BCH (or BCC) sort of fork of P2pool. Is it real? I am not a coding person, but BCH is not far from the original BTC and maybe it is not so big work to do?
hero member
Activity: 818
Merit: 1006
I will be upgrading all of my nodes to 1mb_segwit tomorrow. I suggest that anyone mining on jtoomimnet switch to the 1mb_segwit branch ASAP.
hero member
Activity: 818
Merit: 1006
I'm working on adding fork-friendliness to p2pool now. It's a bit more complicated than I thought at first. You can't just check to see if a share's parentblock is in bitcoind's blockchain and mark the share invalid if not, because orphaned blocks are a thing. Instead, I will be trying to keep a penalty score for each share, where each share's penalty is the previous share's penalty plus 0 if the parentblock is found in the blockchain else the work done if the parentblock is not in the blockchain (i.e. orphan or invalid). P2pool will then look for the share with the greatest (work - penalty). That description is slightly oversimplified, but I hope it gets the main idea across.
sr. member
Activity: 351
Merit: 410
You'll also need to install and run your bitcoind of choice (I recommend btc1, because I think segwit2x is the future of Bitcoin) and edit ~/.bitcoin/bitcoin.conf with your bitcoind's RPC username and password (or pass it to p2pool from the pypy run_p2pool.py command line)...

In addition, I highly recommend adding the following two lines to your bitcoin.conf:

Code:
blockmaxsize=1000000
blockmaxweight=4000000

This enables your P2Pool node to mine full blocks. I'm not sure if btc1 changed its defaults, but on Bitcoin Core the default blockmaxsize and blockmaxweight values are 750 kB and 3 MB respectively, meaning that if you left your bitcoind to its defaults, your P2Pool node would be mining artificially smaller blocks.

If you decide to use the btc1 (SegWit2x) client, simply double the above values after the 2x hard fork.

Also, I highly recommend connecting your bitcoind to the public FIBRE network. This helps your bitcoind to be notified of new blocks as quickly as possible. And if your P2Pool node finds a new block, this helps propagate the newfound block to the Bitcoin network as quickly as possible, reducing the chances of that newfound block being orphaned.

Instructions for connecting your bitcoind to the public FIBRE network can be found here.



I recommend btc1, because I think segwit2x is the future of Bitcoin...
Is it its source code?: https://github.com/btc1/bitcoin

Yes.
legendary
Activity: 1308
Merit: 1011
I recommend btc1, because I think segwit2x is the future of Bitcoin...
Is it its source code?: https://github.com/btc1/bitcoin
hero member
Activity: 818
Merit: 1006
I tried installing pypy/p2pool on Windows a try myself, and while I got a bit farther than you did by installing twisted from source, I eventually ran into a roadblock because twisted requires pywin32 on Windows and pypy does not support pywin32.

However, I had better luck when I tried to run p2pool with pypy on the Windows Subsystem for Linux (Ubuntu for Windows). If you have Windows 10, I suggest you give that a try:

https://msdn.microsoft.com/en-us/commandline/wsl/install_guide

Once you have Ubuntu for Windows installed, run bash and then copy-paste these lines into your bash command line:

Code:
sudo apt-get update
sudo apt-get install pypy pypy-dev pypy-setuptools gcc build-essential git

wget https://bootstrap.pypa.io/ez_setup.py -O - | sudo pypy
sudo rm setuptools-*.zip

wget https://pypi.python.org/packages/source/z/zope.interface/zope.interface-4.1.3.tar.gz#md5=9ae3d24c0c7415deb249dd1a132f0f79
tar zxf zope.interface-4.1.3.tar.gz
cd zope.interface-4.1.3/
sudo pypy setup.py install
cd ..
sudo rm -r zope.interface-4.1.3*

wget https://pypi.python.org/packages/source/T/Twisted/Twisted-15.4.0.tar.bz2
tar jxf Twisted-15.4.0.tar.bz2
cd Twisted-15.4.0
sudo pypy setup.py install
cd ..
sudo rm -r Twisted-15.4.0*

git clone https://github.com/jtoomim/p2pool.git
cd p2pool
git checkout 1mb_segwit

Those commands should also work if you're on native Ubuntu or Debian Linux

You'll also need to install and run your bitcoind of choice (I recommend btc1, because I think segwit2x is the future of Bitcoin) and edit ~/.bitcoin/bitcoin.conf with your bitcoind's RPC username and password (or pass it to p2pool from the pypy run_p2pool.py command line), then from your p2pool directory, run

Code:
pypy run_p2pool.py
newbie
Activity: 43
Merit: 0
The file cannot be accessed because it is being used by another process? Maybe something is running which is using that file, or maybe it crashed uncleanly and left that file open. Try closing everything, rebooting, and trying again?

Reboot done. same problem. i also retryed to install it again manualy by downloading twisterd-17.5.0.tar.bz, extracting direkt to pypy directory and running setup.
here is the output :
Code:
C:\pypy2-v5.8.0-win32>pypy C:\pypy2-v5.8.0-win32\Twisted-17.5.0\setup.py install
Traceback (most recent call last):
  File "C:\pypy2-v5.8.0-win32\Twisted-17.5.0\setup.py", line 17, in
    with open('src/twisted/python/_setup.py') as f:
IOError: [Errno 2] No such file or directory: 'src/twisted/python/_setup.py'

C:\pypy2-v5.8.0-win32>pypy C:\pypy2-v5.8.0-win32\Twisted-17.5.0\setup.py
Traceback (most recent call last):
  File "C:\pypy2-v5.8.0-win32\Twisted-17.5.0\setup.py", line 17, in
    with open('src/twisted/python/_setup.py') as f:
IOError: [Errno 2] No such file or directory: 'src/twisted/python/_setup.py'
interesting here ist that "src/twisted/python/_setup.py" is there ... is not missing !?
but wath du tell me these line : "with open('src/twisted/python/_setup.py') as f:"  i do have no disk drive f:
hero member
Activity: 818
Merit: 1006
The file cannot be accessed because it is being used by another process? Maybe something is running which is using that file, or maybe it crashed uncleanly and left that file open. Try closing everything, rebooting, and trying again?
newbie
Activity: 43
Merit: 0
oky.

so,
1st. I cloned pypy. Downloaded it, extraktet and place it on C:\pypy2-v5.8.0-win32\. Ther was nothing to install, only the pypy.exe.
2nd. I Downloaded get-pip.py and installed it using "pypy.exe get-pip.py". This automatik installation was sucessfull.
     Command "pip list" shows following installet packets now :
     
Code:
certifi (2017.7.27.1)
cffi (1.10.1)
chardet (3.0.4)
greenlet (0.4.12)
idna (2.6)
pip (9.0.1)
readline (6.2.4.1)
requests (2.18.4)
setuptools (36.2.7)
urllib3 (1.22)
wheel (0.29.0)
zope.interface (4.4.2)
3rd. i would like to install Twisted by automatic installation with pip using "pip install twisted"
   that returns :
Code:
C:\pypy2-v5.8.0-win32\bin>pip install twisted
Collecting twisted
  Using cached Twisted-17.5.0.tar.bz2
    Complete output from command python setup.py egg_info:
    no previously-included directories found matching '.travis'
    no previously-included directories found matching 'tests'
    warning: no previously-included files found matching 'examplesetup.py'
    no previously-included directories found matching 'src\exampleproj'
    no previously-included directories found matching 'src\incremental\newsfragments'

    Installed c:\users\x\appdata\local\temp\pip-build-l8fwso\twisted\.eggs\incremental-17.5.0-py2.7.egg
    Traceback (most recent call last):
      File "", line 1, in
      File "c:\users\x\appdata\local\temp\pip-build-l8fwso\twisted\setup.py", line 21, in
        setuptools.setup(**_setup["getSetupArgs"]())
      File "c:\pypy2-v5.8.0-win32\lib-python\2.7\distutils\core.py", line 111, in setup
        _setup_distribution = dist = klass(attrs)
      File "c:\pypy2-v5.8.0-win32\site-packages\setuptools\dist.py", line 325, in __init__
        self.fetch_build_eggs(attrs['setup_requires'])
      File "c:\pypy2-v5.8.0-win32\site-packages\setuptools\dist.py", line 446, in fetch_build_eggs
        replace_conflicting=True,
      File "c:\pypy2-v5.8.0-win32\site-packages\pkg_resources\__init__.py", line 855, in resolve
        dist = best[req.key] = env.best_match(req, ws, installer)
      File "c:\pypy2-v5.8.0-win32\site-packages\pkg_resources\__init__.py", line 1127, in best_match
        return self.obtain(req, installer)
      File "c:\pypy2-v5.8.0-win32\site-packages\pkg_resources\__init__.py", line 1139, in obtain
        return installer(requirement)
      File "c:\pypy2-v5.8.0-win32\site-packages\setuptools\dist.py", line 518, in fetch_build_egg
        return cmd.easy_install(req)
      File "c:\pypy2-v5.8.0-win32\site-packages\setuptools\command\easy_install.py", line 672, in easy_install
        return self.install_item(spec, dist.location, tmpdir, deps)
      File "c:\pypy2-v5.8.0-win32\lib-python\2.7\contextlib.py", line 24, in __exit__
        self.gen.next()
      File "c:\pypy2-v5.8.0-win32\site-packages\setuptools\command\easy_install.py", line 636, in _tmpdir
        os.path.exists(tmpdir) and rmtree(rmtree_safe(tmpdir))
      File "c:\pypy2-v5.8.0-win32\site-packages\setuptools\command\easy_install.py", line 2241, in rmtree
        return shutil.rmtree(path, ignore_errors, onerror)
      File "c:\pypy2-v5.8.0-win32\lib-python\2.7\shutil.py", line 247, in rmtree
        rmtree(fullname, ignore_errors, onerror)
      File "c:\pypy2-v5.8.0-win32\lib-python\2.7\shutil.py", line 252, in rmtree
        onerror(os.remove, fullname, sys.exc_info())
      File "c:\pypy2-v5.8.0-win32\site-packages\setuptools\command\easy_install.py", line 1689, in auto_chmod
        return func(arg)
    WindowsError: [Error 32] Der Prozess kann nicht auf die Datei zugreifen, da sie von einem anderen Prozess verwendet wird: c:\users\x\appdata\local\temp\easy_install-jqgatn\incremental-17.5.0\README.rst

    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in c:\users\x\appdata\local\temp\pip-build-l8fwso\twisted\

i also tested to install twisted by downloading the scource, extract it into the lib directory and tried to install it with command "pypy "c:\pypy2-v5.8.0-win32\libs\twisted\setup.py install" the Error outputs was nearby the same.
does someone have an idea wath i do wrong ?
hero member
Activity: 818
Merit: 1006
Post your error so that others can learn from it. You probably aren't the last one who will want to get pypy running on Windows.

You might be using the version of pip bound to python2.7 instead of the version of pip bound to pypy. You may have to install setuptools for pypy (I'm not sure it comes installed by default, unlike Python), or install twisted via a mechanism other than pip or easy_install.
newbie
Activity: 43
Merit: 0
i dont yell to any one Cool. Because i am thanksfull you talking to my an spend me these time.
i will switch to pypy

by the way, i dont think about these Multicore think  Grin

Edit 18.08.17 - 6PM :
p2pool on pypy is not able to me  Cry. i dont get twistet installed to pypy.
pip is working but i get many error until "pip install twisted"
-> so, if somone does have time to help me please send me an PM (dont want to spam in here)
hero member
Activity: 818
Merit: 1006
You have 2 physical cores, 4 logical cores on your CPU. P2pool is single-threaded, so it can use at most 25% of your CPU (in terms of logical cores). Your CPU is pegged at a little over 25% for the Pythonw (p2pool) process. Your CPU is unable to keep up. Try pypy.

Keep in mind that utilization is a misleading metric. What really matters is latency -- that is, how long, in milliseconds, it takes p2pool to process an incoming share and hand out new work to workers. On my nodes, that's normally around 50 ms to 150 ms on pypy, and 250 ms to 1000 ms on Python. With an average of 30 seconds per share, latency of 1000 ms corresponds to a 3.3% DOA rate. If your CPU is pegged at 100% for one core on the p2pool process, that means that work is queueing up, and before you can process an incoming share, you have to process all the other data that has been flung at your p2pool node before it can process the recently-broadcast share. This can turn 1000 ms latency into 3000 ms latency.

You can think of it like a leaky bucket: if you have a bucket with a hole in the bottom and a faucet pouring into it, you can measure how long it takes for any given drop of water leaving the faucet to make it through the hole in the bottom of the bucket. If the hole is bigger than the faucet, then there will never be much water in the bucket, and the amount of time it takes for a drop to pass through will be small. But if the faucet is bigger than the hole, the bucket will fill up to the top and start to overflow the sides, and it will take very long for drops to make it through the hole, and some drops will never make it at all. Your bucket is overflowing. Some data that gets sent to your node never gets processed, and the data that does gets processed after a substantial delay.

The high difficulty, on the other hand, is normal for p2pool, and not responsible for your high DOA rates. Actually, since you're CPU bound, you would get slightly (but not significantly) better performance with an even higher difficulty, since that would mean your miner would send fewer shares to your node and require less processing on your node. But again, that's not significant.

Go switch to pypy and it should work fine. Or yell at me and get me to make the code faster.
newbie
Activity: 43
Merit: 0
Realy CPU usage looks ok to me :
following datas are Peak (i cat this Hardkopies on peaks)
http://www.c-xantus.de/syslast1.jpg
http://www.c-xantus.de/syslast2.jpg
http://www.c-xantus.de/syslast3.jpg

more strange vor me is this dificult think :
http://www.c-xantus.de/s7.jpg

but i will test pypy at this weekend, to test it.
hero member
Activity: 818
Merit: 1006
Xantus, I see a very high DOA rate on your node, roughly 50%. I suspect your CPU is not fast enough to keep up with the task. You may want to try switching to pypy instead of using Python 2.7. Pypy's CPU usage is about 3x lower than Python 2.7's, although its memory consumption is about 3x higher. I haven't installed Pypy on Windows before, but I expect you'll need to reinstall twisted, zope.interface, and whatever other things you had to install with Python 2.7.

Sorry for not mentioning this earlier; I should have predicted this issue and sent you down the pypy path from the start.

Once I get caught up with getting my p2pool code to support all forks and altcoins, I'll work on cutting down CPU, RAM, and bandwidth requirements for running a node. For the time being, though, we'll need to meet p2pool's high resource requirements with pypy and mid-range hardware.

Other than that, it looks like your node is working.
newbie
Activity: 43
Merit: 0
I've pushed two changes to https://github.com/jtoomim/p2pool/commits/1mb_segwit:

1. Allow mining with clients that do not support all locked-in or active forks (e.g. Bitcoin Core which does not support segwit2x) if p2pool is run with the --allow-obsolete-bitcoind command-line option.
2. Switch back to using time.time() in order to hopefully fix Xantus's "exceptions.OverflowError: mktime argument out of range" error on Windows

Thanks jtoomim,
i downloaded an started now  (before go to work). up to now it looks fine.
lets see this evening ...

update : 9:45pm :
still runs, shares are downloaded i think :
Code:
P2Pool: 17707 shares in chain (10554 verified/17712 total) Peers: 12 (6 incoming)
-> is that ok that only a part is verified ?

oh, and the Local hash rate lock a little bit strange to me, seams because the difficulty for mining engines is very high.
Isn´t that an dynaminc think vor every node his selv harshrate ?
but im on the right p2pool part an back online again ;-)   - thanks so far !
->http://131.117.146.223:9332/static/
hero member
Activity: 818
Merit: 1006
What are the downsides to increasing the PPLNS windows/length of the sharechain?
For p2pool specifically, or for pools in general? I'm assuming the former.

P2pool needs to store all of the shares in the sharechain plus a few. If the sharechain is 8460 shares long, this means that p2pool needs to store at least 8460 shares in its sharechain. In practice, p2pool stores about 2x as many shares as that, so the sharechain length is typically around 16,000 shares in length. Each share references each transaction in the block that that share would have made, using somewhere between 2 bytes (in my lowmem branch for transactions that have been included in previous shares) and 70-ish bytes (for new transactions, including Python's object overhead). With roughly 2000 transactions per share, that turns into about 10 kB (lowmem) to 50 kB (p2pool master) per share, or 160 MB (lowmem) to 800 MB (master) of memory consumption. Multiply those numbers by something like 4 if you're using pypy.

There are also some minor CPU usage costs on sharechain size when processing shares (e.g. calculating payouts for the coinbase), but they're mostly alleviated by using a skiplist in the work-done-by-user calculations.

I would like to strip all transaction data out of the share data structure in the share chain in order to cut this memory footprint issue and reduce the CPU requirements for processing shares, but until that is done, increasing the share chain length is a bad idea.

The coinbase transaction would not change much in size for typical p2pool use because it almost always contains at least one share per active p2pool user anyway. A single ASIC miner is generally able to mine more than one share per 3 days (the current PPLNS length target).

The coinbase transaction size is not currently a significant problem for p2pool, as it only takes up a few kB once every few weeks. If p2pool grew bigger and if blocks got more frequent, it could make sense to pseudorandomly determine which users get paid out from each block, or to keep a separate balance state for each user that is independent from the payouts, and make payouts only when that balance state exceeds some reasonable threshold.

Is there somewhere to check whether larger miners are setting their share difficulty too high? That it negatively affects small miner's variance.
You can go onto a p2pool node and use the share chain explorer to look at different shares, the difficulty used for that share, the mining address that created it, and the minimum difficulty that p2pool would have allowed for that share if the miner had chosen a low-diff share.
Pages:
Jump to: