Topic: An enhanced alternative p2p “pool” – a Think Tank. (Read 3175 times)

Indeed it is. I've been really busy lately & have not been able to spend much time with it, but hopefully that will change within the next few days. I'm still looking for a programmer/coder with balls of steel to help me get this started - so if you are that person, or you know of someone - please drop me a line.....

Ah! This sounds promising - is this project still in the mix?....

+1.
Add something like zeromq on top of that to reduce the network/block latency/propagation delays and we'd have a winner.

How about turning this into a framework, for not only p2p mining, but also the coin network too.

Why do we need a separate bitcoind and litecoind daemons, when they could become plugin's.  Most coins need mining pools, so roll it into one app.

It would make creating new cryptocurrancies, much easier.

This is one area for potential improvement over p2pool, but I believe it would have to be included in all clients.  Otherwise, you wind up with p2pool style merged mining where you only get the merged mining rewards if it is your node that solves the p2pool block.

Hi FlappySocks,

Funny you should say that, I had a long meeting/discussion yesterday with a potential coder and this subject was bought up. Although my first intention was to make the software Bitcoin only to start with, but with having provisions for adding other alternative coins when demand/time allowed, we toyed with the idea of maybe developing the software as a stripped down "bones" program so that the user can add which ever add-on they required to mine their preferred coin. This has never been done before and opens a whole new kettle of fish, but theoretically it makes great sense as it would help keep the program light, the user would only need to download the add-on he/she required for the coin they wanted to mine, as well as making it easier for the devs to implement updates to the add-ons as and when needed, which the user would pull when required. I like this idea as it is both ground breaking & serves a purpose, but more discussions are required before making a decision as it is a bold move. I will update the starter thread with this though.
Been going over some names for the software too lately and am hoping for some suggestions from you lot. A couple of the best so far are Multipeer & Multicoinp2p, but am totally open to this, so feel free to put a name forward - you might even win a prize.......

That does seem a little more reasonable

Still, it takes a serious chuck of developer man hours to put something as small as 22k lines of code together.

FYI : BFL 480 Gh/s hashing daemon with cgminer up and running
https://bitcointalksearch.org/topic/m.2514804

edit : correction - p2pool is more than 22000 lines of code (and more than 8e5 characters)

Please can we have a plugin architecture to support different coins.  I have Bitcoin & p2pool running on a server, and today I just added Litecoin, which turned out to be less than easy.  Ports clashing, etc. I also intended to add one of the new experimental cryptocurrencies, but Iost patience, and will try another day.

Very confusing to have command line options & console output saying "bitcoin", when it's litecoin.

Ideally, I would like to see the new p2p pool software advertise to it's connected nodes, which coins it supports.  And then the nodes know which byte streams to share. No need for different ports set up for each coin.

You can join in the discussion here: https://bitcointalksearch.org/topic/poolyrralnet-p2pool-backed-mining-pool-alpha-234841

It started off just trying to get testers for my proxy which is written in Ruby.  Someone else came along and is looking at integrating it directly in p2pool using Python.  I really don't care if my thread devolves into a generic p2pool proxying discussion, so feel free to try and drum up support for a version in Go.  In the short term, I'm primarily interested in bringing down the block times for p2pool, and proxying is simple enough that having multiple approaches should help us all to understand the problem space better.

If anyone thinks I'm being hypocritical after my hard stance on not duplicating p2pool, as .m. points out, p2pool is 8e5 lines of code and my proxy and logging is providing all the functionality I need to operate a pool and is sitting at 185 lines.

Hi all, I have found this thread googling p2pool + golang
I believe "notme" has a good understanding and I like his (proxy) proposal.
Is there any publicly available info about "Mhash/sec at miner's side" per "actual reward" for p2pool compared to other pools ?
If there is not clear advantage, I would not throw more than 8e5 lines of python code away, not mentioning I do not understand much of it.
But I would certainly enjoy going step by step and trying implement and possibly improve little pieces in google GO (asi it is well designed,compiled,clean,safe,concurrent .... language - try to play/edit/run/learn at tour.golang.org).
And that will improve lags/problems with "more miners" or "high Ghashes" connected to p2pool node as well.
I understand the urgency to allow connecting ASIC miners to p2pool. Avalon is mentioned not working,BFL not working(sufficiently) neither, but may improve soon : https://forums.butterflylabs.com/bfl-forum-miscellaneous/2164-demo-unit-forrestv-p2pool.html#post27922
Bitfury has no devices yet, KnC miner as well, ASICminer is mentioned to work well ("notme").
But - all vendors are (or will be) selling chips and there are many HW design threads and there is no reason not to support p2pool well.
I have no chips at my posession, but they can be simulated with many GPUs.
There is already an effort to replicate bitcoind and a client in golang.
So I will be happy to help with 'proxy' effort - but probably we shall create a new thread ?
Good luck to everybody !

Meni Rosenfeld derives something different here: http://bitcoin.stackexchange.com/questions/4953/how-will-a-massive-increase-in-hashpower-affect-orphan-rates/4958#4958

Your ideas are not bad ideas at all - they're just not applicable to what I want to do. Good luck. 

I get that, I just don't see any ideas or suggestions here.... just talk about needing such ideas/suggestions.  And all of my ideas are bad in your opinion because I don't want to reinvent the wheel.

Anyway, I'm going to go back to being productive.  Best of luck to you two.  Hopefully you can find some other people to join in your conversation who are more enthusiastic about duplicating effort.

I'm sorry, I'm not sure you get my drift. I'm doing this project for many reasons, some of them being: because I think it's a challenge, it will be fun, it will benefit mine & everyone involved knowledge & understanding of Bitcoin & p2p networking, because I want it to be better, because I believe it will benefit Bitcoin & mining, because I want to & because I can. This project is not about putting any pool out of business, it is about joining them in business with healthy competition which benefits Bitcoin & miners alike. It will be in competition with all pools, not just p2pool - all pools.
I started this thread to gather ideas & suggestions from the community, completely open & transparent to everyone who would like to get involved, it is not a "why" or "what for" thread - if anything it's a "why the hell not" thread, and the people who I'm interested in hearing from & possibly working with are the "why the hell not" type of crowd, plenty of them on this forum. I want to start from afresh & build it from the ground up because that is what I choose to do, in the knowledge that it has been 100% developed by the guys behind it - I don't believe I need a justification or reason to do this. On the contrary, it would take or need to be an exceptional reason for me NOT to do this.
Again, I wish you the best of luck with what you are trying to do, I hope it works for you. Of course, you are always welcome to give some positive input any time in the future if you wish. I'm open to everything - hence this thread.

Peace 

How do you intend to do it without a proof of work chain?  You have to have a way of reliably measuring the work of the various pool contributors.  All I've seen is complaints about p2pool and no new ideas other than the one I came up with, which seems to be a non-starter anyway.


I've grown up with the open source ethos.  It's about community and building on the work of others.  Reinventing the wheel seems pointless to me.  Perhaps you are right that a better system can be built from the ground up.  But I can't see it.  What I do see is a clear path from what is working today to what we want in the future.  So again, I ask you to lay out a path from 0 to your goal.  I've already laid out one from p2pool to your goal.

I'm sure nobody wants to replace any pool, I just want to make a fully functional & efficient p2p pooling system. History shows that pools come and go - if a pool shuts down it is not because of other pools - it is because it was unable to provide it's miners with a good service that can compete with other pools. Ultimately, it is miners who decide weather a pool operates or not, not other pools. Sure, development is not going to be an easy task, I made that clear at the start, but it's also not as hard as many people believe it to be if it's done properly with the right programmer/coder. On a personal note, if I were a coder and was asked to develop such software, I would certainly want to develop it myself from the ground up rather than try and rectify someone else's programming with patches, workarounds and proxies. But that's just me.

I can't see how you're going to do that if you are going to keep the mine-on-top-of-longest-chain system.   If you're going to do a new p2p pool, I think the fundamental design should be changed.  But feel free to convince me otherwise

Now that I've had some time to think about it I'm convinced my idea for equalizing variance won't work without punishing new address when they first show up on the network.  Without such a probationary period, people could mine to as many separate addresses as they want.  Even with a probationary period it might make sense to mine to several addresses if we don't get the incentives quite right.


Let me be clear.  I'm not here to replace p2pool.  I believe that most of the problems people have with p2pool are possible to overcome without changing the fundamental design.  My posts are to help you guys understand the technicalities of what you are attempting to do.

Completely agree Kinlo, as I said earlier - a fresh start from the ground up is required. I appreciate your input Kinlo, it's always good to hear from someone who thinks the same way & realises what is needed - please do keep the ideas & suggestions coming - the more input from the bitcoin community the better chance there is of developing a successful p2p alternative. 

The reason I've been making all my posts are simple: My message to you is think outside the box:  Do not make just another bitcoin clone.  p2pool did it, and it works fine but we see it has some drawbacks.  If you're going to create another implementation of the same basic idea, you're going to end up with the same problems.  I am not telling you I know the answer.  This is a difficult question to solve.  But if you want to start an alternative to p2pool, make sure you use a totally different way of thinking so you will have a product that doesn't work the same way so you have your own advantages (and perhaps disadvantages) over p2pool.

Hmmm. Maybe some kind of basic self check feature could be incorporated into the software that could test connections to both the user node and a remote node - giving simple connection results & maybe advising the user of potential connection issues? This would also be of great benefit to the less experienced user/beginner as well as having the benefit of less connectivity questions on the support thread.....User friendliness was one of my original suggested requirements......

EDIT: Also, the more users who have stable/optimal connections the better it is for the network.

Those who get the shares first will always have an advantage.  I'm not sure it is possible to remove this advantage unless you somehow punish peers who have low latency.  Increasing the share time to 30 seconds would address the other concern.
But, yes, it is harder to maintain a sufficiently connected p2pool instance.  It is really easy to be sufficiently connected to a centralized pool or a particular p2pool instance.

I think I agree with you there Kinlo, this is one of p2pools current drawbacks for sure. I'd like to see this p2p application fall in line with all other pools in this regard, if it works for them it should work for us.

I'm afraid this is not true.   Regular pools give you 30 seconds to submit a share.  p2pool uses the first share it finds of a certain difficulty, and you have to build on top of that.  So you need to race to be the first in p2pool in rounds that last 10 seconds.  You have 30 seconds on a regular pool and you don't have to race any peers. 

p2pool is greatly affected by your connectivity, regular pools are not

Yes, that’s a very interesting & fresh approach that could definitely lead somewhere. I also like the idea of keeping the field playable for everyone – it’s very in line with the p2p ethos. This is exactly what I meant by the software being able to adjust, smart software so to speak, the payout system would be easy enough to work out I’m sure. Good thinking.

I don’t think it’s a case of throwing out p2pools software, more like letting it do its own thing, and learning from its mistakes/inabilities. Like I said, competition is healthy. It simply doesn’t have the scalability or flexibility “built in” to its foundations to allow it to adjust to new conditions/hardware. It was great back when gpu’s were all there was, in fact it was the best, but things have changed since then and software has to be able to change with new technology – future proof so to speak. Over the last few months I’ve seen many experienced coders/developers try and fail to get it to play nicely with various hardware – even a couple of well known & respected miner devs had a try at it with no joy. This to me screams of “build a new, better & updated one”. I know at least one of them has been asked to do this before, but did not have the time or inclination to do so, I hope he’s reading this; maybe he could give us some helpful info……(hint hint)
All options have to be looked into, there is always room for improvement & new ideas.

Notme, what is your preferred choice of code - both generally & suggested for an application like this?

Contradicting myself, one possibility if we did throw out p2pool would be to have the pool hand out per address difficulty requirements and then ensure the share rate from that address matches what they expect.  Then we allow a constant number of shares per address per time, regardless of how much hash power is producing it.  If course, there will be variance in the share rate, and that means that it we will have to use stochastic models to "ensure the share rate matches what we expect".  This might leave the payout system open to some sort of gaming, but I need to think about it some more.

This would result in every participant seeing the same variance no matter their size, but every new participant raises the variance for everyone.

If you are badly connected to any pool you will have many stales.  I still don't see how p2pool is any worse in this respect.  I'll concede it has problems with SOME ASICs.

Fundamentally, it comes down to a tradeoff between high stales and high variance.  If you slow down the share rate, it is much harder to find a share and variance will go through the roof.  In order to keep everyone happy from big miners to small, we need a tiered system.  This is actually what I am working towards with my proxy project, but you seem to want to throw out everything already built and start from scratch rather than work incrementally towards a system that suits everyone.  Yes, I want the lower tiers to eventually be decentralized, but I need to work out the kinks in just proxying to an unmodified p2pool for now.  Ultimately, we need to allow multiple addresses to take credit for p2pool shares rather than a single address per share.  Once this is in place, we can have second tier pools that submit p2pool shares that pay out based on their easier share chain.  A centralized proxy is just a first step towards this vision and will help us understand a part of the target system that is achievable today.  Then there is the issue of BFL ASICs.  Once we have a tiered system, we will be able to lengthen the share time on p2pool to something acceptable for all hardware and the second tier pools can provide easier shares and lower variance.

I still don't understand why we should throw the baby out with the bathwater when there is a clear path to achieving the variance and stale share characteristics we desire for every size miner.

Exactly - more shares = more payout, this is mining, and is why all other pools are more successful. If the new p2p program is to compete & survive then it has to at least match what other pools are offering as standard, but preferably better them. We all know that statistics show that p2p has the highest payout of most pools over time, so if we can add to that the lost/stale/doa/orphaned percentage of shares then p2p will undoubtedly flourish. These are high targets of course, but it's what is necessary to be successful. As for the program language, I think ultimately this would be at the discretion of the programmer that him/herself is most competent & comfortable using, I quoted c or c++ simply because it was the most widely used/accepted code, but as with everything I'm open to ideas & options, although to be honest, so far I have had bad experiences with python - but this could be simply because of the way it was written. I'll edit the original post to reflect the changeability.

I like the direction this is going, more input please........

I don't agree.  Yes indeed, if the block is "stale" for p2pool it can still be a valid bitcoin block.  Therefore this particular issue doesn't hurt p2pool in regards to what the pool will earn in bitcoin (altough the 2nd post I did should influence)

But regardless, if you are badly connected to p2pool, you will have many stales, so yes, the bitcoin blocks you find will not be punished by that effect, but the p2pool shares will be.  And if you can find less p2pool shares, then you will automatically earn fewer coin....

+1 for go   I'm using it a lot too, including parts of my pool.

In any case, I don't think there is a big issue with python.  As long as it is properly packaged, people don't really care.  C or C++ would just slow down development imho

why not use go lang, I think it is a good web service program language.

ASICS hardware is not compatible with p2pool as it stands without a patch/workaround - and it's far from optimal, leaving the user with a less efficient & very expensive miner. This is why people don't use them on p2pool, and why I want to incorporate ASICS & every other hardware setup into the program, there should be no need to have to hack it or apply patches to make it "sort of" work - it should just work once the program knows/detects what is connected to it. And, it should work properly, adjusting accordingly. This is what the aim should be, as I mentioned in the first post, otherwise we will be going the same way as the other p2p program and will achieve nothing with the same results. The only way to achieve this is to change the programming and think differently, a new approach is needed, and a programmer who is capable of thinking slightly out of the box will be able to do this I'm sure. If it is to be successful then it must be compatible with everything, no compromises.

This can be done, with the right people & attitude.

As for "p2pool's high "stales" actually don't hurt payout one bit" - of course it does: 100% stales = 0 payout, 0% stales = 100% payout - adjust accordingly. It might not matter so much with p2p - but it still matters. If other pools can have 99.8% then so can p2p. Miners don't want to spend 30,000 on the latest rig only to see 25% of their investment go up in stales straight off the bat- 25% of stales is 25% lost revenue, no matter what way you look at it.

p2pool's high "stales" actually don't hurt payout one bit.  If a block is found, but it is "stale" for p2pool, it is still a valid bitcoin block.  It will still be broadcast and p2pool miners will earn the reward.  As long as you don't have more stales than the average p2pool miner you will not lose because of the higher level of stale p2pool shares.

There is however an issue with BFL miners because they can only change work once every 5 seconds.  From what I've read the other ASICs work fine with p2pool and I can confirm no issues with ASICMiner hardware.

p2pool uses very short rounds.  Every time a round is started, all current work is going to be basically useless, and your miner needs to restart all work. 

I understand especially some asics have big issues with that.   longer rounds should reduce the negative influence greatly, and should actually increase the hashrate of your miners, as you can continue mining for longer periods, without having to forcefully restart your miner.

One of the "problems" with bitcoin is block propagation.  Once a block is found, every other node needs to receive the block and mine on top of it.  If you don't mine on top of the longest chain, you will loose, and basicly, your block will become stale (sometimes called orphan)

To calculate the chance of this happening - anyone can jump in and correct my math: we have the following formulla: p/i

p would be the time it takes to propagate a block over the network, and i would be the interval time.  If we look at bitcoin, the value for i would be (on average) 600 seconds, as every 10 minutes a block is found.   Let's say it takes 5 seconds to propagate over the network: we would have 5/600, which means 0.8% of the time you are mining, that you are at risk of not being on the longest chain, and that you will create orphans, and if you find a block the same happens in reverse, someone else might build upon your block and causing your block to have a chance to be orphanned.

p2pool uses the same algorithm as bitcoin, only in a smaller version, their "blockchain" has an interval of 10 seconds.  Since the network is smaller, the propagation times are lower/better, but still, if it would take 1 second, you're down to 10% chance of finding a stale.

So unless my math is wrong, it is clear that shorter rounds will lead to higher stales.

So, anything "new" should find a way to use longer rounds inside the p2p network to make sure the stales are reduced.

Hey notme,

Good to hear from you. I don’t want to bash p2pool, far from it – I’m a fan & have been using it for nearly 18 months up until a couple of months ago, when I left due to the lack of support for the next generation of mining equipment, loss of work, badly implemented stratum support & general lack of development. Me & other miners there asked for many months for these issues to be addressed but to no avail, so it’s time to move on. Had these issues been addressed when their miners requested then I’m sure it would have been in a much healthier state than it is now. The simple fact is that if a pool can’t support all mining hardware, especially the next generation that is slowly shipping out now (?) then it will eventually fail. I hope it survives, as competition is only good for everyone – a look at other pools confirms this, and there are more pools appearing now than there ever was – so why not p2p?
I have always said that a fully functional p2p pool would blow the other pools away, but it has to work with everything – like every other pool out there does. Spinning as many proxies as you can is all well and good, but the non-compatibility issues will still be there, it just seems a pointless exercise – why not just fix the compatibility issues?  A fresh start is what is required, from the ground up.

What say you? 

I suggest you put together a list of weaknesses you see in p2pool.  Other than the fact is is written in python, I don't see why we should completely throw it out.  If there are fundamental problems with the design how do you intend to overcome them?

Overall, I'm not opposed to a separate p2pool effort, however it would need significant advantages over the existing p2pool to be viable.  Currently p2pool is struggling to produce 5 blocks a week., so I'm not sure there is as much demand as you suggest.

Greetings miners & members of the Bitcoin community,

I have been active in Bitcoin circles for over 2 years now and started mining shortly after that. I am a great fan of p2p, I believe in what it stands for as well as the protocol. I am also a great fan & believer in Bitcoin, and believe that Bitcoin & p2p are the perfect partnership for taking Bitcoin to the next level. The advantages to p2p mining are many, but the most appealing to miners are that it is almost completely DDOS proof – meaning no “down” time for anyone, as any miner or pool op will tell you, this is the scourge of Bitcoin mining.

I find it strange that, although there are many “standard” pools to choose from, there only seems to be one p2p version, leaving p2p fans no alternative but to use that “pool”.  I would like to see this change, so am writing this post in an effort to see what kind of feedback/demand/ideas there are for developing a viable alternative to the single p2p “pool”. As we all know, competition breeds innovation – if there is no competition then innovation can be slow or even cease completely, leaving many miners including myself a little frustrated. If Bitcoin is to continue to grow as rapidly as it is doing, then miners & pools must do the same, it’s integral – one needs the other. This is becoming even more apparent with the arrival of the next generation of mining hardware, if the pool software is not compatible with, or constantly updated to deal with these advancements then miners will choose a pool that is. Simple. Just as important, maybe more so, is the ability of pool ops/devs to listen to what their users/miners requests are and deal with them in a fast, efficient and timely manner. This is the nature of p2p, is why it’s so successful & can only improve everything for everybody. After talking to many miners, I feel that there is not only a demand for another p2p “pool”, but there is need for it, and there is much to be gained in having a viable alternative p2p “pool”, especially in regard to efficiency, compatibility & stability.
So, with regards to this, I am inviting anyone who thinks they can help contribute in any way to setting up the new software. I’m no programmer or coder, so would be especially interested to hear from you geeks out there! I have listed below some of what I believe to be of importance, in no particular order (or maybe there is?) and welcome suggestions/ideas and additions will be added accordingly:

1)   Software should be written in C or C++.  I believe this to be the best, most compatible & most widely used language, but is changeable of course.

2)   Ease of use.  With the growing popularity of Bitcoin and new users it is important that the software will work out of the box – with tinkering available to advanced users.

3)   Cross platform compatibility.  Obvious really.

4)   FULL Stratum support for ALL devices.  This is essential for its future success, plug & play. No patches or half baked workarounds.

5)   Maximum efficiency.  No more valuable lost work due to stales, rejects, DOA etc. I see no reason why p2p should be any different to other pools in this respect. 99.8% or more.

6)   Low resource usage. The software should be able to run on most home PC’s comfortably, thus encouraging more beginners.

OK, that’s a start. This list has the potential to get very large I’m sure, and I welcome any feedback, ideas, suggestions from anyone & everyone. I realise that this is no easy task, it is definitely something I am not capable of doing by myself, but if enough people come forward who have the required ability & are willing to help me perform such a task – then maybe we can make a little progress.   Let’s see what happens…..

Peace.  

Update:  Thanks to the miners who have kindly offered to help with testing with their asics/fpga hardware, this is greatly appreciated.

I have had contact with a few programmers, but am still looking for someone with more bitcoin experience - if you think you are that person - please get in touch.

I've also been asked a few times the "what's in it for you?" question - I'm not doing this for financial gain, I'm trying to get this going purely for the benefit of p2p mining & Bitcoin decentralization. I see p2p mining as THE way forward for Bitcoin, and everyone should be able to participate no matter what hardware they are using. Any & all payments/earnings will be done solely by donation through the p2p mining program at the discretion of the miner, and these will be transparently shared between everyone who has helped in it's development, as is the p2p way. Miners will donate if it's successful or shows promise, they won't if it isn't. This puts the onus on the devs to make it successful. I will just be happy to see it working, that's my aim.

As always, questions, ideas, feedback & suggestions are greatly appreciated....

Peace.