Topic: Fully P2P protocol for mining with tunable variance. (Read 2640 times)

The only disadvantage to the proxy model is that the proxies can decide who you can communicate with through them. Although, this does have advantages too. The proxies could, for example, help keep cheating nodes from being effective.


It's interesting that you can see Ripple from this design The inspiration for this came from first reading a lot about Ripple, thinking about how it could be implemented in a fully P2P way and then seeing someone pondering about doing a 51% attack on p2pool.

The main flaw I see is that it wouldn't decrease variance optimally - you only have your friends mining for you, as opposed to the entire pool. You'd have to be communicating with every other pool member, exchanging PoWs, to reach the same low levels of block-finding variance, which would result in O(n^2) scaling. This may be fixable by making it so some nodes can proxy work for others - combine lots of small exchanges into some large ones, though this would probably require trusting them.

As an aside, while reading this thread I couldn't help but think "This approach is to P2Pool as Ripple is to Bitcoin." Not sure whether that's a good or a bad thing.. Ripple is undoubtedly more scalable than Bitcoin.

I expect only those who want to further reduce their variance would take the risk.

You can also keep a simple balance based book-keeping on each address you've had work trades with. A configurable tolerance value (a percentage would probably do) can then be used to determine when to completely give up on the node. The system should also give priority to cooperating with the nodes that have the best ratio in your favor. This is clearly not 100% efficient but that's not the real question to answer. The question is, is it good enough? If it's good enough, I can leave further improvements to the self-interested miners with the skills to do it. I think a good enough system can be built.


Yes, true. But first I need to design the system to a point where modeling is possible.


This is a good point. It means the protocol will need to have some way to find out if a) someone has blacklisted your node and b) if so, provide information about the shares you missed. I'll have to ponder about this. A blacklist is ineffective though, addresses are plentiful and easy to change, so it'll have to be a whitelist based system.

The point b above is challenging. it needs timestamping, otherwise you can't verify that the shares could ever have been valid blocks. However, this is a timestamping service... I wonder if it'd make sense to make each separate ping-pong of share exchanges a separate sharechain in itself. That way, when your share goes unreciprocated by someone, you can save the share and all (or just a few with the lowest hashes?) of the reciprocating shares you got in response to that share from others and use those to prove the time. This should be enough to make it much more expensive to falsify this evidence than you can benefit from it.


Kind of ironic that I'm now thinking of sharechain based solutions when I advertised this as not having a sharechain.  Still, it's not one central sharechain, so it doesn't create the same limitations and vulnerabilites (mainly the 51% attack). This doesn't need a network-wide consensus, afterall.


SPOF is only a problem when you're limited to one central server. This would most definitely not be. In addition, using such a server rather than joining a p2p network where you'd have to route other people's shares will reduce the bandwidth your node needs to use. Due to this, there will be economic incentives for running a switch server. The most obvious way of accepting payments in this case would be as a coinbase outputs to the switch's wallet address.

Even the p2p network could support asking for extra work to the node's address in exchange for share routing. I expect this would solve the "network collapse" problem you outlined but still give users choice about how much expected value they're willing to sacrifice for lower variance. This would, however, complicate the protocol. I don't currently have an idea of how to implement this for the p2p model though. This is a very general networking problem, though, so I wouldn't be surprised if a good enough solution already exists somewhere.


Yes, there is an obvious cost to reducing variance that needs to be paid as increased bandwidth requirements. This will effectively limit the variance improvement to the percentage you're willing to pay for it. I personally would prefer a variance of payment every two days or so. Too much variance reduction also increases the Bitcoin transaction fee costs when sending the bitcoins. (This is the reason I don't mine on p2pool. The payment size I'd get with my hashrate is too small.)

So, I don't quite agree with you that the network traffic would be a problem. Especially since it's possible to assign a cost to the transport layer.


I guess p2pool is O(1) due to the difficulty adjustment to average the rate to a share per 10 seconds.


I believe this is worth investigating, even if it turns out to be a dead end. I'm studying computer science currently and am planning on doing my Master of Science Thesis on this subject. I'm doing a preliminary design for Bachelor's degree first though. Thank you for taking the time to write down your thoughs. I appreciate it.

I understood that but I don't see how it will work in practice: who will decide to make the first share for free? What happens during network disconnects or miner downtimes ? If you allow them how do you protect yourself from leechers that will simulate these events.


This needs to be modeled and computed. As I said there's not only bootstrap but error recovery to do.


I mentioned 2 above: bootstrap leeches and share ignoring masquerading as error or downtime. If a node has more network errorr or downtime than yours do you blacklist it?  If you do and everyone does nobody can participate. If you don't you accept to work for others for free even if it's only for a small amount. Imagine you protect yourself from say any 5% leech (some node that doesn't reciprocate for 5% of your shares) by dropping any node with such a behaviour, what do you expect will happen if someone decides to bombard each of the participating nodes with a DoS attack (a simple SyN flood for example) in turns? Every node will start to blacklist DoSed nodes and you will end up with a fully disconnected setup where you'll have to rebuild the trusts between every node from scratch.


The problem is that your proof can be missed for perfectly innocuous reasons that happen regularly to all participants: the cooperation will thus eventually stop between them all. If you allow for a margin of error, you give a way to cheat. Any way you look at it, the problem is that by eliminating something like the sharechain you eliminate any way for participants to verify reliably that a proof of work really exists and agree on this. So you break anything that would rely on this missing agreement.


I'm just saying that there are problems popping everywhere in my mind when I look at this. Maybe the easiest to understand is the one I just wrote above: this is a system based on trust but you don't have a way to reliably determine the trust you can have in another node.


I'm not sure what a centralized option would bring vs p2pool for example (it is trivial to setup a separate p2pool pool if you want to mine amongst friends). Any central server would be a SPOF so why not just use p2pool?


If you select based on addresses as you will try to have the most addresses in a single share (if I understand correctly it's a direct consequence of wanting to have the less variance thus the most other participants) a single share will be broadcast by all nodes owning one address in a share to all other nodes this is O(n^2) traffic. You want to fine-tune this by only forwarding shares if you are its miner (or delegate this work to someone else which means accepting to forward shares for others too: same traffic in the end) but even then it's still O(n) traffic. This is not good, "network collapse"-level not good (if you see this project used by more than small loosely-connected teams of 10 nodes).

To make it usable, you want O(1) network traffic if you can and worst case scenario O(log(n)). So here you have another show-stopper until you find a way to reduce this traffic to something not linearly dependent on the number of miners you want to reciprocate with (note: small miners are the most impacted by this as they want more links with others than large miners in order to achieve the same variance).

I know I sound overly negative but all my dev instincts tell me the direction you want to go is a dead end and all common questions you ask yourself when designing P2P networks (error handling, network scaling, robustness to attacks, partitioning events handling, ...) don't have clear answers, only difficult or arguably impossible problems to solve. This isn't even taking into account the acceptance factor : miners cringe when you mention a potential single digit percent loss of revenue and giving work for free to be accepted in such a scheme will simply make them run away.

I think you're missing a key point about how this system would work. There's no contracting, just proof of work done. You receive proof that someone mined for you and you add them to the next coinbase with equal expected value, then you mine until you find a share and send that share over as a proof of the returned work you did. Then you remve them from the coinbase you're mining. If you receive another share from them, you add them again. This forms a chain reaction that continues until one side defects.

The cost of getting this started, even assuming 99% leech ratio is a very minimal investment. If someone doesn't reciprocate, no problem, you just ignore them for the rest of the session. Once you've found enough reciprocating partners, even the minimal losses stop.

Pool hopping is a very specific way to benefit from a pool that uses the proportional payment method. Redefining it in the way you have does not really serve any useful purpose. If you wish to discuss ways miners could realistically cheat on each other in this system, that could be quite productive in getting this system fleshed out better though. I highly doubt there are many ways to do that though.


The crucial difference to promises here is that the main method of operation is not promises. It's proof of work done. No proof, no more cooperation. Nothing this algorithm does is based on trusting promises. It's very cheap to to try if another miner is willing to reciprocate, it can be with a miniscule fraction of the share's value (which is, at this moment around 1636 satoshi per difficulty 1 share). If they don't reciprocate, no problem, just forget about them. After a short startup period, the miner will have a list of other reciprocating miners and can stop looking for new ones.


This is still an unrefined idea that I'm slowly fleshing out. Yes, a "reciprocity daemon" is pretty close to what I'm planning. However, I don't understand your criticism. Are you trying to say that because no-one is using this, no-one will use this? There are lots of different options to use for peer discovery. Are you trying to say that none of them work?

This could be done in a p2p way or in a centralized way. The centralized option could mean that a server, much like a pool server, would function as a switch connecting different miners to each other. The main difference to a pool would be that you wouldn't need to trust the switch, the worst it could do to you is to not relay your data. The other difference to a regular pool would be that you could use several different switch servers at the same time. I'm also thinking it'd make sense for the miners to use this connectivity to exchange new transactions with the nodes they're cooperating with.

The p2p way would resemble the switch server in that every node would do the tasks of a switch but only for a few nodes. a Bitcoin-style broadcast network could work. I've been toying with the idea of having nodes relay shares to others but only when the share contains both it's own address and the address of the one relayed to in the coinbase. There would only be a few connections per node, so useless connections would be replaced until useful ones were found. I expect this would naturally cluster the network so that once you find a reciprocating node, you find tons of them at the same time.


Yes, you're correct. I'm very much thinking about the long term with this system. A generic protocol like this is something that could eventually be implemented with the reference bitcoin.org implementation.

Eliel's goal seems to be a system that automatically negotiates forming ad-hoc pools between peers. This is as opposed to the current system where miners choose to cooperate with an existing pool setup by someone else (eg, p2pool is setup and maintained by forrestv). I agree with you that this is unnecessary considering that there are already multiple options in the decentralized pool category (BitPenny, p2pool, Eligius, EclipseMC, BitArena, etc), but it is still interesting in theory and might very well be the future of pooled mining.

round and promise of a cut is not what makes hopping possible.
Hopping is possible when you can expect more reward from the system than your fair share by changing who you are "contracting" with based on expected income probabilities. But anyway let's not discuss this point as there are more basic flaws here.

When you say there is no "promise of a cut" you are playing with semantics again: a share with a coinbase giving a percentage to one address must be a promise of a cut: it means that its miner was using a coinbase that will give this address a cut if a block is found. The miner who owns the address wouldn't start working on a coinbase giving back the same amount of work if it didn't consider that the other miner will continue working with a cut to it in the coinbase so these shares are indeed a "promise of a cut". If you don't agree with the share being a promise of a cut, the alternative is pretty bleak: there's no way for a miner to check that reciprocity is met and so the system is wide open for cheating.

Sorry but this protocol seems full of big holes and I really don't see the advantage: I'm not even sure what the "needlessly strict pool boundaries" are. Would you advocate a system where there is no pool at all and everyone should look for individual partners with a local "reciprocity daemon" finding/maintaining partners and checking that everything is going on nicely? That would be a mess: with p2pool we have ~200 users (and it's arguably high variance), should I contact them all to setup the thing? What about small miners, who would bother contact them (they don't help your variance much, so why bother)?

There is no promise of a cut. There is only proof of cooperation. Cooperation continues as long as it stays reciprocal. If one side stops cooperating, the other side stops too, on a very short notice. That's all. That's the reason no payment method is needed. You get paid whenever someone you're cooperating with, at the time, finds a block. It's a share for share protocol. Every share can be forgotten about almost instantly.

This is not a hoppable scheme. To be hoppable, there needs to be both the promise of a cut and the idea of a round. This has neither.

Semantics: if you send shares promising a percentage cut from your work with a coinbase allocating n% to an address there is a payment method and it's called proportional. To reduce variance people will want to be connected to as many other miners as possible leading to a protocol to establish new reciprocities. Given that the allocation is proportional, miners will have the opportunity to hop.

This is won't be a pool... I'm pretty sure I already explained this. This is an idea for a protocol that will blur the needlessly strict pool boundaries. Let me give an example that should illustrate this. With this protocol, you can be cooperatively mining with miners A and B who do not cooperate with each other. This will still work.

Yes, p2pool's sharechain does in effect do the thing I'm proposing here. However, it also does things that are not really needed in my opinion. P2Pool uses the sharechain is to synchronize the coinbase transactions for every miner mining on P2Pool. For this, it uses a payment method called PPLNS. The protocol I'm describing here does not need a payment method.

The idea of a centralized valid work is completely done away with. Any share that had a coinbase that would've paid me if it had been a valid block serves to prove that the other miner is cooperating with me. There is no need for such strict protocols as used in P2Pool.

p2pool's sharechain already implement this mechanism. The only difference here is that each participant has a different view of what others have worked on because there isn't a reference of valid work to check (the sharechain). If someone is joining your distributed pool, they know nothing about the others and won't reward them with the expected amount so be rejected. The same problem will happen when a miner is temporarily down.

So you want miners in your distributed pool to have a way of sharing a known set of valid work (the very same for all participants unless you want to allow cheaters on board) to base their distribution on. The sharechain in p2pool does that, what's your solution?

First, this is not PPS. The math used resembles what is used for calculating the PPS reward but it's used for a very different purpose.

You're trying to think about this in terms of how it's done in a pool. This method does not have the concept of a round, therefore it's hopping proof. There is nothing to hop into or out of in the first place. A buffer is also nonsensical since there is no trusted party aside from Bitcoin itself.

When you mine in a regular pool, the pool keeps track of the shares you submit and calculates a payment to you accordingly. With the method I'm suggesting, You get paid whenever someone who is mining for your address, finds a block.

The trick is that the coinbase transacion can pay any number of addresses in a single block. That is how p2pool handles the payments.

I'll try to make this clearer with a simplified example. Let's assume you were mining fast enough to find a block per month on average. You also have 29 friends who are also mining at the same speed. If all of you mine separately, each of you will find 50 BTC per month on average but sometimes several months will go by without you finding any.

Because you'd prefer more steady income, you get together with all 29 of your friends and agree that whenever one of you finds a block, everyone will get 1/30th of the block reward. Assuming everyone does this, each of you will then receive 1.67 BTC on average every day. However, the problem with this setup is that it's very easy to keep mining to yourself and take the 1.67 from everyone else and no-one can prove that you're doing that.

We need a way to make sure that everyone who is participating is actually mining according to the agreement, otherwise it just falls apart. The way this has been solved thus far is by pools. Everyone mines to the pool's wallet(1) and the pool receives shares from them proving they're doing so. It's then the pool's job to calculate how much of the payment belongs to each participant and pay them accordingly.

The way I'm proposing is another way of solving this, without any kind of trusted intermediary(2). In my system, you and your friends would each keep mining separately. However, each time you find a block that meets the share difficulty, you send that plus the coinbase transaction to each of your friends. Your friends will do the same. Based on the shares your friends have sent you, you can check that they're sticking to the agreement and actually mining for you too. The end result is that each time you or your friends finds a block, all of you will be paid the agreed on amount.

However, in practise it's quite rare that everyone mines at the same speed. That's where the Tit for Tat algorithm comes in. The Tit for Tat algorithm means that if one of your friends stops mining or just slows down, your miner will mine for them at the previous allocation until the value of the work you've done for them hits a predefined limit. After that, it will redistribute the block reward allocation to match the new hashing speeds.

I hope this clarified the misundestanding.

(1) Eligius and p2pool are exceptions to this as they actually pay directly from generation to the miners.
(2) Even p2pool has a kind of trusted intermediary. It's called the sharechain.

I don't see this working as you have presented it.

PPS only works on a centralized pool that can handle a buffer.  What happens when there is an unlucky round and all the shares you have received add up to more than 50BTC?  On a lucky round, does the miner just keep all the unclaimed shares?


In order for this to not break on an unlucky round and be hop-proof, that 1 BTC is going to have to be a variable amount.  This is why p2pool has a share chain.

The payout algorithm in this case would be an emergent property arising from the tit for tat principle being used between individual miners. There is tracking of shares, but that's not used for the purpose of calculating the payment amount. Instead, it's used for calculating the expected value you received from the other miner's share. With this algorithm, there's no need to remember received shares, the miner can just toss them after parsing them and noting the expected value received.

A pool that pays with the PPS payment algorithm calculates the value of a share at difficulty of 1 as follows:
<Block Reward> / <difficulty> = <value of one share>
Currently: 50 BTC / 3054627 = 0.00001637 BTC

This is the expected value for a difficulty 1 share. A more generally useful form that calculates the expected value a share has for you looks like this:
<Payment in BTC> / <difficulty> * <share difficulty> = <expected value>

Here Payment in BTC stands for the amount of BTC you'd have been paid had the share been a valid block.

A simplified version of the tit for tat algorithm goes as follows:

• Receive a share from another miner
• Calculate the expected value for you from the share.
• Include the other miner's address in the block you're trying to create with equivalent expected value.
• Once you find a share, send the share to the other miner and remove their address from the block you're creating.

The result of two miners both using this strategy would be something like:
1. Miner A tries to mine 1 BTC for miner B and sends miner B a share proving this.
2. Miner B tries to mine 1 BTC for miner A and sends miner A a share proving this.
3. Jump to 1.

In practise, the algorithm would need to be more complex since modifying the coinbase transaction too often hurts your hashing speed. However, in this simple form, you can easily understand the basic principles behind it.

One way to usefully visualize this is to compare this with a regular pool. In a regular pool, You have the pool in the middle linked to every miner where the pool receives all the shares and calculates the corresponding balances with some method. With this p2p model I described, all miners would be linked to each other and send shares to each other proving to each other (as opposed to just the pool) that they're playing fair.

I guess I should have said, "sounds like a less efficient p2pool"

If each miner is tracking all of the other miners they are working with, aren't they going to be tracking it with something like a share chain?

How are you planning to pay out? Proportional? We've seen how well that works...

Very different. p2pool is based on a sharechain that resembles Bitcoin's blockchain that is then used calculate each miner's payment with the PPLNS payment method. This method I'm describing does not have either of those. This method actually mixes the mining protocol and payment method so that they can't really be separated anymore.

Sounds like p2pool

I got this idea for how to implement a fully p2p mining protocol. It's based on how Bittorrent works. Tit For Tat.

The way mining pools work currently, is that they credit you for each share you mine. The share serves as a proof that you're actually mining what the pool wants you to mine. However, the pool is not really needed. Individual miners can just as simply each mine separately and cooperate by proving to each other that they're including each other in the blocks they're mining.

So, basically, Tit for Tat. When someone sends you a share that includes your mining address in the coinbase transaction, you can return the favor by mining a share that gives equal value in return and sending it back, proving you did so. when two miners both do this, this will cause a chain reaction that will continue until one of them stops mining.

Now, imagine all miners doing this with many others. You could effectively control your variance by varying how many others you do this with. Other ways to the same effect would be adjusting share difficulty and the amounts paid to others in coinbase.

To get things started and find trustworthy mining partners, you can optimistically mine for other miners to find out which of them are willing to return the favor.

Well, that's about it. Let me know what you think.