Topic: Getting rid of pools: Proof of Collaborative Work - page 5. (Read 1930 times)

@anunymint
You are so sharp, I mean too sharp.  

Really? Just like 2 minutes after I referred you here, you digested the whole idea and closed the case?

First of all, switching between pools does not change anything and I prefer not to waste the readers' time to argue about it. Just reconsider your claim and please, take your time.

Secondly, in this proposal, miners do have to validate transactions they are committing to and making any objection about it, is just saying that they have to leave validation to a limited number of trusted nodes, validators, authorities, whatever,  like what is the case with pool mining.  

You are full of surprise and this is another weird thing you are suggesting,  and again I can't imagine even arguing about such claims.  

I'm familiar with this literature tho, this is the outcome of any crisis: journalistic revisionism full of  deconstructional claims, ... just because something is going wrong.

I like it, seriously, it is inspiring but ultimately we have to adopt and improve instead of giving up and trying to invent a strange thing that is at the same time objective, scalable, decentralized, sharded, ... and the miners do not need to validate the transactions they are committing to, while they are consuming their resources in the mining process!

Anyway, I referred you here to reject one specific claim you have made about PoW that you are claiming it is doomed to mining variance and inevitability of pool mining as a result and you conclude that it is the most important factor that makes PoW networks infeasible for sharding schemas.

As I see you are not going to share and tell us how it would help, PoCW I mean, to implement sharding, I'll just try "reverse engineering" here:  

When you are saying that bitcoin won't let sharding because it is doomed to centralized pool mining, logically I come to this conclusion:
this variant of PoW (current proposal, Proof of Contributive Work) resolves this issue, and makes it feasible to implement sharding on a PoCW network. Right?

I afraid I haven't explained my ideas regarding this issue in previous post thoroughly.  Actually it is not possible to have Prepared Block's hash in the respected Net Merkle Tree (it leads to recursive definition with no practical exit condition).

Anyway, I decided to include it in the proposal formally and I hope it could help.

Please refer to the starting post a few minutes later.

So you are saying that the preparation block's hash is included in the net merkle tree.  Sounds good.

I'm not really sure about the twin idea, one block is a part of the blockchain and the other block is floating around somewhere?  If they are exactly the same then wouldn't they just be the same block?  I'm not sure how you can have two exactly identical blocks, unless one is a slight bit different.

Yes we are getting synchronized I suppose.  

As of the risks you mentioned in forgetting the Prepared Block:

Firstly, it is to be kept in mind that this block's Net Merkle Root points to a coinbase transaction which charges the fees to the miner.'s wallet address. He should wait (praying for it to get enough contribution) until it is included as the Net Merkle Root of the finalized block.

Secondly, I have made a bit more assessments about the selfish mining attack which I described in my last post and I have come to a very interesting result: If we add a simple restriction for full nodes to reject just the last block in the chain where there is not a valid Prepared Block with 5% difficulty around.
This simple restriction will lead to a very excellent defence to this attack without enforcing much overhead. As a result we will have a new definition for the latest block in the chain: it is the one that has a twin (with same Net Merkle Root, 5% difficulty and same parent) present in the network. When a new round of mining is going to take place, miners would just point to the latest block that has such a twin.

It looks to me a waste of space  to keep this twins alive for more than a few rounds because  I don't see any reason to use Prepared Blocks for bootstrapping and/or partial resynchronization yet.

Hmm yes we should keep in mind finding something besides the internet would be better.  Ham radio is an option if the data size is very small.

I guess to answer my own question; yes a flood attack is possible but it is ok.  Difficulty would be adjusted so, for example, the block will typically fill up full of contribution shares every 10 minutes.  Instead of taking on average 10,000 x blocktime for a small miner to win his first block, he will win a little bit every block on average.  The big miners will fill up the block with small difficulty nonces, but the small miner has 10,000x better chance of getting one of his small difficulty nonces into the block.  Nice.

I think I am with you on the preparation block (same thing as "initiation block" I presume).  The person who can win that somewhat higher difficulty nonce will gain the transaction fees which will immediately show up in their wallet.  So this block never needs to become a part of any hash to prove that it happened?  It seems a little risky in my mind that it just disappears after it is won.  But I can see that it is just needed to "prime the pump" so the collaborators have something to check their ledger with.  Seems a tad bit risky though because it seems the prep block is just a suggestion.  I think from you explaining that "preparation block skipping" attack that the network may loose preparation blocks entirely, since it appears that the prep block hash doesn't need to be a part of the collaboration share.

Two points I have to clear here:

• First: A miner who finds a Collaboration Share with a nonce better than 0.0001 will be rewarded with proportional amount of the block reward eventually, given he has chosen the right Net Merkle Tree to mine for, the one that network has converged around it
• Second: A miner can mine any number of shares and publish them

Plus, it is important to note that Collaboration Shares are very small payloads (like 50 bytes) and there is no reason to set the quality limit that high.

In current Bitcoin blockchain once a block is found by a miner (being a solo miner or a pool operator) it takes a lot of communication overhead to be propagated, because the peer that is receiving the block should become aware of the exact Merkle tree path behind the Merkle root present in the header, at least the transaction ids should be transmitted and in some cases the actual transaction body needs transmission.

Also, I'm generally against hesitations to utilize networking bandwidths. I think it is very weird design strategy for a decentralized consensus based system running on an infrastructure like Internet.

Prepared block won't go to the Blockchain ever and the 5% difficulty level set here won't be checked later. It is just a convention for lubricating the convergence mechanism . Let's dive a bit deeper into it:

Once a Prepared Block is mined and propagated, receiving miners will check its integrity and difficulty before starting to contribute, so the sole fact that some miners contribute to it and produce shares for its Net Merkle Root it is enough proof that the block has convinced them already.

One possible attack scenario by a selfish owner of a very large mining facility could be something like this:

A selfish miner may choose to bypass Preparation phase by simply producing a Net Merkle Tree and starting to produce Collaboration shares and propagating them in the network.
Receiving miners may be fooled that they have missed something and there IS actually some Prepared Block out there and decide to join the race before it is too late.
This way the selfish miner has hijacked transaction fees without consuming enough resources and more importantly he has accelerated the mining process artificially which has problematic consequences for difficulty adjustment algorithm of the network.

Mitigation:
Decent, loyal miners won't contribute to a work that can not be validated by querying the Prepared Block, so such an attack won't work unless a compromise has been made between miners that are not expected to be a 50%+1 majority. And for this minority there will always be a problem to distribute the illegitimate gains properly, thus they have to set a lower limit, say 1%, and yet they are risking to lose the main war, ever and ever because in most cases they are converging to a work that the majority just don't commit to it.

When it comes to bootstrap or partial synchronization for example, full nodes won't ask like 'From where you got this Net Merkle Tree?' because it just doesn't matter anymore.

What I was getting at actually was this: what if there is a really good miner.  You don't want this miner giving contribution shares every time he gets a .0001 nonce, because then he would fill up the block will all his contribution shares.  You would rather have him wait until he gets a .01 nonce for example that way he can get a bigger part of the pie yet not crowd the block with tons of small value nonces. If there was a set time limit he could say "ok I can wait until I have at least a nonce of .01 difficulty before sharing it, because I know in that given time frame I should be able to find one that high". The way this can be done is to reward higher difficulty blocks a little bit disproportionately higher.  But the problem remains there still there would be the incentive to post his lower difficulty nonces too, so I'm not sure what the solution would be to get him to not share his low difficulty nonces.

It may not be vital but it would be good if a flood attack like this could be avoided.



Wow very cool.  So what you are saying that the "Prepared Block" is always mined first and has it's own proof of work requirement separate from anything else?  If so we could think of this Prepared Block as the "real" block of the blockchain.  So the winner of the Prepared Block gets the transaction fees, and the Contribution block mined later is for the block reward coins?  So the only problem is a little bulkier and slower blockchain, instead of 1 block mined per transaction group, there are two.  Not a bad tradeoff though when you consider not only the risk of pools on the network but the risk of pools in terms of loosing your money from being hacked, and making accounts and passwords for different pools and whatever.

Exactly.
Yes. It is set to be at least as 0.0001 good as what the calculated network difficulty enforces and in a worst case scenario, there would be like 9,500 Coinbase Shares  packed in Shared Coinbase Transaction, in my terminology. Depending on the implementation details it implies up to 64 KB size for the transaction.

In practice you can expect much lower numbers down to just 2 transactions. So, regarding the normal distribution of probabilities, I guess 4,250 is a good assumption for the average. Hence 32 KB or so for Shared Coinbase Transaction.


Once the very first Prepared Block is mined and propagated in the network, for average rational miners there is no more reasons to keep trying to mine another competitive instance of such a block, as long as they are sure that the mined block under consideration is valid and won't be rejected by peers, instead they would rationally accept the fact that they have lost the chance of gaining transaction fees and switch to Contribution Phase, producing shares for the Net Merkle Root (the transaction ledger) of the same block. It is the most innovative part of my proposal, I guess.

Obviously the above mentioned process converges exponentially because with every new Collaborative Share issued and propagated, the most popular Net Merkle Root gets even more support from the network. I think before we get to even close to 20% of the needed shares , practically the whole network would be mining the same work.

Ok thanks for your explanation it is helping me go back to your original post and make more sense out of it.

So the net merkle tree is a merkle tree without the block reward.  So if you win a nonce for that (collaboration share) you don't automatically get the block reward alla bitcoin.

I was wrong that it is a participation trophy.  It is a weighted block reward for each participant (collaborator) that varies based on how "good" their nonce was.  Is there a set time limit for collaboration shares, so a miner can basically "wait untill I get a nonce -this- good" before sending it in?  Or should they just find a bunch of low difficulty nonces?  Is there a limit as to how many collaboration shares can be included into the shared coinbase transaction?

How does the "convergence on an accepted transaction ledger" happen exactly and automatically? 

@buwaytress

Thanks for the support and sharing the referenced BIP, I just posted a reply in your topic.

As you  may have noticed and I have tried to prove there, that is somehow too conservative and can't change things more than like what you say 'a bit'.

Always reading with interest and happy to see that people are always trying to work on this problem of pool centralisation, although I'm not sure any of the direct threats (51% attacks for example) would really happen since pools themselves, no matter how large, have historically understood the fine balance they need to maintain to avoid miners just disconnecting from their pools to join others.

Nevertheless, it is at these finely-maintained balances that centralised pool decisions are a concern. I just wanted to share something else I read (a BIP draft) yesterday that looked instead to just change the protocol to give a bit less power to pools when it comes to creating block templates.

Your suggestion of a way to somehow make individual miners count for the work they contribute, as well, is something a lot of people would support. This makes me wonder if anyone has ever calculated the cost of wasted work (work that never counted), I'm sure it's a LOT.

Thanks for the explanation and yes, people who are used to pool mining, would experience the network like a huge pool. Good analogy.  

By the way, I prefer keeping the title as is, Proof of Collaborative Work.

PoCW is just an  improvement to PoW that inherits its core objective ideas and could be categorized as an implementation variant rather than a new algorithm.

No matter whose ledger it is, rational miners should contribute to the first ledger (at the start of contribution phase) and converge to the most popular one (as this phase develops). Obviously the ledger under consideration should always be examined for its validity.

@GLP

Thanks for your encouraging reply. Glad to see you are supporting the core idea and I'm looking forward for your future contributions to this project.

As I have mentioned earlier, we are in an intense situation and we need to act like a samurai, both quick and perfect. Unless,  we do need spiritual, social, economical and technical contribution unlike a samurai.

I strongly support your suggested roadmap and will do my best to make it happen. Thank you.

I thinked a lot about this proposition, and finally i think that:
1- It is a Decent one and i have not find any relevant technical objection against.
2- It's problem will be both social & political

So, i suggest you to make a proof of concept of it by implementing it on some testnet, ppl who embrace it can help by putting some bounty to hack it. benefits will be:
1- objectivity on the proof of work.
2- If the political resistence is umbeatable and the social insensitive is not motivated, you will have be allready done a half-step to it's productivity implementation on a new blockchain.

As you may agree, a political resistence is not something that can be forced by any rational/objective argumentation alone.

Thank you for sharing and i really valuate this work, please do not give up by a "BTC or nothing" reasoning, it deserve better horizons.

Wish you the best, i definitely will think more about it and will be glad to share any future thoughts.

So what your saying is... basically Proof of Participation (you can steal PoP I won't be offended  ).  Everyone gets a participation trophy which is encoded into the "real" block before the block reward is distributed?  So basically there is in reality one big mining pool run by the software?  Sounds really good in theory.

Who's ledger would be the accepted ledger for the transactions in that case?  I think that is why bitcoin has one winner, so that one person's ledger becomes the accepted ledger.

Like you I figured the only way to beat pools other than coins like SpreadCoin (not sure if it is successful at it) was lowering the blocktime (which is a double edged sword in many ways) but also another idea I had was increase coinbase maturity to something like 9 months.  I figured that would throw a wrench into pools what do you think?

@ir.hn

I do admit that it is not an easy piece of cake when you get into implementation details , or even the design outlines, directly. But things are changing so fast in blockchain and developments are trending in so many false directions ... We have to be simultaneously perfect and quick. Hell of a job ...

Realising this situation, I deliberately chose to present this idea, from a technical point of view, to accelerate technical contributions. Yet, I believe you would be able find some useful general ideas following this topic.

Here I try to give a more general  and conceptual vision, I wish it would help:

In centralized pool mining that has been the dominant practice for years,  miners have no control on the blocks, they receive it from the server, ready to 'mine' which now is about finding a nonce that causes the hash of the block to be 'good enough' to be passed back as a share to the server.
The threshold of what is considered to be good enough is contracted between miner and the pool server and has almost nothing to do with the blockchain difficulty, it is just considered to be thousands to millions of times lower.

Pool server in turn, besides validating and keeping track of submitted shares, checks whether luckily, one of them is so much good (thousands or millions times better than what the trivial contracted difficulty has forced miner to find a nonce for) to be relayed as a freshly found block and added to the blockchain.
If so, the pool will get both block reward and transaction fees immediately. Shares of the miners who participated in this process, all of them and not only the one who has found the golden nonce, will be payed using one of the several common methods for pool accounting, later.

Miners desperately need such a mechanism to avoid waiting for years to find one block (if any) and pools with enough number of submitted shares are expected to be lucky very frequently.

This is a bad situation for security of the network because number of parties who actually decide about the transaction set to be included in each block and (probably) added to the blockchain is dangerously small. This is a centralization threat and any true bitcoiner should be against it, unquestionably.

Actually, as of this writing, 5 major bitcoin mining pools could easily form a cartel that would have >50% majority needed to perform occasional and intentional double spend attacks against people, for the least.

As I have briefly described in my article, I believe PoW can be improved to eliminate the miner's above mentioned need to pool mining. In this proposal this is done by reducing the difficulty hundreds of thousands times.

This difficulty reduction is achieved by tweaking classical PoW to let miners converge on a common set of transactions and submit their found results to the network primarily and wait until their submitted shares have been summarized in a finalized block to be permanently added to the network. In

In classical PoW it is done upside down, a finalized block is prepared from the first beginning, when the 'nonce hunt' is going to start,  no cooperation, no contribution, this is a gap that maliciously is filled by pools and I believe can be totally eliminated by PoCW, this proposal.

This is a very complex idea and you have defined your method but not your reasons for determining this method.  You may want to start from scratch to explain this and give the idea, most basic theoretical implementation and then what problems occur that lead you to add the new factors that you did in the design.

A lot of thought went into this and it is hard for us to get up to speed with your thought process.

What you are doing is giving someone a recipie and saying it makes a better cake but not enough explanation for why eggs whipped at 675 rpm are needed exactly. 

I'm sure it is genius and mabye a couple pepple could follow it but I'm not amoung them.  Also realize that not many of us probably even understand the underpinnings of how pools work exactly.

5% difficulty requirement for Prepared Block causes one block to be generated in every 3 seconds, during the preparation phase, network latency and the required time for miners to decide ending preparation attempts and starting contribution phase may take up to 10 seconds, I guess.

When the first few Prepared Blocks have been already propagated in the network, other miners have tried their chance to win the fees and failed to do so. It is not a big deal tho, in the worst case it is about wasting like 5% of the resources and the transaction fees at the same time. Just like when a new block arrives and you have missed your chance to hit. Typically total transaction fees for a fully saturated block is about 0.5 btc i.e. 0.04 with respect to the current 12.5 btc block reward.
From this point on(having few published Prepared Blocks in hand) , the race is about winning the block reward (main meal) and it is of all miners best interest to forget about the previous race (winning tr fees) and focus on the live one(winning block reward).
They will share an inizialization block, adding their address, and their, partial, proof of work to earn score.


as soon as a miner get enought shares(with partial proof of work) it can begin trying to solve a more difficult proof of work to get only a partial block reward. But producing a final block is very hard as any contrib share can point to a different net merkle tree(because every miner will add a coinbase transaction using his address to earn fees), so he have to validate the same transactions more and more times to be sure final block wil be considered valid. So I hope it is better for a miner to simply  autoproduce his net merkle tree and his contribution shares so he doesn't have to share block reward with others contributors and to get the full fee booty.

So I can not understand where is the point in using contribution shares from others miners?