Topic: Getting rid of pools: Proof of Collaborative Work - page 6. (Read 1930 times)

I'm only trying to understand I'm not trying to judge your work, but I need some clarification.

As I can understand a net merkle is a merkle tree where the coinbase transaction have no block rewards(only comulated fee) so each miner will try to publish his net merkle tree with a coinbase tree pointing to his address.
why are you saying there will be only one or (barely) 2 o 3  initialization blocks? why a miner should use a net merkle tree with a coinbase pointing to an address owned by other miner? where is the incentive to start contribution phase loosing transaction fees? or I'm missing something.

They will share an inizialization block, adding their address, and their, partial, proof of work to earn score.


as soon as a miner get enought shares(with partial proof of work) it can begin trying to solve a more difficult proof of work to get only a partial block reward. But producing a final block is very hard as any contrib share can point to a different net merkle tree(because every miner will add a coinbase transaction using his address to earn fees), so he have to validate the same transactions more and more times to be sure final block wil be considered valid. So I hope it is better for a miner to simply  autoproduce his net merkle tree and his contribution shares so he doesn't have to share block reward with others contributors and to get the full fee booty.

So I can not understand where is the point in using contribution shares from others miners?

@goddog

No it is not a pool of any kind, instead a new type of PoW. I see you are too lazy to read my opening post  

In classical PoW, although miners dedicate resources to generate blocks with average value, in terms of difficulty, they are completely ignored, (this is why people join pools). My Simple question can be formulated this way: Why just don't let this blocks to be accumulated as a proof of collaborative work?

Classical PoW, the way it is implemented in bitcoin, Ethereum, ..., fails to do so because typically the block is hashed as a whole. It is because the block header contains the merkle root that in turn has committed every single information effective on the blockchain state, both conventional transactions and the coinbase.

This prevents miners to collaborate because each miner generates a different coinbase (reflecting his reward address and amount) that produces a different merkle root in turn.

In PoCW, we separate coinbase transaction from the Merkle Tree. This way, miners can share their 'good hashes' for a Merkle root that is getting most popular over time and when enough 'good hashes' are accumulated, they try their chance to generate the final block that now can use the found shares to prove its quality and as an evidence for the way it suggests for distribution of the block rewards, at the same time, by means of the used shares that are unforgeable and can not be separated from the miner's wallet address who generated them.

In Collaboration phase, it is of miners best interest to converge asap on a popular Merkle root (transaction set) and submit their found shares that satisfy the trivially set difficulty.
When producing Finalized Blocks, miners have to include the previously found  share (as they are) and this way they have to admit the founders share, fairly.

In other words, in PoCW miners mainly focus on the actual ordinary transactions made by users and share their efforts to take care of them. Instead of artificially ruining the transaction set by injecting their reward expectation and delaying the consensus mechanism for a fortunate accident of a miner to 'find' a block with high enough difficulty.

PoCW looks to me ways more decent and fair and at the same time very solid, compared to PoW.

I hope this brief explanation could help and encourage you to read the opening post in its entire and more carefully.  

I mentioned p2pool only as a starting point, I know it has a lot of problems and flaws,
From the little I can understand your proposal can be applied to the current pow system as a new kind of decentralized pool software(like an improved f2pool), you can call it C2pool :-D

However I can not, really, understand how your proposal should work. From what I can understand yuor finalization block , is the only thing that matters, so miners will simply fake all others data, or selfish mining all of them at no cost, to maximize their profit.

Also I think it is very susceptible to network latency and/or split, making it vulnerable very vulnerable to a wide range of sybil attack.
But as I was saying for sure I'm missing some foundamental point.
So can you explain me why a miner should be cooperative? I can not see where are the real incentives.

@goddog
P2Pool has been around for a long time but as of this writing it has not done good enough (like 0.00005 of the total network hash power, last block found according to their official stats page dates back to years ago! ) it is because of it being a lot more resource consuming than my proposal, and less efficient, I guess. I'm not happy to say this, but it looks like a failed project to me.

Nodes in the P2Pool protocol should maintain a separate blockchain plus performing a lot of validation checks on the submitted shares and there is no convergence mechanism unlike PoCW proposed here.

In PoCW, miners just converge on a Prepared Block's merkle root and focus on hashing instead of going through verifying and accounting submitted shares. It is just in Finalization Phase that they need to collect a set of valid shares with the sum of their difficulties being sufficient to satisfy the target difficulty of the network.

Another problem with P2Pool is that the level of convenience provided  is not enough (120 times  lower difficulty). PoCW, proposed here, can accept shares up to 100,000 times easier, like what centralized pools are capable of.

I'm sorry, I hope I can not fully understand your proposal, but if you are against centralized pools, why  do not simply try to improve p2pool protocol? Pow have to be simple so it don't need tuning costants finding some magic to have things working. Simple mean less errors and problems.

I hope pools are not a problem if miners can point their hashrate where they want, at any time at no cost.
however p2pool is a nice try to have  decentralized pools. You can try to develop a decentralized pool, if it is better than centralized pools solutions, it will be adopted by miners.

Motivation
For bitcoin and the altcoins which are based on common PoW principles,  centralization of mining through using pools, is both inevitable and unfortunate and puts all the reasonings that support the security of PoW in a paradoxical, fragile situation.

A same problem does exist with PoS networks. Things can get even worse  there because of the fact that most PoS based systems enforce long run deposit strategies for miners that is highly discouraging for them to migrate from one pool to another because of the costs involved.

Satoshi Nakamoto's implementation of PoW that is the core of bitcoin client software is based on a winner-takes-all strategy which is the fundamental factor behind two critical flaws: mining variance and proximity premium which are the most important forces that participate in forming pooling pressure.

Until now, both mining variance and proximity premium are known to be unavoidable and hence pooling pressure is considered an inherent flaw for bitcoin and other PoW based currencies.

In this proposal, we are suggesting an alternative variant of PoW in which the traditional winner-takes-all is replaced with a collaborator-takes-share strategy.

The problem of solo mining becoming too risky and impractical for small mining facilities appeared in 2010, less than 2 years after bitcoin had been launched. It was the worst timing ever, although Satoshi Nakamoto made a comment on bitcointalk about first pool proposals,  it was among the latest posts Satoshi made and he just disappeared few days later from this forum, forever, without making a serious contribution to the subject.

This way, the confused community came out with an unbelievable solution for such a critical problem, a second layer centralized protocol, named pooling, boosted by greed and ignorance, supported by junior hackers who as usual missed the forest.

Bitcoin was just 2 years old when pooling age began and eventually dominated almost all the hashpower of the network.

A quick review of Slush thread in which Satoshi has made the above referenced reply, could reveal how immature and naive this solution was and has been discussed and how it has been adopted: In a rush with an obvious greed.
Nobody ever mentioned the possibility of an algorithm tweak to keep PoW decentralized. Instead everybody was talking about how practical was such a centralized service while the answer was more than obvious:
Yes! you can always do everything with a centralized service, don't bother investigating.  

Anyway, in the thread, one couldn't find any arguments about the centralization consequences or the possibility of alternative approaches including the core algorithm improvements

I think it is not fair. PoW is great and can easily be improved to eliminate such a paradoxically centralized second layer solution. This proposal, Proof of Collaborative Work (PoCW) is an example of inherent possibilities and capacities of PoW. I didn't find any similar proposal and it looks to be original but If there is a history, I'll be glad to be informed about.

The Idea is accepting and propagating works with hundreds of thousands times lower difficulties and accumulating them as a proof of work for a given transaction set, letting miners with a very low shares of hash power ( say of orders like 10^-6) to participate directly in the network and yet experience and monitor their performance on an hourly basis.



Imo, now, after almost a decade being passed, Moore law has done enough to make it feasible utilizing more bandwidth and storage resources and it seems to me kinda hypocritic to make arguments about 'poor miners' and pretending to be concerned about centralization threats and making excuses so for rejecting this very specific proposal that although increases the demand for such resources, can radically disrupt current situation with pools and centralized mining.

This proposal is mainly designed for bitcoin. For the sake of convenience and letting the readers to have a more specific perception of the idea, I have deliberately used constants instead of adjustable parameters.

Outlines

• An immediate but not practically feasible approach can be reducing blocktime (along with proportional reduction in block reward). Although this approach, as mentioned, can not be applied because of network propagation problems involved, but a very excellent consequence would be its immediate impact on the scalability problem if employed, we will use it partially (reducing blocktime to 1 minute compared to current 10 minutes period).
  
• As  mentioned earlier (and with all due respects to Core team), I don't take objections about the storage and network requirements implications and consequences of reducing blocktime as a serious criticism. We should not leave mining in hands of 5 mining pools to support a hypothetical poor miner/full node owner who can not afford installing a 1 terabyte HD in next 2 years!.
• Also note, blocktime reduction is not a necessary part of PoCW, the proposed algorithm, I'm just including it as one of my old ideas (adopted from another forum member who suggested it as an alternative to infamous block size debate and later has been developed a bit more by me) which I think deserves more investigation and discussion.
  
• PoCW uses a series of mining relevant data structures to be preserved on the blockchain or transmitted as network messages
  
  • Net Merkle Tree: It is an ordinary Merkle hash tree of transactions with the exception that its coinbase transaction shows no block reward (newly published coins) instead the miner charges all transaction fees to his account (supports SegWit)
  • Collaboration Share: it is  a completely new data structure composed of following fields:
    
    • 1- The root of a Net Merkle Tree
    • 2- Collaborating miner's wallet address
    • 3- A nonce
    • calculated difficulty using previous block hash padded with all previous fields, it is always assumed to be at least as hard as 0.0001 compared to current block difficulty
  • Coinbase Share: it is new too and is composed of
    
    • 1- A Collaborating miner's wallet address
    • 2- A nonce
    • 3- A computed difficulty score using the hash of
      • previous block's hash padded with
        
      • current block's merkle root, padded with
      • Collaborating miner's address padded with the nonce field
    • 4-  A reward amount field
  • Shared Coinbase Transaction: It is a list of Coinbase Shares  
    • First share's difficulty score field is fixed to be  2%
    • For each share difficulty score is at least as good as 0.0001
    • Sum of reward amount fields is equal to block reward and for each share is calculated proportional to its difficulty score
  • Prepared Block: It is an ordinary bitcoin block with some exceptions
    
    • 1- Its merkle root points to a  Net Merkle Tree
    • 2- It is fixed to yield a hash that is as difficult as target difficulty * 0.05
  • Finalization Block: It is an ordinary bitcoin block with some exceptions
    
    • 1- Its merkle root points to a  Net Merkle Tree
    • 2- It is fixed to yield a hash that is as difficult as target difficulty * 0.02
    • 3- It has a new field which is a pointer to (the hash of) a non empty Shared Coinbase Transaction
    • 4- The Shared CoinBase Transaction's sum of difficulty scores is greater than or equal to 0.95

• Mining process goes through 3 phases for each block:
  • Preparation Phase: It takes just few seconds for the miners to produce one or (barely) 2 or 3 Prepared Blocks typically. Note that the transaction fees are already transferred to miner's wallet through coinbase transaction committed to the Net Merkle Tree's root for each block.
  • Contribution Phase: Miners start picking one valid Prepared Block's Merkle root, according to their speculations (which become more accurate as new shares are submitted to the network) about it to get enough shares eventually, and producing/relaying valid Contribution Shares for it.
    As the sum of the difficulty scores for a given Prepared Block's Merkle root grows we expect an exponential convergence rate for the most popular Merkle root to be included in Contribution Shares.  
  • Finalization Phase: After the total scores approaches the 0.93 limit, rational Miners would begin to produce a Finalized block
• Verification process involves:
  • Checking both the hash of the finalized block and all of its Shared Coinbase Transaction items to satisfy network difficulty target cumulatively
  • Checking reward distribution in the shared coinbase transaction
  • Checking Merkle tree to be Net
• UTXO calculation is extended to include Shared Coinbase Transactions committed to finalized blocks on the blockchain as well
• Attacks/forks brief analysis:
  • Short range attacks/unintentional forks that try to change the Merkle root are as hard as they are in traditional PoW networks
  • Short range attacks/unintentional forks that preserve the Merkle root but try to change the Shared CoinBase Transaction has  zero side effects on the users (not the miners) and as of redistributing the shares in favor of the forking miner, they are poorly incentivized as gains won't go anything further than like %2-%10  redistribution ever.
    
  • Long Range attacks with a total rewrite agenda will fail just like Traditional PoW  
  • Long Range attacks with partial coinbase rewrite are again poorly incentivized and the costs won't be justified

Implementation

This is a radical improvement to classical PoW, I admit, but the costs involved are fair for the huge impacts and benefits. I have reviewed the bitcoin Core's code and found it totally feasible and practical form the sole programming perspective. Wallets could easily be upgraded to support the new algorithm as well,  but a series of more complicated issues, mostly political are extremely discouraging but it is just too soon to give up and go for a fresh start with a new coin, or just manage for an immature fork with little support, imo.

Before any further decisions, it would be of high value to have enough feedback from the community. Meanwhile I'll be busy coding canonical parts as a BIP for bitcoin blockchain, I think it takes like 2-3 weeks or even a bit more because I'm not part of the team and have to absorb a lot before producing anything useful, plus, I'm not full time, yet

I have examined the proposed algorithm's feasibility as much as I could, yet I can imagine there might be some flaws overlooked, and the readers are welcome to improve it. Philosophical comments questioning the whole idea of eliminating pools don't look to be constructive tho. Thank you.


Major Edits and Protocol Improvements:

• June 10, 2018 09:30 pm Inspired by a discussion with @ir.hn
  
  
  • A Prepared Block should be saved in the fullnodes for a long period of time enough to mitigate any cheating attempt to avoid Preparation Phase and using non-prepared, trivially generated Net Merkle Roots.  
    
    • Full nodes MAY respond to a query by peers asking for a block's respected Prepared Block if they have decided to save the required data long enough
      
    • For the latest 1000 blocks preserving such a data is mandatory.
    • For blocks with an accumulated difficulty harder than or equal to the respected network difficulty, it would be unnecessary to fulfil the above requirement.^*
    • Prepared Block and Preparation phase terms replaced the original Initiation Block and Initiation Phase terms respectively to avoid ambiguity
    Notes:
    ^* This is added to let miners with large enough hash powers choose not to participate in collaborative work.

• reserved for future upgrades
• July 3, 2018 02:20 pm inspired by discussions with @anunimint
  
  
  • A special  transaction was added to Shared CoinBase Transaction to guarantee/adjust proper reward for the finder of Prepared Block and some enhancements were made to include both block reward and transaction fees (partially) in the final calculations.
    
    • In Finalization Phase, miners should calculate a total reward for the miner of its respected Prepared Block as follows:  
      preparedBlockReward = blockReward *0.04 + totalTransactionFees *0.3
    • Shared Coin Base Transaction should include one input/output address-amount pair that adjusts the amount of reward assigned to the Prepared Block Miner in the ordinary coinbase transaction
    • All the calculations needed for calculating rewards assigned to the miners of finalized block and shares will be carried out on the sum of network block reward and 70% of transaction fees collected in transactions that are committed to Net Merkle Tree
    Note:
     This change is made to both guarantee a minimum reward for miners of Prepared Blocks and incentivize them for including more transactions with higher fees  

• reserved for future upgrades