Topic: Could/should we change the protocol to restrict the market share of mining pool? (Read 2359 times)

Hi, kuzetsa. Thanks a lot for your detailed post. I wish I could provide more comments after I get time to read more about GBT and P2PPool.

Ah, that'd be cool, but I feel as though it's little more than baseless optimism, and seems like you might be "repeating the unsupported claims" propaganda-style about GBT's claimed features and utility, despite the fact that certain aspects of the claims are not even explicitly part of GBT's BIP22 / BIP23 specifications or design.

First, I will start by saying that I am NOT opposed to any sort of GBT implementation(s) which can achieve as many of these goals as possible.

That said, I think there are some systemic flaws in the "GBT server" mining design...

Here's my understanding of some ways in which GBT is basically toothless and impotent, and thus unable to fix [some of] the problems with centralized mining pools, and potential abuses of power by pool operators...

For example, the recent incident where a popular pool operator "wrote, and then released a bitcoin patch which modified their pool's transaction handling code to ignore bitcoin's inherent fungibility, discriminating and severely rate limiting certain types transactions" ... which ultimately inspired the original poster to create this thread:

(and luke, please do feel at liberty to correct / refute these points if any of it is not fact-based)

---

... Evaluated from a "stand-alone vacuum" (see below, *mempool*), GBT by itself doesn't do anything to prevent a pool operator from:

1) intentionally censoring / filtering transactions (as per luke's patch which heavily rate-limits address reuse to once per block)

2) intentionally including transactions associated with executing a double-spend (because either the pool operator themself is the one doing the double-spend, or they were bribed, compelled, or blackmailed into using the pool resources to help dishonest parties pull off a double-spend) *cough* GHash.io *cough*

3) intentionally adding non-bitcoin data for an alternate crypto currency's blockchain (sha-256 based, like namecoin for instance) which is piggybacked on the bitcoin blockchain merged-mining style

... These things can be done by ANY mining pool operator (including a pool which supports GBT) and are not likely to be detected until the block(s) have already been mined using the centralized pool's hashing power.

---

As for point #3, the only way GBT can help, is for a new feature (the details for any such mechanism are not explicitly standardized, or even described in BIP22 / BIP23, but rather, the implemenation is simply "left to the imagination" of anyone choosing to implement "GBT client" based mining software) in BIP22 / BIP23 based mining software (or better still, using a protocol, implemenation, or mining system which adds transparency for miners, but does so in a completely zero-trust manner, and thus not subject to the whim of a centralized "GBT server" pool operator) to have some implementation for fully-configurable, fully-automatic screening rules to inspect the contents of the template sent by the pool...

... ideally, these rules should be automatically used by the mining software, and customizable with full opt-in miner ("GBT client" / end user) configured filtering / alerting / pool-banning (stop mining on this pool, it's doing something bad so switch to a failover pool) levels which are automatically triggered upon detection of anything [in the workload, GBT-based or otherwise] which the miner doesn't wish to blindly participate in...

(...point #3 continued, and a bit of #2 as well) Regardless of any transparency added as a result of having the complete [GBT] template available for inspection, upon "reading the fine print" about BIP22's core GBT protocol functionality, there is an explicit rule wherein the "GBT server" operator is able to flag a transaction as being a "mandatory transaction" (a per-transaction boolean true/false, GBT's JSON-encoded transaction object, field name is "required")

-- as such, because of this "read the fine print" protocol rule, the pool will not credit a miner for proof of work shares which do not include any transaction marked as mandatory. This "mandatory transaction" (fine print) protocol rule gives full power to the "GBT server" (pool) to force a miner to mine whichever transactions the pool operator requires to be included in blocks, and so the full power (and full potential for abusing pool resources) still remains in the hands of a "GBT server" operator...

...even if the work submitted by a miner is "technically valid" as per the rules of the bitcoin network itself, the "GBT server" / pool operator's BIP22-compliant server will reject any shares without the required (fine print) transaction(s), automatically punishing users for not "doing as they were told" (miners who want to opt-out of merged mining, miner-detected double spends, or even just miners with their own preference for arbitrary or specific types of transactions being opted out of, etc. etc. etc.)

---

As for point #1 and #2, the "GBT client" (miner) side will need access to a bitcoind-type mempool in order to be able to compare the pool-issued [GBT] template against the miner's own preferences...

There is no way to build a zero-trust solution to the issues in point #1 and #2 without the "GBT client" having access to a local bitcoind node (not necessarily a full bitcoind / bitcoin-qt node, but at least SOME version of a software which speaks the full bitcoin protocol and has a mempool full of transactions, and therefore able to audit anything undesirable described in point #1 or #2)

If GBT's form of giving miners control over their own "decentralized" mining forces them to run their own local bitcoin node anyway (just to be able have a local mempool on-hand... see below, *mempool*) then they might just as easily give up on fake-decentralzied, and take the extra time to set up a p2pool node and use it for actually-decentralized mining instead.

---

*mempool*

WITHOUT at-minimum, providing the "GBT client" / miner access to some form of zero-trust mempool or otherwise provably accurate list of bitcoin's p2p network dataset for unconformed transactions...

... the upper-most limit / extent to which any "added security" can ever be provided by BIP22 / BIP23 GBT is to add "transparency and accountability" by hypothetically being able to inspect the contents of whatever work has been sent to the miner...

... to my knowledge, at this time there is still a total lack of any public, working, proof-of-concept "GBT client" implementation with the ability to use zero-trust mempool data (local bitcoin node or otherwise) for the purposes of auditing and amending the suggested template sent by the pool ("GBT server") in order to make it easier for the miner ("GBT client") to mine bitcoin in a more decentralized manner.

---

GBT isn't automatically decentralized just because the BIP22 / BIP23 specification for GBT doesn't explicitly forbid compliant implementations from taking //unspecified, but not-forbidden liberties// meant to mitigate the risks of using a centralized "GBT server" which assigns work to slave ("GBT client") miner nodes.

Like I said near the start of this write-up:

GBT's "decentralized" features look pretty toothless and impotent, and further, it really seems as though most of the decentralization in GBT is vaporware and boastful posturing / propaganda...

If I was a pool operator, and if I really wanted to harm bitcoin in any of the above outlined ways, GBT doesn't have in-built mechanisms to empower miners to help safeguard the bitcoin network, to be able to mine in a way which best suits their own personal conscience or aspirations, to exercise their liberties of free speech in the form of their "vote by hashpower" voices as miners, or to combat any potential pool-operator abuses on an "actually, literally, by-definition CENTRALIZED" bitcoin mining pool such as the flagship pool supporting GBT mining protocol, eligius.

... the power (and potential for abuse) held by centralized pool operators is no less dangerous just because "GBT exists, and a handful of miners are willing to waste extra bandwidth and CPU cycles just to use it"

---

Here are ways in which p2pool is decentralized, but GBT is not (despite any of luke's propaganda you might've been suckered into believing)

p2pool by its very design, already requires each p2pool node to have zero-trust access to the bitcoin network (as described above in *mempool*)

P2pool does not use a centralized server to issue work. That's already more decentralized than slave-mode "GBT client" mining...

The p2pool decentralized "pool" (a p2p network in its own right, same as bitcoin) only requires participating nodes to follow a single rule in order to get paid for their "proof of work" shares by p2pool:

If the submitted proof of work credits the pool, this valid proof of work is counted as a share, and so p2pool will FIFO-style (first-in / first-out) add that share to the p2pool sharechain, a decentralized list of addresses which need to be paid (PPLNS style) by any p2pool node which finds a block (and a share continues to be paid by the p2pool network until that share drops out of the fully distributed / replicated / decentralized p2pool sharechain due to the way in which outdated PPLNS sharechain entries age gracefully, and ultimately expire)

... if the "work" done by the node meets the bitcoin network "difficulty" target, the newly found block is then directly broadcast to the bitcoin network, and quickly becomes part of the bitcoin network's blockchain since p2pool nodes themselves are already connected to the bitcoin network (p2pool even uses the GBT JSON-RPC function: "submitblock" on the bitcoin node directly connected to that p2pool node... however, for slave-mode "GBT client" miners, there is ZERO requirement in BIP22 / BIP23 for a slave miner to have access to a decentralized bitcoin node)

---

As per the original topic described in the original post of this thread:

Yes, I feel as though limiting the power of mining pool operators should happen (even if not by way of reducing the percentage of hashpower held by any one pool)

Additionally, I STRONGLY support the idea of adding additional functionality to GBT (or any other protocol, not limited to GBT, up to and including an overhaul of the way bitcoin adds new blocks to the blockchain) to reduce the influence of any one (potentially abusive) bitcoin pool operator or node, etc. etc. etc.

My understanding is that once GBT protocol is finalized, everyone can write his own miner software follows this protocol. So there can be multiple version of Eligius miner programs, and Eligius cannot reject certain clients as long as they follow the protocol. Therefore, it could be much better than current situation, where miners either join the pool and follow operator's rules or leave.

That wasn't trolling.

I was seriously trying to ask real questions about your purpose on this thread.

It's not even a new question that you've never been asked about some of this, and then simply refused to give straight answer. It's been more than a year since these GBT "fake decentralization" issues were first discussed, yet you still haven't answered the question posted:

---

Basically the same as my bulleted point #3 (in part 3 of 3, from a couple posts back)

You have chosen not to refute the statement that:

By simply using GBT protocol to mine, a miner is not magically able to re-add the transactions which were filtered out by the eligius (or other) pool operator. There is still not even a working "proof of concept" implementation for a GBT client which is able to do this.

You really spent 3 posts on trolling and FUD? Amazing.

-- Part 3 of 3 --

Please explain:

Since BIP 22 doesn't even describe anything other than a centralized server feeding work to slave worker nodes (miners) ... where is the documented language about any form of decentralization in GBT (BIP 22)?

• 3) Additionally, this thread was explicitly created because pool operators could arguably be seen as having have too much power... For example:
  

What are the basic steps for mining software configuration procedures for a miner (GBT client) who chooses to mine on eligus (which supports GBT mining, so theoretically according to BIP 22 this should be possible) to be able to re-add the transactions which were filtered (because of address reuse) by the pool, so that this miner (GBT client) is not subject to the whim of the GBT server operator (pool operator of eligius, namely you, and your filtering overreach and abuse of the powers afforded to you, as being discussed in this thread)

---

Cool story, bro.

Tell me more about how your solution to not being able to comfortably run a full bitcoin node on beaglebone embedded hardware is... wait, don't those KnC miners come with a well-tuned, embedded version of cgminer out of the box?

What was your point anyway?

What is the point of your stupid anti-p2pool commentary / what was that post even about?

I was making an observation about the implementation of GBT in p2pool; that is to say, it seems to me that p2pool's GBT implementation actually works quite well, despite being a python script rather than optimized / compiled C code.

I know for a fact that a single p2pool node with 30 connections to the p2pool p2p network running on "not awesome" hardware (1.3 GHZ "green" cpu rated at 17 watt TDP) only uses about 14% CPU utilization for 6.7 GH/s, and most of that is just the p2p overhead you seemed to be bothered by ... or something (I'm not sure what the point of that comment was either)

Increasing the hashrate would be VERY scalable, and should hardly increase the CPU load on the p2pool node, if it even increases by a measurable amount because the p2pool node is talking to cgminer via stratum protocol, and cgminer itself is using almost no CPU (it almost doesn't even register) to say nothing of the stratum itself, which is not an inherently high-overhead protocol.

As a matter of fact, I know that knc saturn (WITHOUT that bertmod BFGMiner mod installed) has no problem at all with using stratum protocol, so it could easily talk to the same low-power server which I was using to feed work to a BFL ASIC Jalapeno (the 6.7GH/s miner in question)

I know some p2pool node operators who have comfortably handled a workload of more than 1 TH/s without the node becoming CPU bound and shitting itself, getting a huge hashrate reduction, etc.

GBT is not a bad protocol, but BFGMiner happens to be "not very good" for GBT, despite the fact that the GBT implementation in BFGMiner was written by the author of BIP 22

---

For a minute, I was half tempted to say "challenge accepted" for testing the p2pool script on a 1ghz beaglebone black, but I don't have one on-hand for testing... it's really not all that much slower than the 1.3ghz "green / power efficient" cpu I have on one of my p2pool nodes (it just happens to also host its own bitcoin node since it has enough ram and disk for that... perhaps connect an external usb hard disk to the BBB? I'll leave that for someone else to figure out)

-- Part 2 of 3 --


What's all this rubbish about GBT being decentralized AT ALL?

What are you even claiming?

• 2) BIP 22 client / server protocol claims that the blocks are (quote) "left to the miner to (optionally) customize and assemble" and as such, implies that a client (miner) has the ability to customize the selection to their own preferences for which transactions to include into blocks...
  

This claim in BIP 22 wasn't based in reality.

To clarify, I mean to say that the claim was not based on any "actually been implemented" reality (hypothetical doesn't count) at the time BIP 22 was drafted (February 2012) and even a year later (February 2013) there was still no implementation allowing a miner to view, audit, or modify the list of transactions proposed by the server.

The transactions to be processed by the client (miner) might be hypothetically independent from GBT server (pool) input...

but...

... the hypothetical process for miner (GBT client) modification of the transactions sent by the pool (GBT server) isn't even explained in BIP 22

Isn't it true that BIP 22 (GBT specification) never clearly described any straightforward algorithm, technique, process, API call, protocol, or otherwise "usable mechanism" where miners have a non-hypothetical ability to view, audit, or modify the list of transactions proposed by the server?

And furthermore, isn't it also true that to this day there is still no GBT implementation which has the required extensions / features to be able to do such things?

srsly, luke...

Why you gotta go bringing up GBT again?

The stuff you tried to claim just now is kinda ... well....

A fact-based "not talking out my ass" rebuttal follows:


-- Part 1 of 3 --



What the fuck are you even talking about?

Are you saying that you feel like the decentralization (the p2p part) of p2pool is somehow inconvenient?

• 1) The pool (described in BIP22 as "server") which gives out work via GBT is a single point of failure:
  

Where / which exact section or wording in BIP 22 is there any language OTHER than "server" which clearly describes how the GBT JSON-RPC call is to be used for "not-using-a-centralized-server, and therefore, by definition literaly decentralized" mining?

(as an aside, modern p2pool uses GBT under the hood when communicating with bitcoind, despite the fact that p2pool was around in 2011 nearly a full year before GBT BIP 22 was even drafted)

Isn't it true that for a "client" (miner which mines via GBT protocol) the "server" (mining pool, so let's just say "eligius" as an example) being used by that miner is a centralized, single point of failure?

If a single p2pool node goes down, it's not centralized, so it doesn't act as a single point of failure and prevent every other p2pool node from being usable for mining.

Thanks for your explanation. Yes, who owns the block is decided by the coinbase tx, not who broadcasting it.  So in GBT, the pool just checks whether the coinbase tx is correct and don't mind what other transactions are included. Each miner can choose other txs by themselves right?

Your reply is great and will be greater if there's no sentence like that.

I am not the person who believe they need to change Bitcoin in order to save it.  I am a person who believe Bitcoin needs to be changed by someone who knows more than me (note 'could' in my title). Besides, I admit it may be a silly question in advance.

But thanks a lot anyway for your explanation.

GBT is just as decentralised as p2pool.
And unlike p2pool, GBT doesn't require the overhead of p2p.

Good luck getting p2pool to work in KnC BBB hosts...

Right, you can't edit the work after you already completed it, that would change the hash.

The current version of p2pool checks to make sure any submitted work credits p2pool, and only if the submitted work pays p2pool (in the coinbase transaction) does the work / share submitted get added to the p2pool sharechain (which ultimately contains the list of who needs to get paid whenever someone manages to find a proper bitcoin block)

The p2pool node to find the block then transmits the block, because it doesn't matter who broadcasts it. (This block will credit p2pool, as well as whichever transactions the node operator configured to include or not include, depending on their preference for fee & blocksize configuration, or any patches installed, etc.)

The reward tx is part of the block.  I always find it humorous when people believe they need to change Bitcoin in order to save it yet have not even the most basic understanding of Bitcoin at a technical level.

When a miner solves a block he is finding a nonce that when ADDED TO THE REST OF THE BLOCKHEADER and hashed produces a hash below the target.
The blockheader contains the merkle root hash.  The merkle root hash is a hash of all tx in the block.  One of the tx in the block is the coinbase tx.

It doesn't matter who broadcasts a block the reward is tied to the block created. Changing anything in the block will result in new hash thus the "solution" (nonce resulting in hash below target) is tied to a particular block's values.

Isn't the thing you're describing basically just a less-decentralized version of what p2pool already does by its very nature? (and p2pool even manages to do GBT correctly from a python script, but without 100% cpu load at relatively modest hashrates)

---

Reference:

I mean if the miner gets enough data and happens to find a block himself, he may choose to publish it himself instead sending back to the pool. Maybe a silly question since I have not read the BIPs carefully yet.

The miners still send the pool a proof-of-work including proof the pool is being paid for the reward.

Thanks, Luke. I will read BIP 22 and BIP 23 carefully. Currently I am curious how you find a balance between giving back block creation right to the miners and avoiding selfish miners to hide the blocks they find for themselves. I am sure I can find answers in 2 BIPs written by you.

This is why decentralised mining is so important. When I finish GBT, pools won't have nearly as much influence as they do now.

I think you are too hasty to argue with others. Actually what you saying here exactly supports what I proposed.
Sybil Attack is harmful because 'one person has too much influence on the p2p network', and that's exactly what a big pool operator can do. 'Prove by Work' avoids someone to create a large number of nodes, but cannot prevent someone from creating a pool and attracts a lot of other nodes to contribute hash rate and becomes essentially same as Sybil Attacker.

One thing you said is correct. The current pool operator could create tens or hundreds of pools if there's some restriction on pool size. So restrictions on pool size does not eliminate big pools, but at least increases their difficulty.

Monopoly is not free market. You may think miners can easily switch pools so it is still a free market even there're only 3 - 5 big pools. This, however, is not true. It's only true if every bitcoin user has equal power of hashing and miners are all bitcoin users care about the future of BTC. Now, however, hashing power belongs to some professional miners or miner companies, who do not care the bitcoin user's benefit at all. Therefore, no free market here. DeepBit is pulling down by DDoS attack, not because miners leave there voluntarily. BTCGuild does not excceed 40% because their own discipline, not because the miners leave them voluntarily. Most large scale miners now only care about profit, not the future of BTC, since they are not necessary BTC believers, but mostly profit hunters.

No.  It isn't just tx which are psuedo anonymous, NODES are also pseudo anonymous.  Generally in decentralized networks this brings up a problem called the sybill attack.

https://en.wikipedia.org/wiki/Sybil_attack

Say there are 50,000 nodes.   What you don't know is that I personally control 49,900 of them.  Not very decentralized huh.  To the network all the nodes appear as equal peers but in reality a super majority are controlled by me.  Given the low cost of running a node security can't rely on majority of nodes.  It would be easy to circumvent.

Bitcoin sidesteps this problem with the proof of work.  A hash is a hash that is all that matters.   100 TH/s from a single node is no different than 1 TH/s from 100 nodes.   If such silly measures were attempted pools would simply operate out of multiple nodes or split into multiple virtual pools (where miners are transferred from virtual pool to virtual pool automatically to load balance).


Still topics like this make me wonder why so many people fear free markets.  Luke has miners because he built a pool and people saw value in it.  If in the future they don't see value in it the share of his pool will decline.  Case in point take a look at DeepBit.  At one time people feared DeepBit would 51% the network.  Seems quaint now.

p2pool is decentralized, and so it would be particularly hard to restrict if it ever got big...

but at the same time, each and every p2pool node is pretty much free to run or not run whatever sort of crazy patches for transaction-restriction / inclusion rules they want to use (or even just stick with defaults... it's really a matter of whoever configured it on a per-node basis 'cause that's how p2pool works)

More decentralization means 'less regulation'.

The one word no one wants to hear "Regulation".

One of the main reasons why people love BTC is  that they think BTC is democratic and nobody can damage it since it is protected by millions of users. Call me stupid if you want cause I feel quite unconformable now when I realized just 3 to 5 persons (the operators of BTCGuild, GHash, Eligius, BitMinter, and Slush. See https://blockchain.info/pools) can effectively damage the who ecosystem.

One example is the current proposal of Luke Jr (see https://bitcointalksearch.org/topic/miners-time-to-deprioritisefilter-address-reuse-334316). Yes, this time that patch may be not so harmful and you happen to side with Luke. But eventually you may disagree on other more serious issues but you have to put all your trust on that 3 to 5 operators.

Could we do something in the protocol, so that no mining pool could exceeds 5% of the market share and bring back BTC to be real decentralized again?