Topic: An analysis of Mining Variance and Proximity Premium flaws in Bitcoin (Read 519 times)

https://goolipell.medium.com/and-thats-one-star-landscaping-cafcc18fb9d
https://goolipell.medium.com/3-books-to-help-you-stop-overthinking-9ebd070641cd
https://goolipell.medium.com/amish-brothers-rape-12-year-old-sister-get-probation-961c55d10745

Unfortunately this conceptual point can also be extended to transaction fees and the conclusion is that Bitcoin was intentionally designed to become entirely centralized in a winner-take-all finale.

You’re probably aware that there was already research modeling the potential for Bitcoin to be incentives incompatible as the protocol mining reward (aka coinbase) shrinks relative to the transaction fees. That research is somewhat debatable because it hinges on some assumptions.

Yet I think the more certain failure is much more simplistic. I was actually making this point for years also, and your thread enables me to frame it as an analogous concept. Each miner’s revenue becomes more dependent on transaction fees as the protocol block reward (aka coinbase) shrinks, yet transaction fees aren’t uniformly distributed across blocks. And this non-uniformity will become more excessive as Bitcoin is used occasionally by $billionaires who drop their huge value transactions on the blockchain and which can include a much higher transaction fee (that’s negligible to the value of the transaction) to insure their transaction jumps to the front of mempool. Constricting the block size to ~7 txns per second was a crucial design feature to insure the severity of this non-uniformity. This will have the analogous bankrupting effect on those minority hashrate miners and pools with less than the miner or pool with the most hashrate. So eventually all miners must end up in one pool or be bankrupted. Attempting to find a way to eliminate pools will not fix the problem if the protocol block reward is allowed to decline to 0 as it is so designed in the Bitcoin protocol.

At that point the single pool or miner can censor transactions, double-spend and dictate exorbitant transaction fees. So eventually the entire wealth in the system ends up in one entity on the hills of Jerusalem. Bitcoin appears to be a 666 neutron death star. Bitcoin was not designed by a lone hacker.  

EDIT: bankruptcy occurs over the long haul because that little bit of revenue advantage by even the slightly reduced variance accrues over time as reinvested in more mining hardware and capital to withstand the market price fluctuations and difficulty adjustments along with other unpredictable events such as the timing of the release of more efficient ASICs. It can be even simpler than that though because the largest capitalized miners may have an advantage on obtaining the cutting edge ASICs soonest. Tangentially everyone will be surprised if a surreptitiously mined legacy fork of Bitcoin is suddenly released in July and has already reached the next halving because the hashrate and thus difficulty adjustments had been monotonically increased since 2017.

Well, I'm not a big fan of conspiracy theories or a great mind reader, so let's skip this part to avoid derailing the topic.

As a proud Iranian I'm declined from accessing private repositories like what you have referenced by shitty github. So, please refer me to a url from free world a Chinese one e.g.  

Briefly speaking, you suggest that block reward halving in bitcoin makes it even worse for medium to small miners/pools to compete because ...because of what?
Unfortunately, I didn't find any direct correlation between: 1) centralization of mining because of pooling pressure and 2)halving.
It is true that halving is an important bolding factor for the scaling problem because without scaling there need to be either a significant drop in the amount of electricity-consumption/work which leads to huge security problems OR a dramatic increase in transaction costs and forcing day-to-day finance off the chain. Obviously, it would be a disaster by its own, but it has not much to do with centralization of mining that I'm discussing its basic mathematics (not economics) here.

Back to your conspiracy theory about winner-takes-all being intentionally implemented in bitcoin for dooming it to centralization, I've something to add here:
Centralization of mining is not a fate for bitcoin, there is a solution, a brilliant one, no forks, no upgrades, nothing, centralized pools are jokes, BIG jokes tho, they became dominant because of a series of incidents: Satoshi disappeared, Slush was a jerk, and most of the guys now considered as being prominent were just noobs with a very narrow vision. I'm not going to disclose it now, for some reasons, but suppose, just suppose there is a decentralized alternative to pools and revisit your dark idea of bitcoin fate.

From your post I understand you expect bitcoin to do something about the infamous power law distribution (and centralization) of wealth, well, sorry for the disappointment but bitcoin was not meant to do this from the first beginning, all it was supposed to do (and still is an obligation for it), is decentralization of monetary system, i.e. issuing and transacting money.

Excellent summarization.  

This topic didn't get enough attention in spite of the very crucial point that I was trying to prove: proximity premium per se is not a big concern when it comes to pooling pressure and mining centralization problem in bitcoin: Without the Bernoulli distribution's variance, i.e. without the winner-takes-all approach to PoW, proximity premium is not a determining force toward pooling.

AFAIK, before this article nobody had ever noted the interrelation between the two and the determining role of mining variance.

@gmaxwell
Reviewing this thread, I found that you have done almost a complete rewrite of your post above and my reply no longer covers yours properly.
Good news is your edit has signs of progress and I appreciate it, but I suppose it is not best practice, editing and re-editing,  so ...

Mining as a lottery (with single winner) suffers from variance as I have proved mathematically above-thread: winner-takes-all approach means Bernoulli distribution and for small p (probability of winning all instead of nothing) it results in very high and intolerable variance. Although mining is based on probabilities it is not gambling, high variance is not tolerable when the stakes are high. For large ps (>0.1) variance is very low and they can survive but for smaller shares it becomes too risky, they have just to back-off or join a pool.

Mining, the way you (and Satoshi was used to) put it,  as a lottery literally died 8 years ago. It belongs to POB, Plain Old Bitcoin, when the stakes were low enough and it was a fun and an experiment.  

As of proximity premium, what you call it 'progress',  the mining scene doesn't change radically. Your race analogy is good but as an extreme case scenario. In the real network we have a very short window (< 1%) for miners to take advantage of this premium and as I have discussed extensively the total hashpower of miners increases when the premium decreases (only a minority are in the premium in each round).

So, we have a short window of time and a minority of hashpower that is taking advantage of their proximity to the previous block's miner. Suppose the miners in premium have 10% power, their chance to win the next block would be like 1*10^-3  After the window closes (without a block being found) the scene is reset to become a lottery again.

As I understand what you are saying is 'if we don't take care of this issue, progress, the 1% window gets larger and this phenomenon becomes more dangerous'.

My answer:
1- The whole purpose of this thread is to convince you and other community members that this flaw, progress/proximity premium is a logarithmic homomorphism of mining variance flaw. For instance if the window would be 1 minute (10%) the chance for the 10% miners in premium to mine the next block would be not more than 1%.

2- Bitcoin relay network has been improved since bip 152 and compact blocks (as of my knowledge you were a contributor, kudos) and since then with FIBRE and similar relay networks which efficiently connect miners, the propagation delay is under control.

3- Generally discussing, propagation delay is a technological problem, much simpler than  game theory and mathematical challenges built into bitcoin. It is feasible to shorten propagation delay even more by improving relay protocol and using more advanced technologies but you can't do anything about variance as long as you keep winner-takes-all approach in Nakamoto's design unchanged.  


it is where you deserve to be slandered Greg  

You are literally ignoring all my arguments without refuting them! How is it possible to have premium without high variance? Instead of discussing theoretically you choose a 'proof by example' methodology:
You are saying that selfish mining and %50+1 attacks are real! Good for you, but what does it have to do with mining variance? A big miner (or a cartel of such miners) can damage network with our without variance.
My concern is prohibiting such a centralization by eliminating pooling pressure here, is it that hard to understand?
Nakamoto's winner-takes-all approach put bitcoin in danger 10 years ago, it took just 2-3 years for this mistake to become the main centralization factor and now you, one of the lead developers in charge of keeping bitcoin safe and decentralized, instead of isolating this factor and contributing to fixing it try to distract us by mixing up everything.

My proposal has zero relation with hashcash. Hashcash is an old proposal for DDoS and spam prevention, 2 decades old. What's the point here? Who says hashcash is low variance? You mean low difficulty?
So, your argument is because I advise lowering the difficulty (somehow) to reduce and control the variance, I'm rehashing old hashcash story, again and again? Is it how you read?
I'm sure that in the depths of your hearts, guys like you, are confident that nobody is capable of suggesting any thing new about bitcoin because  every possibility has been discussed already and noobs (like me) are just rehashing old stories. Right?

It is truly dangerous, and it is why I think you guys are no longer qualified to take the lead. You may be helpful or not as a developer but you are dangerous as a leader.

bitcoin is a system and as a system it needs evolution and improvement. We should observe and improve and no proposal should be cut-down by claiming it as a re-hash of a "broken" idea.

You are right. It is really sad.

First of all, Of course variance can not be fixed. What the heck? We are talking about mining variance flaw, about high variance for average miners who are forced to join pools because of it! My concern is fixing this flaw not the variance which is a mathematical index of how sharp (low variance) or flat (high variance) are distributed probabilities.

Plus, how fool you suppose your readers are?
This is it?
Variance, higher than network diameter(WTF?),  is essential to network to converge? WHY?
Again, your proof by example:
Kidding? It is not a zero variance situation, actually it is a system with a standard deviation of 1.  A normal random distribution!.

You have to check the concept more deeply: By mining variance flaw we are discussing the problem of players with small relative hash power that the cardinality and distribution of blocks they mine in a typically long period (which follows a Bernoulli distribution) has a large standard deviation that translates to high risk of hitting nothing at all (or a lot more than what they deserve). i.e. A high risk, high stake gamble.

Given above definition, what the heck are you talking about?

So, this is it! You are proving to some journalist that I'm not competent!  

We have trolls, scammers, shills, who instead of contributing in discussions and adding value, deliberately choose the other option.

Now we have a new phenomenon, a guardian troll, who tries to prove everything is ok, bitcoin is great, no vulnerabilities,  no flaws, no need to improve and these guys in cheap forums like btctalk are a bunch of noobs who have no clue and journalists can rest assured that everything is ok.

Very interesting, thank you for sharing!

It is not true! My claim is that a moderate increase in 'block size' (it is not the same as a wild, endless increase, a radical one as you put it) has zero effect on current (and recent years) scene of bitcoin mining, because it is already dominated by pools.
You claim otherwise? refute my reasoning up-thread, there. Please prove me wrong and show me that you guys have just one simple reason to avoid increasing block size 2-4 times.
Plus, I'm not a fan of block size increase, instead I support block time decrease which was undermined by your debate just like the first one.

Wrong again! Lottery, high stake lottery, with a winner-takes-all paradigm is not what a rational miner wants to participate in ever. You don't get it correctly. What I 'm saying and have proved thoroughly, is that for a miner with relatively low share in network hash power, say 0.00001 (which is  a LOT by the way) the variance gets so high (a standard deviation of 1.38^* in a year!) that it turns the mining scene into a high stake lottery for him and it is why he has no choice other than joining a pool.


Now what does it have to do with proximity premium and propagation delay?  

In this article, I'm trying to prove that proximity premium is an exaggerated consequence of mining variance. You think I'm wrong? Prove it.

Oh please! Give me a break ... You are senior, I get it but you have lost focus, I see it. I don't buy your history of glory and conquer, whatever. I see nothing good in current situation with pools and ASICs and who is responsible? YOU!

So, accusing me of being noob, the one who doesn't read and, ... just makes me laugh. And accusing me of carelessly 'smearing' you and guys like you is like saying 'Don't say a word against us or you are insulting us, smearing us. We are elites, we have been around for years, we are the ones who have a voice, you just STFU and watch our show'. I don't buy this either.

Wrong! As long as you are speaking of proximity premium and its relation to mining variance, I've shown that the first is a reflection of the second in a higher order. You think otherwise? Prove me wrong or refer me to true research that have proved it to be wrong.

I think you are the worst moderator ever Greg
You come in a topic, happened to be a pure analytical topic, and go completely off-topic, I have to report you to God

^*FYI: Reducing block time to 1 minute would reduce this variance from 1.38 to 0.436, such a variance is ways more tolerable and our miner would consider solo mining more seriously. See? A variant of onchain moderate scaling (block time decrease) reduces mining variance such that a considerable number of miners would continue solo mining and help keeping the network decentralized. Right? How do you justify your blind objection to such a proposal? By progress argument? Come on! Reducing variance helps miners to be compensated for their opportunity costs in real time because the variance is improved and they have more practical chances to hit.

----------------------------------------------------------------------------------------------------------------------------------------------------------------------
EDIT: By the way, since when trolling in a topic by a mod, deserves merits? @bob123 and @Foxpup don't you have any sense of responsibility?

What you are describing here as "Proximity Premium" is normally called "progress". Normally we think of mining as working as a lottery: Your chance of finding the next block is just simply your proportion of the hashrate out of the total.  But in a system with progress things work more like a race: the fastest party wins more often (or even always, if there is a lot of progress). Lottery vs race is a near perfect analogy-- in a lottery your chance of winning on a ticket doesn't change because of your prior ticket, but in a race your chance of winning in this step depends critically on all the steps before it.

Progress can be introduced into mining in many different ways, -- including from things like bad POW designs that try to reduce variance by making mining incremental, or blocks simply taking non-zero time to spread and validate across the network -- but the end result is the same: Progress creates an insidious centralization pressure where the bigger miner actually gets a better return on investment.  Due to re-investment even small amounts of progress could conceivably completely centralize a system even absent other centralization pressures.

Progress is a lot more hard hitting than most people expect. You might look at 6 seconds of delay and wonder how could that matter against a 600 second block interval.  But because of the nature of poisson processes most blocks are found much closer to each other than 600 seconds.  We saw the direct effects of this in the network, when blocks started getting larger than 200k propagation times started getting upwards of 2 seconds and miners saw higher orphaning rates and consolidated onto fewer larger pools, even giving a single pool a super-majority share of the hash power for a little while.  After we invented and deployed new technology that made blocks propagate faster, the consolidation reversed.  Unfortunately, there is probably no amount of progress which is "safe" -- at best it's only small enough to be too small a centralization pressure to worry about compared to other issues, and the improved propagation requires cooperation -- if a large miner intentionally produces blocks with unrelayed transactions will immediately have the old slow propagation which will improve their own bottom line.  When it comes to security you have to build for the worst case, not just hope people will cooperate instead of doing whatever is best for themselves. So long as the difference between the worst case and typical is small there isn't much incentive to exploit it, for the moment it seems to be working.

Concern about progress isn't new-- It's been part of the regular understanding of the engineers working on Bitcoin for a long time-- and bitcoin's creator probably understood it too considering that he chose a 10 minute block interval (smaller numbers _look_ fine until you start trying to work out the effects of progress).

That is why I find it disappointing that A day ago you were claiming that there were no reasons to not radically increase block sizes and slandering all the engineers who've maintained Bitcoin its whole life. You have been more or less calling the people that built so much of what everyone uses malicious criminals, in post after post.

Today you've miraculous discovered that larger propagation delays result in progress which turns mining from a lottery into a race favoring larger miners-- Welcome to the state of competent Bitcoin development circa 2011. And thanks for the proof that you haven't even bothered to read any of the relevant discussions which you felt so easy about smearing people about...

What will you realize tomorrow?  Maybe that progress and variance are not one in the same and in fact almost any variance reduction proposal increases the potential for progress once you realize that participants will behave strategically instead of honestly. For a simple example of where making variance lower makes progress worse, say that you require that a block present 10000 POWs of 1/10000th the difficulty instead of one (your proposal basically does that, in fact): That is a system with very low variance (1/10000th) and almost perfect progress: If you had two miners with 30% hashpower that didn't share or only shared work with each other and four miners with 10%, the 10% miners would almost _never_ find a block. Again, not a new result: Low variance hashcash using multiple POW was described in the hashcash paper (section 6), and people have re-proposed it once or twice a year on BCT since the start. Maybe that if you don't bother researching you'll just rehash old and broken ideas and if you smear people you'll largely get ignored instead of getting responses that help you improve your own understanding...  Or maybe not, your posting history is littered with pumping your belief in scammer wright, after all.  

I think that is just kind of sad and very frustrating.  Even the idea of "fix"ing variance is somewhat confused on its face. Variance is utterly essential to the operation of the system, it _cannot_ converge without variance significantly greater than the communications diameter of the network: Imagine a toy example with a world wide network of nodes and two miners with equal hashrate, in a system with no variance they'll keep producing blocks at exactly the same time, so once the network forks once it'll never heal-- nodes will just follow the closest miner. The same holds when variance is too low rather than zero: you get long reorgs because far away nodes will only switch chain heads when a nearby miner in a tie gets unlucky enough to fall behind more than enough to overcome the communications delays, which happens only rarely if the variance is low.  This isn't speculative, e.g. there was an altcoin called "liquid bitcoin" (IIRC) with a fixed difficulty, once there was enough hashpower to make blocks fast it fragmented into hundreds of separate chains that were unable to heal and form a single consensus. Coupled with strategic behavior by miners (such as large miners delaying sharing their work), your proposed change would also create tremendous amount of progress (IOW centralization pressure)-- just like the example I gave above.

In spite of pages and pages of discussion no one has until now pointed any of this out to you, because almost no one technically competent will bother even reading your posts because you haven't bothered learning what you don't know and you spend a lot of energy making really nasty and unprofessional insults-- I only read any of it because your seemingly incorrect claims of "vulnerabilities" caught the attention of journalists and I was asked if you were full of it or not.

Right.  This is a huge disincentive to mine for most.

It seems that the "perfect" pow of algo would reward every mining participant exactly in proportion to their work expended towards securing the network, no matter how small without variance, and without need for mining pool middlemen that take a cut.  This should incentivize almost everyone to mine.

A practical problem is that rewards for such mining efforts are likely not even representable by the native blockchain token, eg less than 1 satoshi, and also fees are prohibitive.

I will read your proposal.

Interesting read, i wonder what other thoughts are on this.

Preface
The problem of solo mining becoming too risky and impractical for small mining facilities appeared in 2010, less than 2 years after bitcoin had been launched. Although Satoshi Nakamoto made a comment on bitcointalk about first pooling proposals,  it was among the latest posts Satoshi made and he just disappeared few days later from this forum, forever, without making a serious contribution to the subject. Bitcoin was just 2 years old when pooling age began and eventually dominated almost all the hashpower of the network, putting it in danger of centralization as its obvious consequence.

Since then it has been extensively discussed and become a classical problem named Pooling Pressure  in bitcoin and PoW networks which is mainly escalated by both mining variance and proximity premium flaws.

In this article I'm trying to show that:
1-Mining Variance is an inevitable consequence of bitcoin's winner-takes-all approach to PoW.
2-Proximity Premium is basically an amplified version of Mining Variance, Hence another consequence of that approach.

Readers may be already familiar with my proposal for fixing these flaws and removing the unfamous Pooling Pressure from bitcoin and its clones and other blockchains that inherited winner-takes-all from it, i.e all minable coins in the market!
As a matter of fact I was writing the (almost) final version of this proposal when I found myself examining these two flaws more extensively and I thought it would be more helpful to publish this part separately.


Mining variance flaw in traditional PoW
The binary nature of Winner-Takes-All strategy in bitcoin,  implies that a miner's chance to win a block with a relative hash rate of p, will follow a  Bernoulli distribution.
The variance of Bernoulli distribution is known to be p(1-p) Hence the standard deviation for N consequent blocks is:

For N=365*24*6 = 52,560 blocks sqrt(N) ~ 229.26
Calculating standard deviation of p for  the following series  (representing respected hashpower ratio of some miners)
0.1,           0.01,             0.001,                 0.0001,                0.00001 ,       0.000001
generates the series
0.013,        0.043,           0.138,                 0.436,                 1.380,            4.362

This quadratic  increase in standard deviation as the hashpower ratio of the miner decreases is a direct consequence of Bernoulli Distribution which in turn is a result of binary nature of winner-takes-all approach (you win/you lose, 1/0).

This illustrates how the risks involved in mining with medium to low hashrates in traditional PoW, are that high to make solo mining for average miners just like participating in a very high stake lottery with a single winner each round. This is  kinda gambling that only hobbyists may be interested in and no rational investor would take it as a serious investment opportunity, unless he might be able to install and run very large facilities.

The direct consequence has been experimentally proved to be a pressure toward forming pools, typically centralized entities who aggregate their own hashpower with their clients' reducing the number of human beings in charge of mining in the network dramatically to very low thresholds.

Proximity Premium flaw
Although Mining Variance is important enough to push miners away from solo mining it is even boosted more by proximity premium, another flaw:

Propagation delay of announcements (most importantly new-block-mined information) gives the node nearer to the source (primarily the source itself) a premium by which they start mining the next block sooner than other participants that are losing their electricity and opportunity costs meanwhile. A thorough mathematical evaluation of this flaw is not available as of this writing, and as of my knowledge it would be hard to produce one. The model proposed here is rather qualitative and based on simplifying assumptions to be used instrumentally to prove my point here: Proximity Premium flaw is an amplified version of Mining Variance (analysed above), hence it is another consequence of bitcoin's winner-takes-all strategy.

The P2P network of bitcoin can be modeled as a weighted undirected complete graph, PP with nodes as the members of P, the vertices and edges being weighted by the number of nodes in the shortest path between every two node (minimum number of hopes from one to other) in the actual P2P network graph.

Assuming that for every specific type of information there is a same propagation delay for adjacent nodes (not necessarily an exact approximation of the real network),  the edges in PP would be assumed as a scale of the amount of time needed for information to be received (and verified) by each peer.

Every node p_k of such graph, partitions the network to the subsets of nodes having the same weight on their edges to p_k. i.e.:

i.e.
I_k,0 = {P_k}
I_k,1 = {p_i| p_i∈P & h_k,i=1}
...
...
I_k,m = {p_i| p_i∈P & h_k,i=m}

Where m is the maximum edge weight for node k on the complete graph And we have I_k,0 U I_k,1, ..., U I_k,m = P.

Now, one may be interested in how the cardinality of each partition I_k,i is dependent on i. Obviously, it depends on P2P network topology and differs from node to node and for each node from distance to distance. But if we had a choice to impose a strict constraint on the minimum and maximum of peers and at the same time forcing nodes to peek their peers randomly we would observe that in the first few hopes we have few nodes 'near' to the source while the majority of the nodes are 'far' enough to be considered in danger of losing opportunity costs because of working on informations being outdated already.

It should be noted explicitly that the impact of this proximity flaw is not only determined by the distance but also by the nature of the information under consideration.

In Bitcoin network there are 2 types of information continuously produced and distributed by peers: transactions and blocks.

While transactions happen and are relayed constantly their importance for miners are not critical and they don't lose too much because of propagation delay of this type of information. The only risks involved is the impact of such a delay on verifying blocks containing such transactions and missing the opportunity to include them when they have higher fees.

But for 'new block mined' events, the impact of being 'far' from the source is disruptive and causes miners to lose opportunity costs linearly with the length of the time they are kept in darkness. This opportunity cost is proportional to the miner's relative hashpower and could be expressed by means of a scalar variable.

It is theoretically possible to have a thorough analysis of Bitcoin network and suggest the average opportunity costs of each miner when blocks are found in other parts of the network. Such an index would be a function of the network topology because it is expressed  as the ratio of resources miner holds.

At the first glance it may look somewhat odd, big miners lose more because the ratio would be applied on larger resources. But another factor should be taken in consideration: the fact that miners with larger resources have a proportional hashpower and the probability for them to be in bad topological position is again proportionally less than small miners.  

So, everything seems to be in an equilibrium , big miners get premium more frequently and it compensates for situations that they are losing opportunity costs because of topological reasons and the fact that they have not mined the new block. Small miners lose a percentage of their resources which is not too high but this lost happens more frequently.

In reality, two factors change this picture radically:

Firstly,  the topology of the network is not that random and even. Large mining facilities utilize more powerful .
(set of) nodes able to maintain large number of connections with better bandwidth and fault tolerance.

Secondly, and most importantly, mining variance discussed above applies here in an order of magnitude worse scenario:

For a node to take advantage of its premium (in the very short window of time it is in premium) it has to build and mine a new block from scratch.
For a small mining facility it is practically impossible. e.g.  for a node that mines a block every 1 month (yet it is what you may get from a farm with 20 S9s)  or so, it is very unlikely to happen in the fraction of seconds it is in premium.

Hence, while small miners are losing opportunity costs every single round, they will never have a practical chance to be compensated ever.


It is the same assertion that I've made in the beginning and is the main motivation for publishing this analysis:
Proximity Premium flaw  is basically an exaggerated version of Mining Variance flaw and any fix/improvement to the latter would fix/improve it as well.