Topic: Why aren't alternative implementations encouraged? - page 2. (Read 685 times)

Very much agree. Different implementations should use absolutely nothing but documentation for reference. Unfortunately, most modern programmers only know how to solve problems by reaching for libraries so very few would be capable of producing a from-scratch clean-room implementation.

One thing that should be clarified is the definition of "implementation". In my opinion if you are translating code from another project/language into your own project/language then you are not "implementing" anything. If any bugs exist in the original project it will be transferred into your project and additional bugs will be introduced during translation since languages are different.

For example some implementations of BIP32 that copied each other without understanding the protocol had a bug where they didn't pad the keys properly when hashing them. It happened because they were copying each other instead of implementing the protocol. Other stand alone projects that actually implemented the protocol like Electrum never had such a bug.

Not quite.

Nodes will follow the valid chain with the most work (most work usually mean longest, but not always). The key word here is valid. If the node receives a block that it determines to be invalid, it doesn't matter how much work is done on top of that block, the node will not accept that chain. That's the problem with implementation bugs. If one implementation determines a block is valid, and a different implementation finds the same block to be invalid, that's a consensus failure and potentially a chain split.

For one, orphan races are not a regular event. For another, the average 10 min between blocks ensures that the losing chain is never more than at most 2-3 (very fast but not fast enough) blocks long so transactions in blocks built on the losing split are regenerated and soon re-confirmed in the longer chain. The Txs still exist in the mempool so no double-spends can happen, it's just that they have to be reconfirmed.

@achow101: I agree with basically your whole post, but could use some clarification on this bit:


I'm not yet far enough along my path to understanding Bitcoin to have much confidence in the argument I'm about to make, but I'll try it anyway

Doesn't the chain split regularly^[1], because of occasional mining "races"? And aren't these little splits resolved by simply following the longest chain?

In a very heterogeneous network with many different implementations and roughly evenly divided hashing power, wouldn't splits caused by implementation bugs resolve themselves in the same way?

[1] I mean regularly in the "as a matter of course" sense, not in the "frequently" sense.

It comes down to what tradeoffs you are willing to accept. Alternative implementations does not necessarily mean the network is more resilient.

Suppose there is some bug in one implementation that others do not have, and this bug causes that implementation to reject blocks that other implementations accept, thereby causing a chain split. This could be very dangerous and result in people losing money. And so people use this as a reason to advocate for homogenous implementation because chain splits are generally bad.

On the other hand, there could be a bug that causes the node to stop accepting blocks altogether. If only one implementation has this bug, they would stop accepting new blocks but the network would still continue operating as other implementations are not affected by the bug. And so people use this as a reason to advocate for alternative implementations because the network functioning is better than it being at a standstill.

IMO the reason is just a combination of the first mover effect and people taking Satoshi's word as gospel. Bitcoin Core is a direct descendant of the original Satoshi client and so people prefer to use it. Writing another implementation is also a major pain in the ass - there's a lot of stuff that needs to be implemented to be consensus compatible. Additionally, as you point out, Satoshi stated that he did not thing alternative implementations are a good idea, so many people have latched onto that.

Regardless, I would argue that we have alternative implementations already through people running different versions of Bitcoin Core. The major versions can be substantially different, especially once you get a few versions apart. Different versions have had consensus bugs that resulted in the same issues that alternative implementations pose. In terms of the features argument, people are encouraged to not upgrade until the accept the changes proposed in the new version. Soft forks are now done in minor versions for this reason - to allow people to get the non-consensus features of the major version without necessarily accepting the consensus changes. Just as an alternative implementation could just not implement a proposed soft fork, users can also just not upgrade to the version the deploys the soft fork.

True, and some people seems to support stronger decentralization, without understanding this drawback: soft-forks like Segwit or Taproot were possible, because huge mining pools supported those changes. But imagine that we would have truly decentralized mining, with, I don't know, 10k miners with much lower difficulty, or even more, where each of them would share their 80-byte block headers, that would be later claimed by publishing some data to some nodes. Then, forming a consensus around any change would be harder. And the same would be about many clients: if you want decentralization, you have to accept that drawback, and you have to accept the fact that people would say "no", and then you should know, how to handle it correctly (for example, one possible solution is to follow the heaviest chain of Proof of Work, and allow claiming only valid shares, and burning/locking the rest, so if the coinbase reward is 1 BTC, but the honest nodes only have 1% of the total hashrate, they should take only 0.01 BTC, and the rest should be burned/locked).

@NotATether: You see, I think that's a good thing. With N independent implementations and a reasonable distribution of hashing power between them, then as N gets bigger that "intersection" means a more bug-resilient network.

Of course, it also means a network that is much harder to implement changes on, but I'm not convinced that Bitcoin should be messed with all that often anyway. Personally, I would prefer it if Bitcoin was even more conservative about protocol changes than it already is.


@NotATether: Maybe I'm a masochist, but I think that's a good thing too. It's painful, but that's how "real" distributed consensus should work, IMHO.


@NotFuzzyWarm: On first reading, I struggled to reconcile that I very much agree with this, but you very much agree with NAT, but NAT very much disagrees with me.

NAT hit the nail on the head.
The whole point of a Standard is that no single entity - eg Adobe or Microsoft et al - can futz with it by adding 'features'  to make it more their own format - and in turn therefore attempt to make any other versions less desirable to use - based on their own particular marketing plans. Adobe in the past attempted to do exactly that every time a 3rd party put out a pdf reader/writer like Foxit Reader and initially MS's implementations of it tried to add 'Features' geared specifically  to the 'Doze OS. The whole point of pdf was to have a universal document format that is not tied to 1 specific software vendor or OS and to have documents produced using it to be able to be easily accessed for decades without having to find an old copy of the software that produced it.

Ja the bitcoin protocol is not managed by an outside entity like ISO but until (if ever) it is, having Core as the sole official implementation of it is the best we can have.

The developers involved with Core bend over backwards to ensure that any changes made to the BTC protocol have a REAL and useful functional purpose and do not break older versions. In the case of LN obviously the various developers could care less about that because they have their own agendas.

I enthusiastically disagree.

If there are two independent Bitcoin clients with roughly equal market share, then the protocol becomes the intersection of supported features on both clients. The protocol is no longer singularly developed ahead of the reference implementation. It will be possible for some feature to be added to the protocol (by people who also happen to be devs of the first clients), but the second client torpedoes the feature by not implementing it. This is the exact hell that is facing LN at this very moment.

It's the reason why you don't see any major improvements to the PDF format. Because ISO standards have a tendancy of staying put.

At least the last time I was dealing with it, in the financial markets it's actually closer to the BTC way. Several places had their own 'pretty' front end but the back end was all the same. The corporate users saw something different depending on where you worked, the consumers saw something different. The back end coding was from one place and it really did not change. BUT if you logged into your account from "A" brokerage and a different account from "B" brokerage you would think it was 100% different.

As for big pools and exchanges we are ASSUMING that they are on core we don't know if it is 100% the reference client or some heavily modified version of it or something else they put together. And for security they probably would not and should not tell us the truth.

-Dave

There is no way to check that, because if you have your own client, then it can still advertise as "Satoshi" with some version, so nobody would notice, that in practice, you use your own code. It is the same case as "User-Agent" field in web browsers, where the Tor Browser is simply "Firefox + Windows".

Also, even if some messages are fully removed from the protocol, some nodes could still handle that (for example, P2P marketplace from version 0.1.0), or they could allow more services than they advertise (for example P2P messaging, it is possible if some node accepts free or low-fee transactions).

@pooya87: That's a great point. The hashing power being distributed amongst a set of independently developed implementations would make bugs in a specific implementation pretty harmless.

It's a tricky matter as the way I see it there's good arguments both for and against alternative Bitcoin node implementations.

On a smaller scale we're seeing this play out on the Lightning Network: On the one hand it's great that there are multiple implementations out there as it helps with the proliferation of new ideas and people get to choose which LN node to run based on which featureset they prefer. On the other hand there have been compatibility issues between LN nodes implementations in the past [1], which in similar scenarios may be more problematic on the baselayer than on layer 2. What we also see on LN is that people unfortunately tend to flock towards single implementations [2] anyway. While not an argument against alternative implementations, in the case of the Bitcoin baselayer such flocking would make alternative implementations largely irrelevant.

[1] https://github.com/lightningnetwork/lnd/issues/6023
[2] https://www.bitrefill.com/lightning-toplist

A good question but only mining pool operators can answer this.

I'm aware, and the following question is not rhetorical: Do any other implementations see significant use by miners?


Obviously, not the clearest sentence construction on my part, but that is what I meant.

I am personally a supporter of alternative implementation of Bitcoin because of this reason:
If the first version screwed up (eg. value overflow in 2010 and in 2018), the user experience would reflect badly on Bitcoin.

With multiple implementations of the protocol, the chain's survival is ensured because for example in 2010 when the chain started growing with invalid blocks the alternative implementation would have rejected those blocks and any miner running that full node or multiple nodes would have built the correct chain ensuring that bitcoin network doesn't halt and could even prevent fraud such as double spends when the invalid blocks were removed from the chain.

Alternative implementations do exist though. Like BTCD.

That's true about a lot of other open source projects too. I rarely see any that is well documented. Most of the code in bitcoin core is straight forward though and if you know c++ you'll have an easy time understanding it all, and if you know other languages you should still be able to figure most of it out.

I agree with this but more like the documentation should be improved because it already exists on bitcoin.org, bitcoin wiki and among BIPs but some parts aren't good enough.

There should be only one reference implementation but other alternative options for people to run as full node.

@franky1 posted something interesting (to me) regarding how alternative implementations of Bitcoin are regarded, here's the relevant snip:


I find that really unfortunate. I think alternative implementations should be encouraged.

I've been slowly working my way through old posts in "Development & Technical" (to understand the history of Bitcoin better). These snips by @satoshi surprised me a little:



I guess I can empathize with this stance a little, but I can't help but think this whole approach is kind of backwards. Bitcoin offers unprecedented freedoms (compared to fiat) to users of the network, and that should be loudly applauded. But why don't developers get to freely participate in this brave new world?

Now, I'm aware that nothing prevents anyone from attempting an alternative implementation and that someone could conceivably read the source code and proceed from there. But if the reference implementation is hostile to that idea and the core devs don't properly document what they're doing and work amicably with outsiders then any such attempt is surely doomed to fail?

Take PDF for example. I think it being an ISO standard is a great thing and I would be pretty bummed if Adobe had arranged things so that no one but them could provide a working implementation.

Shouldn't Bitcoin, first and foremost, be a documented standard?

Shouldn't the reference implementation be one of many?

Isn't that more in the spirit of decentralization?