Topic: Why aren't alternative implementations encouraged? (Read 670 times)

when the majority of the main maintainers are working for 3 entities chaincodelabs, brink, blockstream. whom are sponsored by sponsors through a shell game that is rooted back by DCG,.. you might realise you are a bit too late to worry about centralisation of core
(and no dont deflect by listing less important unpaid devs with no merge privilege who only have done a few minor code tweak suggestions)

and thats the point. they need to diversify more.. why.. to diversify(decentralise)

pulling contributors away from core = decentralisation.

forget about 'making 'core smaller' where you think core need to be commander and chief

try and do this instead
have a coffee and sit back. and think of the decentralisation of BITCOIN
where there is no commander and chief. but instead multiple generals cooperating in a single network (think byzantine generals scenario)
ill give you a little hint. bitcoin already solves this byzantine generals problem since day one, but shhhh dont tell anyone you know. they wont like it as it hurts their loyalism to one brand

other brands not treated as opposition. not treated as 'stealing' talent. not treated as a threat to bitcoin.
i understand you have reservations about brand loyalty and corporate fangirlism where you which to defend a certain groups decision making power, yea  get that. but take a moment to reflect on bitcoin. not a certain groups roadmap plan for bitcoin

its not about there being different wallets. its about the hierarchy of proposals of upgrades being cores dominion.

im not saying 'competing' nodes need to fight core. im saying diverse brands not sponsored by the same umbrella company, where diverse brands can be co-decision makers of what gets into the release candidate that all wallets follow. where they offer that code out to everyone and THEN everyone chooses the new feature or not..

you know get back to the whole solution of the byzantine generals thing satoshi came up with, and getting rid of the commander-general(core) regime

EG if there is a proposal.
instead of core just making it like version 23.2 and everyone just blindly updates to it and by default it gets activated
or core puts in a mandatory activation if the upgrade takes too long, (disconnect nodes not flagging or rejecting blocks not flagging to force an upgrade by a deadline)..

but instead where all brands release a version (a) of their own codebases that also has a proposed code addition. where people wanting a feature but not wanting to have to download core. can stick with their favoured brand and still vote it in. and also the other brands dont lose adoption or get treated as enemies when a proposals is submitted because all brands offer it as a version(a) to allow their userbase the choice. thus it becomes a more cooperative open and decentralised vote and not a brand fight. of follow the commander-general or get treated as a sabatour/enemy.

it also helps by having all custom wallets ready for activation rather then waiting for a core vote to then decide if its worth changing custom wallets to then add new code. thus by all brands having the code before activation and offering it to its users, if/when it activates all brands are ready. thus all stay on an equal footing with core, rather than follow the leader regime

this ofcourse means that if another brand had a proposal. core too should have a core version 24(a) include that proposal for the core fans to have the option. where if all brands with a version(a) including core, got majority participation, flagging, and acceptance. then the feature activates.
and core also follow the new change even if they didnt 'invent'/patent/get sponsored for the feature/upgrade

where nothing happens unless feature activates (so no cries of 'forking' for just offering a proposal)

That wasn't the point. The point was, in my opinion, reimplementations should be discouraged because of the reasons that I posted in reply to franky1's post. That was only the hypothetical situation. BUT the actual situation is, there's not enough users that matter who run those other implementations/reimplementations, which could also be debated that it can only pull contributors away from Core, and could make Core's developer community smaller, and therefore possibly making its development process more centralized.

Hello everyone, my apologies for not quoting/following a specific post, I've read and enjoyed all, now I have something to share:


Any  software is susceptible to containing bugs, lurking in the shadows, waiting for the show-time.
However, bugs are not magic, they can be identified and fixed through code review and testing, yet there is a trade-off between 1)the resources one dedicates to this process and the wasted opportunities resulting from delays in launching the code, and 2)the actual risks that are involved and have to be addressed. An infinite resource and time allocation, theoretically can address almost all the risks, but it is not how we develop software in real world.

So, what alternatives do we have to deal with mission-critical software where failure is not tolerable? It is established practice: redundancy. Unlike what is trended here, I'm in support of redundant alternative clients.  

As of the fears, uncertainties and doubts which Satoshi has triggered about the possible chain splits, I'm not proposing it as a naive scenario in which competing clients are exposing the network to unintentional forks, etc. On the contrary, I think it was Satoshi who put bitcoin in bug-related risks by discouraging alternate implementations. Hence, CVE-2018-17144 was his fault and not Peter Wuille's. Devs are entitled to make mistakes once or twice a year, it is the architect who is in charge of fault tolerance and risk management.

Obviously, for bitcoin as a new class of distributed systems, naively recommending multiple implementations is not reasonable, the same as unbelievable recommendation of one god, one client! As a more detailed assessment of the risks would reveal, there are 3 class of untolerable, high priority failures due to bug where non of them is truly addressed by neither of the two architectures under consideration. Let's take a look:

Risk #1: Unintentional forks happen with disastrous consequences, because of different nodes, implementing different clients or different versions.

Risk #2: Nodes explicitly and widely compromise official protocol specs, yet consensus is reached by a considerable portion of the network.

Risk #3: Healthy, protocol compliant nodes are not able to bootstrap as a result of mild versions of the above case, where the breach is considered to be tolerable, hence no roll-back or even bug-fix is applied.


Single (dominating) client architecture:
#1- Is not fully adequate, even for risk #1 because of the versioning and slow adoption process. Despite huge dominance in the network, bitcoin development and versioning follows an exaggerated conservatism and undergoes an exhaustive dedication of time and resource in the upgrade process, a clear deviation of state of the art agile development methodologies with huge resource and opportunity lost consequences.

#2- Fails to meet any measure against this risk. The CVE-2018-17144 mentioned above, is the nearest case (yet not an exemplary one) where a bug can stay in shadows for long times, being inherited version by version, getting enough dominance in the network waiting for the show-time, usually the worst time ever.

#3- Can enforce circumventing measures like what is called "consensus bugs" by Core devs and BIPs are required to comply, a reasonable approach, I've to admit.

Naive multiple clients architecture: ( where we have different implementations installed by the nodes deliberately)
It suffers from the same problems as the single client model with different degrees of severeness, better for risk #2 and worse for #1, to be specific.


I'd suggest a third model: an Enhanced Heterogeneous Multiple Clients Architecture, as follows:

1-We classify nodes into 2 categories, high-end nodes with more critical mission with regard to the owners' expectations and resource allocation (we expect them to be properly aligned, i.e., more expectations ---> stronger nodes installed).

2-We encourage/support multiple alternative clients to be developed by different parties, with diverse platforms.

3-We certify, introduce the client software after passing the appropriate checklists.

4-Implementations are supposed to meet a well-defined integration spec via RPC/API/..., for supporting a robust voting module, called before  changing the state of the blockchain and (optionally) the mempool.

5-We recommend first class users to install 3 different clients of their choice, ordinary less critical users are free to choose one client of their choice,

As a result, bugs cause disagreements in high-end nodes which are resolved by voting, just triggering bug alerts, while the affected low-end clients are not supposed to cause catastrophic damages. Other than very high fault tolerance, it has an astonishingly huge boosting effect on development process.

AFAICT, there is no other comparable architecture for a system such as bitcoin while, interestingly, huge impacts on everything discussed here will be in the horizon, positive impacts, for sure. Bests of both worlds are achieved, and the network is guaranteed to have an astronomical level of fatal failure resistance. Use your own imagination  

Yes.

Yes, 24/7, with some minor halts for maintenance.

For example, there are nodes that accept free transactions, and they are used for P2P messaging. As long as free transactions are rejected by miners, it is safe enough. If those transactions will be accepted, then it is possible to switch to negative fee transactions with proper sighashes, then it is doubtful that any miner would attach its own coins to mine that.

It's great example. Now i'm inspired to try running alternate Bitcoin full node implementation and will write short review/guide about it within next few days.


Just wondering, do you think are any implementation which do more than following most/all consensus? If yes, do you currently run such implementation (even if it's not running 24/7)?

Let's not change the discussion from the title into whether BTCD was a good example or not.

Debatable. Because the BTCD developers are outside the political process of the Core developers. It can also be debated that IF BTCD starts gaining a larger following, it can create their own political process and compete with the Core developers.

In that hypothetical situation, it's probably A "good" for decentralization, but it's also debatable IF the "other implementation/reimplementation" is "still" the Bitcoin protocol.

Well yeah, as someone who is busy drafing a BIP right now (about standardizing the message verification algo - but that's an entirely different topic) - I am very happy that I can spearhead an improvement without having to maintain entire modules of BTC code first. And from what I know, the BIP merely has to reach a consensus with other people like me i.e. other mailing list posters - in order to be generally accepted. Here is a lot of power in the community's hand already, and pretty much explains, in a practical sense, why nobody has attempted a serious alternate implementation - because the BIP system for improving Bitcoin already lets them vote on changes they like and reject the ones that they don't like, so that's what they do. Read below for an explination for why this pevents protocol divergence.


Seriousness can't be measured in terms of developers. Because developers can quit but the project does not become any less serious. The quit factor will always be there, since someone started the project which means there is at least one dev for it.

They might quit because the code is perfect (in their opinion) and nothing more needs to be done. Or they might quit due to outside factors, entirely unrelated to code - especially because most of them work for free. Many an abandoned linux project falls in the second category of this thinking, by the way. We can't say that those are not serious projects.

Let me disagree with two statements that were posted here which I call "assumptions".

1. Alternative implementations would disagree with any protocol change and cause problems
First of all this is not a bad thing, Bitcoin is not a dictatorship to fear any kind of disagreement with or rejection of proposals by others. In fact other opinions contribute to improvement of the proposal and help see the mistakes and come up with better proposal.
Secondly other implementations doesn't mean the devs can't come together to discuss proposals in one place. In fact alternative projects brings developers from other programming languages together.
Finally, if the proposal is any good, there shouldn't be any disagreements or rejections.
In the end the community (ie. miners + nodes + economy) should decide not the developers.

2. Alternative implementations would have inadequate number of developers
This is a false assumption in my opinion. Any serious project would get enough contributors to keep it going. BTCD example I posted earlier is a good example of a project with enough serious developers, Electrum is another example (SPV implementation of the protocol).

What you describe already exists: Sidechains such as LN that at their base level are tied to Core and the main BTC network.

Sidechains lets folks explore a myriad of possibilities as long as in the end it's compatible with the BTC protocol while (hopefully) having zero or at worst only a minuscule chance of undesired impact to the main BTC blockchain. Any risk in using the side-chain MUST be only to the operators of the side-chain service and its users and not anyone else using BTC. Again, if what those side-chain folks come up with truly serves a truly needed purpose then ja, the Core devs should consider it provided it does NOT break anything or otherwise extensively modify the basics of how the BTC network operates.

That's very well put, and exactly how I feel. Have a merit and welcome to the forum!

Having the protocol be the least common denominator is exactly what the protocol architects should be striving for. Everything else is an implementation detail. To keep things truly decentralized the knowledge of how Bitcoin works should be spec driven. This allows every technical person to weigh in on the high level challenges of the protocol through documents like bips. A friendly environment towards more implementations also helps ensure a system of checks and balances where core devs can't dictate the direction of the protocol against the majority of user wishes just because they have market share.

A real-world case study of this approach can be showcased through web browsers. Sure most browsers are based on chromium but browsers like Safari and Firefox (and spin-offs like the Tor browser) have unique approaches to protocol implementations that many internet users value.

Agreed.
Considering that BTC is a Financial System, it is admirable that so far the Core dev team have taken what in financial circles is called a Fiduciary approach to making sure the BTC network works securely as intended including an immutable and easily accessible record of all TX's back to the Genesis Block and nothing that is added breaks its intended purpose.

In lay terms it translates to, "Once something Works - DO NOT SCREW WITH IT unless there is a more than compelling reason to change it/add a 'Feature' to it". When there is a non-critical but valid reason, get a feeling how the major parties affected feel about the idea. If it is worth pursing then review the hell out of the code followed by tests the hell out of it with edge-cases before release. Rule #1 to that is Thou Shalt NOT Break Code! Everything created since day-1 (TX's) MUST be maintained and unchanged and seamlessly accessible by the new code.

I'm more than happy to let the altcoin folks experiment with interesting ideas for using blockchains and I'm certain that the Core developers watch what they do as well to see if what the alt folks come up with could *possibly* fit in well with "The BTC Manifesto" so to speak.

edited for text flow

@HeRetiK: I agree with you, and I'm beating a dead horse at this point, but I have a different definition of "glacial" and would much prefer it if the design was basically set in stone, never to be tweaked again. I don't have the same appetite for improvements that most people seem to.

Of course, that's blithely ignoring real issues like quantum cryptography, but I suspect that even once Bitcoin is "post-quantum" there will still be many more upgrades than are strictly necessary, with each one being an opportunity to get something wrong, with possibly disastrous consequences.


@NotATether: Yup, can't argue with that.

I intentionally worded it like that, because I already addressed why its not encouraged in my first reply (i.e. the protocol becomes the least common demoninator, which is exactly what the protocol architects do not want).

Arguably that's pretty much what happened during the hardforks of 2017. The scenario you describe is less extreme, of course, but in the end that would be the logical conclusion.




Compared to most alts Bitcoin's development already is moving at a glacial pace And for good reason! But I fear increasing the political overhead would pretty much halt development and possibly even increase developer attrition.

I enjoyed reading your post, but I think that what you've mostly addressed with it is why it's not possible (in practice) and not why it's not encouraged (the point of the OP). Of course, one could infer that it's not encouraged because it's not possible, but I don't think that's what's going on.

After digesting some of the answers above, I suppose it's no great mystery why the Bitcoin Core devs don't actively encourage outside implementations. It would make their lives more difficult and give them less control over the protocol and that's probably all there is to it.

I'm not entirely comfortable having a "de facto" majority implementation. I think that's too much power to concentrate in one team of programmers.

Right now, changes to the consensus rules can be blocked by lack of miner agreement. I think it would be better if they could also be blocked by lack of programmer agreement (i.e. an outside team refusing to update their implementation).

I'm aware that this would make implementing new features (especially controversial ones) nightmarishly difficult, but (in principle) I believe that's a good thing.

Obviously, any idea that leads to a glacial pace of feature development is not going to be a popular one, and I'm mostly just fantasizing out loud at this point

I find it useful to elaborate on why we have the diversity of Bitcoin Core versions or, more precisely, why it has been possible so far to preserve successfully such a diversity. The answer is there are no hardforks in the Bitcoin protocol. Traditionally, hardforks are considered highly undesirable because they are backward-incompatible, they make consensus rules softer, causing an inevitable chain split and weakening of the bitcoin community. Part of the reason hardforks are discouraged is that they are forcing everyone to upgrade simultaneously and in a coercive manner, which is against not only bitcoin core principles but also principles of libertarianism and the concept of private property. Moreover, the lack of backward compatibility means that old versions are no longer enforcing anything (at least in a new chain), and anyone who wants to be a part of a new chain (perhaps the one with the majority of hashrate) is forced to upgrade to exactly the same version of bitcoin full node software as everyone else. In short, a hardforked chain will have only one implementation of the full node, which may cause not only an inevitable decrease in a network and development process decentralization but also a complete failure of the whole network.

TL;DR Alternative implementations are encouraged, albeit implicitly, via the universal agreement that hardforks, in general, are taboo.

In my opinion, there is nothing wrong with creating an ISO standard for some format or protocol if you do not plan to implement any major revisions for it, minor ones only. So on the contrary to what might be thought after my first reply, I don't see PDF's "staying put" as a bad thing, and as a file format its better that it stays like that.



Bug-resilience is only achieved if the N independent implementations all review every pull request that is submitted, only allow direct commits from a small group of people, and are tested thoroughly at a protocol level to stamp out regressions and accidential forks.

Can Bitcoin Core afford to do this? Yes, because it's manned by dozens of developers. Whether alternative clients could afford to do so however is doubtful. If you have an alternative client maintained by just 5 people, they might struggle to completely test and review all changes, let alone implement new protocol additions.

The hard fact is, most people do not want to maintain 200,000 lines of reference client code alone, which is the only reason we have a single client anyway. Otherwise this space would be buzzing with reference clients. Many promising OSS projects have been started by a group of a few people but fizzled out a few years later simply because the devs are burned out.

To understand how much 200000 (edit) lines is, I give this comparison: I myself am managing a codebase about six times smaller than this at work [nearly all of it written by yours truly], and even that is exhausting when done alone. I don't see how 5 or even 10 people can update a 200K-line codebase in a timely manner. So the question of having alternative implementations of a program at this scale has already been answered through the efforts of other developers: "No". (It's also the reason why, at 6 million lines of code, you don't see forks of the Linux kernel floating around.)

It effectively means that most clients would not styme progress by rejecting feature upgrades in Bitcoin Core, but they would quite literally not have enough manpower to implement them in the timespan of a few weeks. Thus the implementation takes several months or even years to develop, if at all. This unnecessarilly bogs down the schedule for rolling out the features to users at a protocol level, as there would be many users of the less-maintained client, but no work is being done to actively develop the required support for it although it could be desired by the alternative maintainers. In fact, it is very common for users to be using slow codebase, examples include every exchange whose wallet engine does not support native segwit addresses in 2022.

Or another example - Although some SPV wallets still don't support taproot, imagine if some of them were connecting to the protocol using an alternative blockchain, which itself has not yet implemented the taproot support code for lack of manpower. This would have had a downstream effect for those wallets which wanted to support Taproot addresses, but can't because it depends on support from the upstream reference client.