Topic: Can we validate all the blocks created till date with the first Satoshi client? (Read 1555 times)

This was quite valuable for my own learning. Just my 2 sats.

I spend few minutes to find that commit. While i didn't one with 400k lines change, i found one with 292K addition and 293K deletion[1]. The commit message says it's just reformatting source code, although how many people verify it. But submitting single big commit seems to be accepted by OpenSSL. Here's my finding,

• 420 commit with 1000-9999 line addition
• 20 commit with 10000-99999 line addition
• 3 commit with >=100K line addition[2], although first appearance is migration to Git or GitHub

Anyway, i'll leave command/regex i used to get and filter commit history.


[1] https://github.com/openssl/openssl/commit/0f113f3ee4d629ef9a4a30911b22b224772085e5
[2] https://github.com/openssl/openssl/commit/e613b1eff40f21cd99240f9884cd3396b0ab50f1
[3] https://github.com/openssl/openssl/commit/d02b48c63a58ea4367a0e905979f140b7d090f86

@gmaxwell: Thanks, man. What a great answer! I always enjoy reading your posts. You're an asset to this forum and if 1 out of every 100 members had half as much knowledge as you do, I'd be logged in 24/7

The latter: recent versions don't use OpenSSL at all (or didn't last I checked), and it hasn't been used for consensus since 0.12 in 2016.

Back when Bitcoin was written OpenSSL had been very stable for a looong time and had a good track record of preserving legacy interfaces and behaviors when it changed.  Satoshi could be forgiven for thinking it was sufficiently stable, particularly because it was far from clear how tricky consensus consistency was.  Your use of the word "minimize" makes it clear you know, but it wasn't always quite so obvious how truly difficult it was.

In 2015/2016-ish a number of concerning vulnerabilities were found in OpenSSL and a lot of attention was put on the project massively reworking things, making incompatible interfaces changes, and putting out giant updates (I seem to recall one of their diffs was like 400k lines...).  They managed to make changes in signature validation that would have been utterly consensus breaking for Bitcoin, and did so without even release note-ing the change.

Fortunately by then Bitcoin had mostly completed the move off of it and had deployed a soft-fork to restrict the set of accepted signatures going forward enough that their consensus break wouldn't matter for newly created blocks, as a result of an openssl consensus consistency vulnerability we'd discovered.  Though there is a window of openssl versions + bitcoin versions where it will compile and run without issue then reject some older blocks due to openssl's changes, this footgun is mostly avoided due to the fact that openssl broke their api a number of times around the same time.

It could have been quite a mess, had the openssl behavior changed before bitcoin devs realized it was a potential source of issues. Today AFAIK bitcoin uses no external dependencies in consensus beyond the programming language itself. Maybe some small amount of boost usage still exists in consensus, though there was a long term effort to eliminate it, simply because the authors of external code can't be expected to manage the very special requirements of consensus consistency.

The patches people use to use modern openssl with old bitcoin code paper over the major known consensus change in openssl, but there isn't any guarantee that there haven't been other ones or that its a complete fix (since even at the time openssl made those changes the 'fix' was to upgrade to bitcoin that didn't use it for consensus).  Not that I think it's too likely that there will be extra issues, but if the exercise is to tell if old versions would work, you should reconize that if you use an inauthentic openssl you're only getting an approximate answer.  

Because post-BIP66 bitcoin is very exacting in the signature encodings it will accept, I think it's unlikely that any new signatures have been accepted that really old openssl would reject-- though not impossible: we did also find bugs in openssl's bignums, just none that we could find a way to exploit for a consensus split.  It's more likely that if there were issues it would be of the form of more modern openssl being even more picky and rejecting the chain when old openssl wouldn't have.

Yep, that might have been easier than patching the source. It's not a bad approach, however, if you have the time and interest. I imagine that manually patching each encountered issue will be pretty educational.

Can anyone elaborate on this? I find the wording a wee bit alarming. If someone as knowledgeable as gmaxwell is hazy on exactly what consensus changes might be caused by running with different versions of a dependency, then isn't there something wrong in how the software is put together?

The only sensible design choices are (I would think) to either roll your own stuff or to only call third-party libraries through carefully designed wrappers to prevent (minimize, more likely) unwanted consensus changes from sneaking in. Or is this a matter of older versions getting this wrong and newer versions getting it right?

It's probably better to run with an old openssl... god knows what other consensus relevant changes they've made.

I am trying my best to get 0.3.24 to sync to the tip. Where would the SSL DER guard patch be applied in 0.3.24? I have applied a generous DB_CONFIG file that should allow syncing to the tip.

https://github.com/bitcoin/bitcoin/commit/488ed32f2ada1d1dd108fc245d025c4d5f252783

Thanks.

Got it, goes into the key.h file.

https://i.imgur.com/NuR8kUI.png

I have seen people running newer clients on sub standard hardware / connections / configurations take that long and longer.
There are a few discussions about it. Along with the standard 'just because you can, does not mean you should' for some things.

But it is good to see that there is that much backward compatibility with this and that even years and years later it still works.

-Dave

0.7.x will do so too, you can see me mention it in my post above.

As NotATether  says, OP_SUCCESS makes a forced success.

The idea of OP_VER at least so far as anyone is able to infer was to be able to write scripts with rules enforced by new versions of the script evaluator but which would be accepted by old versions.   But this is vulnerable if you imagine that the script author could be malicious and intentionally trying to crack the consensus.

OP_SUCCESS accomplishes the same end, but it guarantees that a malicious script author can't use it to write a consensus cracking script that passes on 'new' nodes but not old ones.

This means that bitcoin could today just easily add operations in any of the values used by OP_SUCCESS now, with whatever semantics are sensible (e.g. doesn't have to be a _VERIFY style opcode that produces no output and can only fail the transaction) and without a whole bunch of cautious analysis about how they'll be evaluated by older code.

That's all.

I think that at some point people even discussed being able to have a set of script blocks whos success was logically ANDed, where some could use 'future' opcodes while others didn't, which would be even closer to the non-attack functionality of OP_VER (e.g. allowing partial rules for nodes that supported them). But it wouldn't be easy to make a case for taking on a lot of complexity since it's not really clear what use it would have.

There had also been some suggestions a while back of generalized taproot that let you have a tree where each node potentially had key with a potentially hidden script, and then in that case the script could use future (OP_SUCCESS) rules while older nodes would still validate the key's signature, but not the script. This structure was motivated by making signature aggregation still work with softforks-- so the extra complexity would be justified on that basis and the extra 'partial validation' would be a very minor side benefit.

All OP_SUCCESSx opcodes can be redefined (once) using a soft-fork to make it conditionally succeed or fail. For now nearly all of them just abort validation with a success code immediately, before anything is even evaluated.

Considering that the only versions that I know of in Taproot is the defined Tapleaf versions (currently only 0xc0 is defined) and the fact that OP_VER seems to be pushing the protocol version to the stack (doesn't look like the value depends on the stack itself.), and the fact that Taproot interprets 0x62 as OP_SUCCESS, could you explain what you mean here?

Both Jameson Lopp and I have attempted syncing older versions recently: https://blog.lopp.net/running-bitcoin-core-v0-7-and-earlier/ and https://achow101.com/2022/03/syncing-0.5.0.

Without BDB locks configuring, 0.3.20 gets to block 124,275, 0.4 and 0.4 to 258,354, and 0.6 and 0.7 to 364,670. With a high enough BDB locks settings, I was able to get 0.5 to tip after 2 weeks of syncing. I didn't try anything older all the way because I didn't have the patience to wait that long.

Also ran into an interesting P2P incompatibility with 0.5 sending a headers message that was too big.

I tested the oldest that would run at all for me.  But I know that prior to 0.8 that you will have to do something about the BDB locks situation,
I also know that the BDB locks fix shared around back in 2013 will only take you up to 2016 or 2017 or something and then it still runs out of BDB locks.

Part of the reason that the community fixed the BDB issue rather than figuring out a rule that would mandate blocks that were always be valid to all nodes and softforking it in was that BDB was complicated enough that it seemed infeasible to come up with such a rule.

For my purposes I wanted to validate how far back you could go with an utterly stock node, unattended, unconfigured, no debate over what constitutes a hardfork or not.   I was checking this because faketoshi sued myself and a dozen other current and former Bitcoin developers demanding that we write backdoored node software which will hand him control over the huge stash of coins he claims to own (but claims to have "lost" to "hackers"), or otherwise pay him billions of dollars in damages.  We pointed out that even the people who are still developers lack the ability to help him pull off his heist because users simply wouldn't run the backdoored version in any quantity.  Faketoshi countered that users have no choice but to run the latest version.  Part of our reply was 'Users can and do run old versions or completely independent versions, in fact the most recent isn't even the most popular and in fact versions from about ten years ago just work.' So I wanted to make absolutely sure that 0.8.x did work and that the nodes on the p2p network claiming to be 0.8.x were actually working and not fake or something.

For a long time while I was working on Bitcoin I kept a fleet of older versions running specifically to monitor compatibility, but I haven't worked on Bitcoin in quite some time,  and efforts like keeping old nodes running kinda fell by the wayside as the environment become more hostile which switched Bitcoin from being something I loved working on to something I worked on to avoid letting my friends down.


Some recent versions like maybe 0.18+ also get angry at some older versions in the 0.7.x region at least and tend to disconnect them because there were some limits added to some messages like getheaders and 0.7.x will respond with too many.  This isn't a consensus change but would get in the way of doing a test.  Interestingly, running a sufficiently OLDER version would still work (at least for that reason) because they don't produce the messages that anger the recent code.

I think basically 0.8 is as far back as you can go before there starts being a lot of difficulty and asterisks.  It's also hard to impossible to find binaries older than that (as they got taken off hosting sites due to security issues with them) unless you already have them,  so you basically need an old operating system to build them from source.

My belief is that if you work around/fix the BDB locks issue then its likely code all the way back to where Satoshi added NOPs in script will validate the blocks, but anything prior to Satoshi's changes in 0.3.7 in 2010 are likely latently hardfork incompatible due to the verifyscript split.  By latent incompatible I mean that they'll likely accept the current chain, but someone could craft a special transaction today that would be accepted but they'd reject-- so a hardfork but not triggered yet.

(and I say 'likely' because I haven't seen it actually tested. Sometimes something you think would be a consensus break is actually prevented by different code, so there is no replacement for actually testing)

It's pretty clear that Satoshi didn't *initially* have a comprehensive understanding of the criteria needed for consensus stability as people do today.  You can see him figuring it out through changes like eliminating OP_VER are introducing OP_NOPx,  so he made a number of changes early on without much care if they created latent hardforks.  At the time he was doing this stuff, no one else really understood it at all, so there isn't any discussion about the details (at least not in public).   Because the hardforks were latent though you could continue to run the old code, making it appear completely compatible even when it wasn't, so it's only been through careful historical analysis that people have gone and figured out exactly what changed where in what ways.

The definition of "hardfork" has also shifted a bit over time:  Three major kinds of "hardfork" come to mind,  from weakest to strongest:

(1) A latent hardforking consensus change:  This is a rule where a node will accept transactions that the 'prior' code wouldn't accept.

(2) A triggered hardforking consensus change:  This is a latent change where a triggering transactions actually exists in the most work change.

(3) A hardforked chain:  This is a triggered hardforking change that was triggered at a time when enough usage of the 'prior' version existed that a separate chain formed and consensus was actually split in practice.

I think the original usage of "hardfork" tended to be mostly refer to the last type or rarely the others,  but in modern usage the meaning has shifted more to the first type-- largely because a focus on not accidentally creating latent hardforks is how you avoid a hardforked chain.

It's only the last type that has an active impact on users, and this is part of why you'll find quite confident messages from people (myself included) that say that Bitcoin has never had a hardfork except maybe if you want to count that BDB issue.  Yet today you can also find threads like this one where I'm pointing out latent (and maybe triggered) hardforks created by Satoshi (accidentally in some cases, maybe in all cases).

You could also come up with some levels between 2 and 3 that would be interesting and relevant:  Even if a forked chain didn't form were any users forced off software they preferred to use.   If some hereto unknown latent hardfork from 0.3.x days got triggered today it would likely be completely unnoticed and not disrupt anyone.  But if a hereto unknown latent hardfork from 0.11.x were triggered today it wouldn't form a split but it would almost certainly disrupt some users, cause some problems, and maybe leave some people unable to use their favorite choice in software.

Then there are other things like-- is removing a checkpoint that fixes the hash of block 1 a hardfork?  -- in one sense it's a latent hardforking change but one that could ONLY ever be triggered if the entire chain was erased with a gigantic reorg, rendering the entire system unusable and pointless.  Personally I think it's worthless to consider such things as hardforks. I think it's best to only consider things that could have at some point could have credibly created a hardforked chain in an otherwise working system to be hardforks.   If the definition is expanded too broadly it loses its power to aid understanding.   Like, e.g. Bitcoin should be extremely careful about introducing hardforking changes: But should it be extremely careful about removing ancient checkpoints? No.  Only careful to the extent that they might be providing protection against something, not for consensus stability reasons.

If your interest is mostly in 0.3.10 - 0.8  if you want to deal with the BDB and P2P issues then I'm quite confident you'd find the consensus rules are completely compatible.  And if you found that they were incompatible it would be a new and interesting discovery that might inform software development practices-- because if it's not compatible (for reasons other than BDB) it would be due to a yet unknown accidental triggered hardfork.  I'd make a sizable bet no such thing exists though.  If your interest goes much older, I'm less confident but also because of self-inconsistencies like BDB and OP_VER I think it's just less interesting: old enough versions were too broken to be treated as a viable consensus system.

Did you test older versions till you found one that worked or do you just know that they will not?

I'm not motivated to go install & update an old 32 bit OS just to run the old client but it would be an interesting experiment if you did not do it.

You are WAY more informed on this then me, but if 0.8 can sync, if I can can get it running from what I can see 0.6.3 should also, but it may not work properly on a modern OS. Eliminating the security implications running it on 32 bit Windows XP should work.

Just wondering, not that important.

-Dave

You cannot. BCash and BSV have hardfork changes and accept (and mandate) transactions that were never valid in Bitcoin, and their chains contain these things and not even even with configuration changes can you get old Bitcoin nodes to accept them.

In particular, they changed the consensus rules to mandate that every signature use the segwit signature scheme that the Bitcoin community created to solve the O(N^2) hashing problems.  The result of this is to Bitcoin software (of any version) these altcoin blocks look invalid.  In Bitcoin that new hasher also exists, but it only applies to coins that opt-in, so that they still validate under the old rules.  These altcoins are also are invalid from a proof of work perspective because they changed the difficulty update rules to allow them to "instamine" about 100k coins in a short period when the fork was created.  There are a bunch of other deeply incompatible changes in them, none of which can be configured out.  To a Bitcoin node (of any version ever) every Bcash or BSV block looks like it's entirely full of transactions stealing coins with completely invalid signatures and will reject it (except for the fact that it would have also rejected it for invalid proof of work and never processed it because it has vastly less work than the Bitcoin).

Looking at this old thread I think more modern understanding has improved-- there were some changes made by Satoshi that were likely latent hardforks in that someone could craft a transaction that would be valid today but not under the very first release, but those transactions didn't exist then and maybe haven't yet been created.  Though only Satoshi has ever made such changes, and it's not even clear how intentional the compatibility break.

The closest Bitcoin has come to a hardfork after Satoshi is that prior to 0.8 all bitcoin nodes would 'randomly' fail to accept blocks that processed too many inputs the too many threshold was specific to each and every computer, though it started being an issue with blocks >500KB in size.  0.8 fixed the bug (accidentally, because it wasn't even known to exist at the time, only somewhat after 0.8 was released when Hearn pressured miners to produce larger blocks). As such if you want to run versions prior than 0.8 from 2013 (almost a decade ago) you need to get around this random failure issue, which at least for a while could be accomplished with just a configuration setting.  Some people call fixing that bug a hardfork, though from that perspective every node on prior versions was hardfork-incompatible with every other node. The whole idea of a hardfork doesn't really make sense when talking about a system that doesn't form a coherent consensus with *itself*.  And even then, it wasn't even an intentional change but an accidental one.

A couple months ago I synced a totally stock 0.8 release binary from early 2013 all the way to the current tip, with no special configuration or anything-- it just worked (though very slowly).  This test lets you easily be absolutely confident that there have been no triggered hardforks in the last decade. (Latent hardforks, though, can only be found with careful code review-- certainly there have been no intentional ones since Satoshi, maybe never)

Testing older than 0.8 is harder because really old binaries don't work right on current systems and building the old code with a modern operating systems also doesn't work right.  Satoshi also made an incompatible P2P change in 0.2.10 with a time delay that didn't activate until 2012-ish, so any node older than that requires modification or a custom gateway to get connected to more modern nodes.  Back in 2016-ish I'd also taken a 0.2.10 era node up to current then, but now it's too much of a pain to try (and without config changes it *will* get stuck on BDB locks, so its not clear what purpose the exercise would have).

It's also the case that _every_ version prior to the deactivation of OP_VER was latent a hardfork with every other version due to its design being flawed.  Of note: Taproot finally reintroduced the op_ver functionality without the flaw.

Why don't you try asking what you're actually trying to ask?   If you just want to know if Bitcoin is backwards compatible:  Well you can easily check for yourself that it is backwards compatible for a decade, simply by running a 0.8 node.  Going further back, if you do something about the BDB locks glitch, you can go back as far as the last time Satoshi changed the consensus rules, it's likely that you can go further back-- because most of his changes were softforks and whatever were hardforks are still mostly latent rather than triggered.

I think this level of backwards compatibility is both good and important since pretty much no one can say they were ever forced off their preferred node software by compatibility changes.  The same can't be said for numerous altcoins with constant forced hardforks, for example: In Bcash there used to be many alternative nodes "bcoin" "bitcoin-classic" "Bitcoin-xt" and several others, which were all forced off the network by hardforks even though they were actively used and their users were left forced to use software they didn't like (or write their own).

Is not it saying that you can validate all Bitcoin blocks with with some special configuration of Bitcoin Cash or Bitcoin SV client? So, those chains are not hard forked either!

Same can be stated about Ethereum. Ethereum Classic chain can be validated with Ethereum client with some special configuration. So, Ethereum has never forked!

Are not we getting irrational here?

No, Bitcoin has not hard forked before, and all releases are actually backwards compatible since all protocol changes have been soft forks. As 2112 said earlier, you can actually get past the BDB lock issue with some special configuration. So in theory, you could sync a 0.1.0 client fully up to date with some special configuration.

That means, in short, all bitcoin client releases were not backward compatible in nature. Does not it lead to the fact that Bitcoin has hard forked before?

P2SH was a soft fork so older versions which technically do not support it can still partially validate those transactions and blocks. However, in order to fully verify P2SH transactions, you would need to have bitcoin 0.5.4+

Do u mean blocks having P2SH transactions can be validated by first Satoshi client, i.e. bitcoin-0.1.0?