Topic: Stale block rates (Read 349 times)

I think your post supports my position.

A change in the P2P protocol would require or at least enable a behaviour change by a remote peer. Only one of these PRs do that. (Ideally they'd enable new functionality, though not always)

Almost all those commits in the recent change you listed are implementation optimizations, local behaviour changes, or refactorings and don't change the P2P protocol-- many don't change the externally visible behaviour at all.

I would generally characterize more recent activity as being extremely biased towards making changes that appear have generally no effect on behaviour  (if not absolutely no effect), which allows people to feel more confident that they don't break anything and makes the changes simpler because they don't require rewriting any unrelated tests.  This isn't to say that I don't think the changes are good, I think almost all of them are, but I would also say that only a couple of them accomplish anything a user might care about.


So there is only one thing that I would consider an actual change to the P2P protocol, and it's the realization of a protocol written in 2017.

In the context of this thread-- the question was observing stale blocks.  The idea of adding a P2P message to allow relaying their headers has been suggested many times probably going back to 2012. It came up more often post 2017 or so when it became clear that block relay improvements were making it extremely hard to observe stale blocks.  It would require a new P2P announcement since the existing headers message is a statement that you have the block-in-question available for getdata.

But no one is working on that. Now, if it had happened and been in your list I would have described it as an unimportant feature that wouldn't improve life for end users (though it would significantly improve developers ability to gauge the health of the network)-- given that and as you sample there are have been few recent changes that change the protocol, I really don't expect it to happen soon unless something changes.

Things aren't completely stagnant, e.g.

I've seen some activity towards revising ADDR messages. The current addr message are incompatible with v3 TOR hidden services, which have existed since 2017 and have much better properties. If this work is not completed and deployed within a year hidden services will stop working for Bitcoin users. Given the normal release and deployment timelines as well as the fact that the revised version is not merged or apparently merge-ready, I think it is now extremely likely that onion-peer support will be disrupted for users, in practice. Work on this began two years ago but didn't make progress for a long time.

There has also been some activity towards wtxid based transaction relay and the related erlay change.  There was some noise again on p2p encryption but it seems to not be particularly active.

But these changes are all moving extremely slowly, to the point where-- personally-- I forget everything about them between updates to them and don't find it interesting to contribute to them. Especially as I've found the testing framework makes it extremely tedious to work on any P2P behaviour change, because behaviour changes tend to break a wide assortment of unrelated tests, that people are slow to review or comment on them, and that their patches are continually broken by changes which are fast moving because they don't actually change behaviour.

Given that there isn't (yet) a fire behind p2p protocol updates required to keep onion peers working. I don't think it's unfair to characterize things as stagnant from the perspective of the p2p protocol itself and I really can't see purely instrumentation changes like stale-header-broadcast happening any time soon. (And I wouldn't recommend it: I'd recommend efforts be focused on addrv2 and other user-impacting changes like taproot!)

This statement runs counter to my intuition, so I made a quick comparison.  First, I assumed we were talking about the P2P network protocol ("the Bitcoin protocol" could also refer to the consensus protocol, but this conversion seemed to be about networking).  Second, I looked at all merges to Bitcoin Core during the six months up to today and the same period five years earlier.  Completely subjectively, I made a list of all the commits I thought affected how Bitcoin Core interacted with the protocol in a non-trivial way.  Here's the list for the the past six months (2020):


Here's the list for the same six-month period five years ago (2015).  Note, the merge messages back then didn't include a title, so I placed the subject from one of the commits in that merge in brackets:


So there were 22 (subjectively) notable merges in 2015 and 14 notable merges in 2020, a 35% decrease in the number of merges. In both cases, an overwhelming majority of the changes are tweaks to the existing logic mean to improve security, privacy, or reliability rather than more significant extensions of the protocol.  Looking at the titles of the merges, and again very subjectively, I think I'd consider the aggregate of the changes made in 2015 to be even more than 35% useful than the 2020 changes, so the gap between the years is arguably larger than suggested by just the numbers.

However, based on these data, I don't personally agree that "there isn't much activity in improving the Bitcoin protocol anymore."  The rate of change does seem slower, but I think it's far from stagnant.

Not possible-- of course it's always possible for people to talk in private but things have to be made public to have an effect.

I don't think it's good but I don't think it's particularly bad either.  There are a lot of improvements that would be useful, but at the same time Bitcoin works okay without them, and it's better things not be done than done poorly or thoughtlessly.

I assumed there is more development activity now than ever before, but that lots of it is just occurring via non-public channels. You've seemed to have taken somewhat of a break yourself, but perhaps you as well are more active behind the scenes? And are you feeling that things should be being improved that aren't? Or that it's good that protocol development has slowed down?

That is certainly my impression.

Really?

It's really hard to measure stale block rates right now.

Fibre and compact blocks have made propagation tremendously in the network, so now when a block is stale it's usually stale at the first or first couple hops.

As a result, the stale block doesn't propagate well (because it just makes it through zero to a few nodes before intersecting the wavefront of the earlier block).

You can think of it like this: stale blocks come from latency, latency arises from the mining process (miner devices, pool servers), from bitcoind, and from the network.  Advances in block propagation have really zeroed out the latter two classes. But mining process latency hasn't decreased (and in some places it likely has increased as miners have fewer stales to worry about), but the faster propagation means that your node will learn about all stales (including ones due to mining process delays) at a much lower rate.

If you want an even remotely useful figure you need to get the output from getchaintips rpc from a bunch of nodes spread all over the world.

Last time I collected stats I got data from a dozen nodes and I was still learning about more new stale blocks from the dozenth.

Really, there should be a P2P message added that allows nodes to announce the headers which are extremely close to the current tip for network monitoring purposes and which doesn't imply block availability like ordinary announcement does. ... but there isn't much activity in improving the Bitcoin protocol anymore, and improving stale monitoring would be one of the lowest priorities for that sort of work.

We need to be clear on terminology here. What you are referring to here is best called a stale or extinct block - a block which was once part of the main chain, but a longer chain took over. It's a misnomer to call these blocks "orphan" because their parent blocks are fully known.

Orphan blocks were blocks which had no parent. These were seen locally by individual nodes which had not yet received details of their parent blocks. These no longer exist since "headers-first synchronization" was implemented in Bitcoin Core 0.10.0.

That page hasn't been kept up to date in a couple of years.

In terms of keeping track of actual stale blocks - using the definition of stale blocks I provided above - then the best resource I found to do so is here: https://forkmonitor.info/feeds/stale_candidates/btc.rss. They also have RSS feeds for both BCH and BSV - just change the three letters at the end up the link.

I have also never come across any sites that keep statistics about stale blocks. As ranochigo pointed out, blockchain.com does keep a record of orphaned blocks and you can view those stats here: https://www.blockchain.com/charts/n-orphaned-blocks

Stale or orphan? You probably won't get reliable data on stale blocks since reference client doesn't usually accept blocks that are stale.

If two blocks are generated at the same time and a block is built ontop of one of them, then the other block gets orphaned. Its much more common to see these. Blockchain.com does have a page for Bitcoin.

Hello - does anybody know of any decent sites which track stale block rates for BTC vs other cryptocurrencies?

Thanks!