In my discussions with various members of Core, I have reached the conclusion that most of them simply disagree with the design of Bitcoin, which by design allows the consensus rules to be changed by a sufficient majority of miners and users, independent of what any group of technocrats wish.
It is important to remember not to attribute malice where ignorance is equally explanatory. All of the devs I engage with are very strong in cryptography and computer science, which may make them less accepting of the fact that at its core, Bitcoin relies in the economic self-interest of the masses to govern consensus, not a group of educated technocrats. As an educated technocrat myself I can understand the sentiment. It would be better if math and only math governed Bitcoin. But that's not how Bitcoin is actually governed.
At the end of the day, if social engineering and developer manipulation can kill Bitcoin, well then we're all betting on the wrong horse.
That's a good quote here. In fact this governance problem not only happens on bitcoin, but any kind of large scale social community. That's why politics always exists. Notice that politics does not exist on gold since no one can change its property, but bitcoin's protocol is destined to struggle
You have several genius/technocrats each promoting different future plan, and majority of normal users would not be able to understand the pros and cons of those plans. So it finally ends up with politics, e.g. the solution who gain the most support wins, not the most brilliant solution wins. So trying to prove that you are most brilliant does not help to win the vote, instead you must try to study what is voters' favorite
In this case, what majority of uses want? Trust and stability. They might be willing to pay a little bit higher fee or use other off-chain services for daily small transactions, but they definitely don't want to see a split in the core devs and even strange forks that might destroy the whole ecosystem. When bitcoin worth nothing, all its function becomes useless, so to preserve its value is the highest priority here, and in order to do that you must give up on certain functionality
I think you are missing some important aspects to this, Bitcoin is not the same as other large scale social communities, Bitcoin has solved certain age old political problems exactly because of the ability to hard fork and split. The rules of Bitcoin can be changed, and this should and is determined by the market, unlike gold.
I also think that moving the majority of transactions off chain is not a solution to scaling the Bitcoin blockchain at all. This transactional volume needs to take place on the main Bitcoin blockchain in order to pay for its security over the long term, high volume, low fee should be the future of Bitcoin.
We also do not need to give up on any functionality like you say, this is a false dichotomy. Bitcoin can be a currency and a settlement network, they are actually reinforcing and synergistic of each other. Furthermore I think that mass adoption as a currency is critical for Bitcoins success over the long term, also because of the massive and profound change this would bring about in society.
When you say that people will just be able to pay a slightly higher fee I do not think that you get it. If the blocksize is not increased then transaction will become much more expensive and unreliable. It is not the case that you just need to pay a higher fee, in fact there will not be enough space for everyone to transact regardless of the fee that is payed. Bitcoin would become a settlement network for larger financial institutions, it certainly could not be used by everyday people. I do not even see this happening in first place however since under those circumstances I do not think it would be able to compete with alternatives.
Especially considering that it will have to co exist with a chain fork with the same genesis block, which will offer cheap, reliable and fast transactions. Without the need to sign up for or choose any third parties. Because of this utility and ease of use I believe the big blockchain will be able to out compete and therefore also maintain a higher level of security then the small blockchain. You would still have the small, ineffective and obscure blockchain that you want, even though ironically you deny yourself that right by saying that people should not have this freedom of choice over the protocol they adopt. People are not machines, people do have the free choice of protocol and no matter how much you theorise or think that they should not have this freedom, you can not change it. Bitcoin has brought this volunteerism out into the world, it might be a pandoras box but for better or worse nobody can stop it.