caveden I think you misunderstood I wasn't talking about miners but full nodes.
Indeed I thought you were talking about miners. I don't view a smaller number of full nodes as a problem. A single honest full node is enough to spot any attempt of fraud. There'll always be more multiple honest full nodes running.
If eventually the network handles 100,000 tps but the demands on a full node are so high that only the largest of the largest of companies operating in massive datacenters can run a full node then you might as well call those "full nodes" .... banks.
Come on. You know you'll never need a "largest of the largest company operating in massive datacenters" to run a full node.
100k tps? What's that? Visa apparently operates at 4tps according to the scalability page. How could Bitcoin possibly outgrow visa by 25 times in any reasonable time?
The world has ~7 billion people. A day has 86,4k seconds. With 100k tps, that'd be 8,64 billion transactions per day if I'm not mistaken. It's more than 1 transaction per person in the world per day (even if we consider population growth, it'd still be close to one transaction per person per day at least). I mean... Bitcoin would already be the king of currencies if it ever reach such amounts. It'd be something so freaking important in the world, that you can rest assured there would be more than enough honest nodes running. Remember: one single honest node can spot fraud (like breaking the 21M limit for ex.). And also, by that time, we can't even imagine how powerful hardware will be.
The idea that miners would soft fork blocks which are "too big" is equally disturbing.
Why disturbing? Btw, that's not something that can be prevented by the protocol.
First of all it is unlikely it would work. Nobody is going to make a block larger than what all but the tiniest fraction of miners says it "too big". If 10% of miners say they will soft fork blocks over 5MB only an idiot miner would make a block larger than 5MB. You are taking a 10% orphan chance by making the block even a few bytes over the limit.
Yes, the blocksize would only be increased if there's a large consensus, or if the monetary incentives to take more transactions outcome the risks/losses from eventual "orphanage". In this last case, that's supply adapting to strong demand.
See, market coordination.
Another element is that Bitcoin works on the concept (at least in theory) that miners are independent actors. If various miners will soft fork at various levels you have created a disincentive to remain independent. You NEED to know what all your peer miners are doing. We shouldn't be building a system where independent miners (the desired state) is at a disadvantage to miners in coordination.
It depends how you see it. Today there's a protocol rule imposing a 1Mb rule. That's 100% coordination, everybody follows the same rule. By dropping it, you allow a bit more of "independence". Some miners may try a limit slightly larger.
It's unavoidable: there must some planning concerning what's a "too big block", because otherwise the network is vulnerable to flooding. I personally prefer the "planning by the many", and that's miners coordinating what's reasonable and what's not. They will tend to adapt to actual demand. Attempts of "central planning" (elaborating some formula etc) might not produce ideal results, and, as noted yourself, are difficult and error-prone.
Lastly if various miners do have differing soft for levels than an attacker can exploit that to degrade the effective hashrate of the network. This would open the network to a 51% style attack with less than 51% of hashpower. What matters is effective haspower (i.e. hashpower applied to the longest chain). If miner X has a soft fork level of x MB and miner y has a soft fork level of y then by planting various blocks of differing size the attacker could fragment the good miners into working on competing chains. Anytime miners are on multiple chains the network is only as strong as the hashpower on the chain with the most hashpower. Note miner in this case is the entity (pool for most hashers) that is making the strategic decision of what chain to extend and what tx to include. Hash processors aren't truly miners in that they are already following an authority (the pool server).
If I understand it correctly you're talking about a race attack with the intention to double-spend some payments, not a total overtake. Because an overtake wouldn't work out this way in the long run. It might give some higher chances of a race attack, but I doubt it'd be anything meaningful. By the way, I'm not sure I get how this attack would take place...
Say, I generate a block that I know somehow it will hit the soft limits of 50% of the network (in hashpower). That means I break the network, half of it is mining on top of my block, the other on the previous. Now if I have >25% of the network's hashpower (a lot!) I can outpace any of the competing chains. But the thing is that, whatever chain I choose not to mine on will likely outpace the other. Even if it's the chain with the large block, eventually it will be accepted by those who initially refused it. It would still be very difficult to get any double-spend on anything with a meaningful number of confirmations. High value transactions always should always wait for significant amount of confirmations. And double-spending small transactions is not worth the investment.