I'm not sure, if I missed something, but why isn't block size limit defined dynamically based on previous usage (plus some safety margin)?
Is it impossible to implement a self-regulating block size limit mechanism similar to the way difficulty is adjusted, which allows the block size limit to be increased and decreased based on "demand"?
I'm not aware of more recent discussions but I found the first three pages of
this Feb 2013 thread good food for thought.
I've read the initial post of that thread several times and I think that its headline is a bit misleading. Essentially what Peter Todd is saying is that an large blocksize limit in general encourages the miners to drive out low-bandwidth competition. He is actually opposing Gavin's plan as well:
I primarily want to keep the limit fixed so we don't have a perverse incentive. Ensuring that everyone can audit the network properly is secondarily.
If there was consensus to, say, raise the limit to 100MiB that's something I could be convinced of. But only if raising the limit is not something that happens automatically under miner control, nor if the limit is going to just be raised year after year.
According to Peter Todd it is essential that miners
do not control the blocksize limit. He argues based on the assumption of an rolling average mechanism that takes its data from the previous observed block sizes. But that's not an argument against a dynamic block size limit (increase/decrease) in general. The point is, that the dynamic block size limit should not be able to be (substantially) influenced by miners, but instead by the transacting parties. So if it would be possible to determine the dynamic block size limit based on the
number of transactions multiplied by a fixed "reasonably large" size constant plus safety margin you would get rid of the problem.
Decentralization should be preserved by all means possible, because it is the very core that ensures the safety and thereby the value of Bitcoin.
There's risk in everything and nothing is absolute. This attitude would yield the obvious answer: "Don't ever raise the block limit at all".
I'd better say: "Only raise block size limit if required by the minimum amount necessary."