I agree on most points. Something cleverer is needed because the current formula is too blunt. But the scenario you described is actually the system working as intended. The CryptoNote developers didn't want blocks getting very large without genuine need for it because it permits a malicious attack. So miners out of self-interest would deliberately restrict the size, forcing the network to operate at the edge of the penalty-free size limit but not exceed it. The maximum block size is a moving average so over time it would grow to accommodate organic volume increase and the issue goes away. This system is most broken when volume suddenly spikes.
It isn't really broken in the case of spikes, it means you just have temporary congestion delays. You can prioritize your transaction by paying a higher fee or you can accept waiting for some available space (or even waiting until the block size grows). This part is much like bitcoin.
