If you say 'oh then have a limit', then you're back to capping it.
Transaction fees are needed to keep the network secure and provide an incentive against 51% attacking the network. The block size cap protects Bitcoin.
Is 1MB is the right limit? That's another question.
Maybe we agree. To clarify, there are a couple of different questions about block sizes.
Q1) How big would it be helpful for blocks to get? Opinions being:
a) 1MB is right for a long time, for reasons to do with mining on Tor or something.
b) Ideally smaller than the amount of non-spam transactions people want to make, to raise transaction fees by creating artificial scarcity.
c) Ideally bigger than the amount of non-spam transactions people want to make, to make Bitcoin as attractive as possible and create lots of transactions to get fees from, which should in turn raise the value of the block reward.
I take the OP to be talking about (a), when it becomes a case of (b). In other words, say loads of people want to make transactions, but the block size is stuck at 1MB. Long-term does that mean more fees for miners (higher per-transaction fees!) or less fees for miners (fewer transactions, less on-chain activity, possibility that everyone switches to another coin!). What's happened in the last two years doesn't really tell us anything about that, because we're at (a) and (c), not yet (a) and (b).
What hasn't really been discussed on this thread (which is fine, because it's been discussed on plenty of other threads...) is:
Q2) Who should decide how big blocks should go?
a) The community, but in a way that's weighted to Gavin / Developers / Miners, by the current model where the core developers make a proposal, stick new defaults in the client, and it gets adopted if the miners will run it.
b) The community, by some kind of voting mechanism - for example, some people were proposing a plan involving proof-of-stake voting with transactions, to reduce the freedom of the developers and miners to make changes. (The plan as stated also seemed to be designed to prevent any changes actually happening in practice, because it counted the "didn't vote" of people who don't follow Bitcoin politics as a vote for "no change"...)
c) Miners, by a dynamic consensus process: You can mine whatever you like and broadcast it, but if it's too big to handle the other miners will ignore it.
[Minor edits for clarity]