If a fee market breaks down because miners are not acting rationally,
it seems that that could happen with or without a blocksize limit, and
generally to the same degree.
Right. Miners being net econo-rational is probably an axiom for Bitcoin to work in the first place.
Story about imperfect models and not-perfectly-realistic assumptions....
A few years ago, I hired a new electronics technician (from Russia) to work at my non-bitcoin company. We needed to increase the bandwidth of a sensor circuit and I noticed him messing around with bunch of capacitors, soldering them on and off as though by trial and error. This seemed really strange to me so I showed him how to calculate the bandwidth based on the RC filters used and the GBWP of the amplifier. He told me that calculations like that "only works in theory" and that he preferred trial and error instead. Anyways, I showed him how to calculate the correct values for the resistors and capacitors and then we went to look at the results with the oscilloscope.
He was surprised at how close we were able to get to the behaviour he wanted without any trial and error at all.
But I was surprised out how far out the bandwidth was from my theoretical calculation, and I ended up making a slight "trial and error" change anyways! [I missed a couple of nuances about that circuit that I now understand]
The point of the story is that theoretical models only need to be useful--but we must understand that they are often not perfect. By ignoring theory, you'll spin your wheels and progress will be slow; but by overstating the validity of a theory, reality will kick you in the butt and show you what you don't know.
With the academic work regarding the transaction fee market, what we're doing is considering different lenses through which to view the problem, so that we can make incremental progress in our understanding. We
know that these models are imperfect, but we ask what happens if the assumptions hold, so that we can gain intuition about the more complicated real problem.
For example, if we assume that perfect competition exists AND that the marginal cost of block space is zero (the top left square above), then indeed fees go to zero and the Blockchain fills with spam. I believe Dr. Nicolas Houy was the first to study the problem through that lens in his 2014 paper
"The economics of Bitcoin transaction fees." Although the model was obviously simplistic, this paper revealed the interesting insight that a block size limit is economically-equivalant to a fixed-fee requirement.
If we assume that miners communicate a non-zero amount of information about the transactions in the block at block-solution time, but still assume perfect competition, then the marginal cost of block space is nonzero and a transaction fee market exists without a block size limit. I believe I was the first to formally analyze this problem in my
paper from August 2015 (the top right square in the table above). This is obviously simplistic as well because the market won't be perfectly competitive--miners might have some pricing power.
If we assume monopoly conditions (the two bottom squares in the table above), then the "mining cartel" will enforce a block size limit that maximizes the "producer surplus" as shown below.
Reality is somewhere in the middle of all these models. By doing research and asking ourselves what happens given certain simplifying assumption, we slowly build a deep understanding of the complex real phenomena.