I like to think of block space as a precious item that the miners want to sell for as much money for as they can get. E.g.: prime water-front real estate. Thus, high bids get high priority, and vice versa. And expensive requests (transactions with multiple I/O) also cost more. However, it's also a race against time because some other miner might also discover a block. Therefore only the bids that have already queued up are considered. If there's any unsold space at the end of the auction, some of the remaining space might be given away for free -- depending on the miner's politics. E.g.: one reason to reject free transactions could be to encourage users to "bid higher next time".
Let's look at 2 alternate scenarios:
1)
Block space is extremely abundant and there's always enough for everyone. Why pay? Even if 100 miners keep rejecting transaction requests to encourage users to cough-up higher bids, the 101st miner might sign all the 'free-loaders' regardless. Since the network difficulty must match the "resource reward" (made up of new coins and winning bids), low rewards would tend to force the difficulty down.
2)
If block space is too scarce, basically the opposite happens. Miners get rewarded with higher and higher bids... until Visa and Mastercard et al. start looking competitive again. Bitcoin doesn't operate in a vacuum so it should take its environment into account.
So how does one find the right level of scarcity to maintain a multi-dimensional balance of:
D1: high network security
D2: 'optimum'(?) exchange rate
D3: competitive transaction costs
??
By a fixed algorithm? E.g.: the Bitcoin protocol automatically adjusts block size as well as difficulty.
Or by a human-influenced market? E.g.: miners competitively pull block-size up or down.
I think both options are likely to have flaws, and it seems like an extremely difficult problem to model. Since Bitcoin will likely always have external influences (competing currencies and payment systems), and those externalities may behave irrationally, I'm leaning towards allowing a human-controlled evolutionary algorithm for finding that balance.
As well as VISA and Mastercard, the miners face competition from Mt.Gox internal transfers (free and instant and a high proportion of bitcoin users have accounts), other block chains and potentially Ripple. I don't think miners will ever have the pricing power to force large transaction fees on users. With that being the case, income from transaction fees can only be increased by processing more of them.
To address another point, instead of wondering 'why pay for transactions if space on the block chain is abundant', you could also ask 'why pay for transactions if miners are processing them for free?' Each transaction is using valuable computing resources. Any miner that allows an unlimited number of fee-free transactions will be costing themselves money. It is just that up to now, the computing resources required for transactions have been small in comparison to those required for mining, so they have not been considered. As the costs involved in processing high transaction volumes becomes clearer, I believe miners will modify their own behaviour to compensate.
It is like the miners who were operating at a loss when the price was less than $5. They are pushing the difficulty up for all miners, while they are also losing money. Other miners had no choice but to deal with it until they dropped out, or the reward increased.