Exactly, and there are not so few blocks with only 1 tx (the coinbase), and regularly blocks with less than 100 tx.
The problem is that fees need to get much, much higher than 1 or 2 mBTC per tx to make it worth for miners.
Block size increase in itself is not going to increase the number of TPS because of the orphan risk.
To compete on a global scale with just Paypal, bitcoin would need on the order of 700 TPS, or 100 MB block size, at which point the risk of a block getting orphaned would be very very high.
IMHO the key of the scalability battle is not the block size, it's the fee structure and block propagation. There needs to be ways to minimize orphan block risks, maybe by allowing fork merging: a block could be allowed to have multiple next & previous blocks. This way if two blocks come at the same time, rather than one orphaning the other, the next block could accept both.
Currently if block B & C are mined based on block A, then only B or C will be accepted by block D being mined on either B or C.
The proposal would be that D could accept both B & C, and then half the B & C rewards would be burned by D (so the total reward is unchanged).
So the block chain becomes a block tree. Then processing power becomes the bottleneck because you have to validate blocks D,E,F, and G then 8 blocks for the next block reward, etc.