Hi hamiltino,
There are a couple reasons that the average block interval can't be too small (and not a lot of reason to do so -- as you say, you get "more security per minute" from probabilistic attackers on average but mining is a high-variance process so this doesn't translate into any real-time guarantees).
One reason is that you need your blocktime to be much higher than the synchronization time of the network, so that all honest participants are working on the same blockchain tip. All the incentives in the world won't help if your actors physically can't reach consensus! If the blocktime is too small, miners will create new blocks long before seeing any conflicting blocks, and the result is that they will always see the "real history" to be their own, since they see it before other, longer chains have had a chance to be seen. The result is a totally broken consensus, which is what happened with some early "fast block" alts like Feathercoin. This can be mitigated to some extent by more efficient network communication, in particular eliminating redundant data. The core developer BlueMatt has done some stunning work in this direction for Bitcoin.
A weaker version of this problem is that miners always see their own blocks first, which means that in the seconds after a new block is found, there is always disagreement amongst miners about which block to mine on, as some miners know about the new block and others don't. The result is that conflicting blocks may be mined, through nobody's fault, and one is eventually thrown out. The dead block is called "stale" and the proportion of mined blocks which are stale is called the "stale rate". Smaller blocktimes mean a higher stale rate, which decreases the average hashrate on the honest chain and weakens network security.
If the stale rate is too high, this is also a centralization pressure, since a single actor will see his or her blocks immediately and can start mining on them, while others have to wait for them to arrive. In the case of a conflict, the more hashrate this actor has, the more likely she will "win" the conflict, that is, her block survives and the conflict(s) are staled. This means big actors effectively have a lower stale rate than small ones, giving them a disproportionate award.
Another reason, which doesn't really apply to PoW based on simple hash algorithms, but which might to something SNARK-based or which uses Cuckoo Cycle, is that in order for things to be progress-free you need each block interval to involve many, many attempts. (Bitcoin miners, in aggregate, go through something like 300tn nonces on average before finding each block.) This means that your block interval needs to be much longer than the amount of time typical hardware needs to do a PoW attempt. Otherwise you literally have a race where the person with the most hashpower always wins, which again is a centralization pressure.
Another reason is that more blocks means more validation work and more storage requirements for all participants in the network. This may discourage users from doing their own validation or from donating validation effort to the network, which weakens decentralization and network security.
