Not receiving a confirmation within ~20 minutes or so can be embarrassing for an in-person payment, and a 30+ wait can actually be costly -- for one party or both.
I still think that we all think about this wrong.
We are waiting for confirmations to ensure a transaction is irreversible, right? The only payment method which is instantly irreversible is cash. The most common - credit cards - can be reversed for between 90 and 180 days after the payment is made. Similar with debit cards, checks, PayPal, bank transfers, etc. All the payee has to do is phone their bank or payment provider and dispute the payment, claim they were hacked/scammed/phished/whatever, claim the goods or services were not provided as promised, or a dozen other reasons, and they payment can be reversed.
A credit card transaction you see instantly, the money is available for the merchant to spend in 3 to 5 working days, and can be reversed for up to 180 days.
A bitcoin transaction you see instantly, the money is available for the merchant to spend instantly, and can be reversed (with much more difficulty) for up to an hour, maybe.
Now, obviously if you are transacting significant amounts of money then I would never dream of accepting zero confirmations, but if you are making a small payment (cup of coffee, fast food, etc.), then I think it is entirely reasonable for the merchant to accept a zero confirmation transaction, provided it pays a reasonable fee, is not flagged as RBF, and has no unconfirmed parents.
As a result, the vast majority of the security for unconfirmed transactions comes not from within the Bitcoin system, but from external factors such as the large numbers of honest Bitcoin users who would never attempt to defraud their vendors, the tolerance among vendors for small amounts of fraud, the ability (or threat) of vendors resorting to the legal system or other types of recourse, and other factors which have nothing to do with the design of the Bitcoin protocol.
All of these things hold equally true for opt-in RBF (and they’re not unlike the situation with credit card payments in the US, which are easily reversable for months after the exchange and yet which have fraud rates low enough that almost all significant merchants accept them).