I can imagine how in the future criminals would be mugging Bitcoin users and robbing stores that accept Bitcoin, because it would be better for them than bank, since Bitcoin will always have some degree of anonymity and will always be irreversible, unlike banks.
Most merchants aren't dealing with Bitcoin anyway. All that happens when people use Bitcoin to pay for goods is that services such as BitPay convert it to fiat and have it sent to the merchant's bank account.
Even in case merchants do accept Bitcoin directly, they can choose to store their coins elsewhere. No one will ever find out where the merchant in question has set up its wallets.
The only entities being in some form of a dangerous position are the users. If whatever ill minded person sees how you pay with Bitcoin in a store, he can follow you and demand your coins in a not so friendly way.