In order words, would it be safe to say that no wallet is 100% safe, apart from the hardware wallet? Since the hardware wallets are equally multiple coin wallets, what are their downsides?
I wouldn't consider a hardware wallet completely safe. For example, Ledger device pairing practices are less secure than credit cards. You are also highly vulnerable to theft via physical access -- encrypted wallets on unmarked USBs or PCs are safer for this reason.
I think they are great for day-to-day spending, but nobody should be plugging all of their private keys into an online computer, and that's what people are doing with hardware wallets. I think exploits may surface at some point that can overcome the key shielding mechanisms.