Hardware wallets don't execute arbitrary code. They have a defined communication interface that doesn't allow other actions.
The software running on the hardware wallet is also simple enough that one individual can know everything that is happening in the wallet. They can thus verify that there is nothing malicious in the code.
The Meltdown attack was possible because everyone fundamentally misunderstood how authorization worked for memory processes. The same can indeed happen to hardware wallets. Not the Meltdown attack specifically, mind you, but heretofore unknown exploits can and likely do exist for hardware wallets. Anyone who assumes otherwise is incredibly naive.
Generally I can't trust hardware wallets. Linux airgaped computer is the best, because you are never online, but the problem is crafting the raw transaction because the Core client has no other way to do it, I still have to practice that with testnet coins before I consider myself safe to do it with real money.
Same here. I wish that Core wallets were capable of signing transactions in an air-gapped (un-synced) environment. The next best thing is Electrum, which has various weaknesses.