We assume that this is a fact, although we can never claim it with 100% certainty, especially not for those hardware wallets that are not completely open source.
As far as I know there is currently no physical attack that could be successfully performed on Ledger (S, X, S Plus) although they have existed in the past - but that doesn't mean that there isn't one that hasn't been made public, or that there isn't one which has not yet been discovered. Trezor, on the other hand, has a vulnerability that cannot be repaired with a firmware upgrade, and the seed can be extracted relatively easily and cheaply if someone comes into possession of your hardware wallet, and you have not protected it with a passphrase.
The only additional security that the user can additionally set is the passphrase mentioned above, and there is no other way (that I know of) to encrypt the hardware wallet. The only thing you can do is store it safely away from the reach of anyone with bad intentions.