But the problem with a hardware key is that it can also be stolen with the mobile device.
Hardware wallets are not mobile phones, they contain the keys used to hold coins. Although, the most important is your seed phrase (+passphrase if included) which can be used to regenerate your keys and addresses. Hardware wallets are portable and can be carried about, but this is not advisable.
They can steal the suitcase, where you have your documents, smartphone, keys, and even the hardware key.
Have your hardware wallet in the best possible place you think it is safe (not inside safe or places that can be easily noticed by thieves). Like I implied above, the most important is your seed phrase (+passphrase if included).
One thing you have to be aware of, when you physically steal equipment, a good part of the security we have in them can be compromised.
All care is little.
When something like that happens, we should change passwords and pins as soon as possible, and if in accounts and applications where possible, log off remotely.
If your hardware wallet is compromised, best to send your coins to another address generated by another offline wallet. An offline wallet like paper wallet can be created immediately and send your coins to an address or addresses generated by the paper wallet and later buy another hardware wallet.
It is advisable to use passphrase with hardware like Trezor, because even if the seed phrase is revealed to the offline attackers, the attacker will not still able to compromise the wallet because different keys and addresses are generated due to the passphrase added to it. Adding passphrase is another protection, but it is needed along with seed phrase during recovery.