Well, you are describing it perfectly. I can have better security with my hardware wallet without needing any special security measures. I could even use it on "public" computer.
I say "better security" because no matter how many layers of encryption you may be using... at some time you would need your computer to access the wallet in "plaintext" to sign the transactions and that opens an attack vector.
It's much better to have the signing be done inside a specialized hardware....
Plus it is amazingly more convenient to use.
I solved this problem without having to spend money on a hardware wallet by
1- removing SIM card and factory-resetting my old phone, immediately putting it into airplane mode (wi-fi, bluetooth and everything else also manually disabled) and ENCRYPTING WITH STRONG PASSWORD (not draw pattern);
2- installing Electrum via APK stored on micro-SD;
3- generating new wallet in this air-gapped phone, then I imported x-pub to everyday brand new phone.
Now I can see balance, receive bitcoin and create unsigned transactions on the connected phone, then old phone signs transactions via QR-code.
Cheap and easy.
Ok, not bad, you basically have a DIY hardware wallet. Still not as much secure (physical access to phone and chip-off technique among others) and certainly not as convenient to use as a commercial hardware wallet.... But yes, its ok.
Hardware wallets are now less than $100 anyways.... That's also cheap, and easier.