If an unauthorized person gets access to your mnemonic recovery words backup, that person doesn't need your unlock PIN at all. That person doesn't even need your hardware wallet at all (if you didn't setup some weird derivation path or optional mnemonic passphrase for your wallet). You might even don't notice that someone unauthorized had access to your mnemonic recovery words until your wallet is suddenly drained.
Point of correction, I don't see anything wrong with Ledger wallet and they give users the access to their recovery seed, the only issue they have is closed source wallet, it would be great if this wallet is a open source wallet, maybe I would be using Ledger hardware wallet right now if they are open source.
I see quite some things wrong with Ledger crap (I'm surely biased). A Ledger hardware wallet shows you the mnemonic recovery words only when you setup the wallet where you have to confirm every single word as far as I remember. So you have to write it down at setup. That's actually good.
Later you can't reveal the mnemonic recovery words anymore. You can only check the words with a Ledger verification app that takes your recovery words which you have to input in the hardware wallet and verifies that the derived seed matches the internally stored seed of your wallet. Doing it this way is also rather good.
What's wrong with Ledger crap is: they lied about that the seed never can leave the device. They just implemented this with their subscription based Recovery service. So it's only a matter of firmware software to allow your precious seed to leave the device. This firmware is closed-source, suprise, surprise. Dooh!
Being closed source, you have no idea what kind of shit is in their software/firmware. You have to trust a company that lied to you and that messed up other stuff, too. Good luck with that.
And I don't believe any claims of Ledger to go open-source in the important parts. That's just their usual fog candles in any shit storm they earned and deserved.
Always make sure you keep your recovery seed, out of the public places, it's meant to be private, and always make sure that you go with a airgapped hardware wallet instead, the minimal I recommend is a Trezor hardware wallet.
Trezor e.g. is not air-gapped (did I miss something?). I have no issues with non-air-gapped hardware wallets as long as you can verify in the firmware that an USB connection doesn't allow a network connection to and from the hardware wallet and that via USB no keys can leak.