I'd appreciate the help a lot! struggling with the choice and understanding right now
Likely source: https://raw.githubusercontent.com/EAWF/BTC-Toolbox/3938785f186c76598989cc0aa017ad351483d3b1/Images/KeyDerivationTechnicalOverview.png
A hardware wallet likely only needs to store the "random" Entropy Source and derivation path used at minimum from which all other stuff can be derived. It likely stores more like the Master Secret Key and Master Chain Code to avoid all computation steps again and again which wouldn't make much sense.
The recovery words (Recovery Phrase in above diagram) are likely not stored as they can easily and rather quickly be computed from the Entropy Source.
The optional mnemonic passphrase (your 13th or 25th optional passphrase in addition to the recovery words) shouldn't be stored on the device as it is only needed to derive the Master Secret Key and Master Chain Code from the BIP32 Root Key Derivation. If the optional mnemonic passphrase is empty ie. not used, the derivation steps still use as default 'mnemonic' as optional passphrase as an input to the 2048 rounds of PBKDF2 mill.
I'm pretty sure this Master Secret Key and Master Chain Code are stored and secured by a hardware wallet. If Ledger crap e.g. has a dedicated unlock pin for a wallet with optional mnemonic passphrase then, that unlock PIN secures that unique Master Secret Key and Master Chain Code of a derivation with that optional mnemonic passphrase. This adds some convenience but hides security of a complex optional mnemonic passphrase in addition to the security of the Entropy Source behind a short PIN (security-wise not such a big issue as you have very limited number of tries to enter such a unlock PIN). I would still recommend to have an unlock PIN at least eight digits long or using more complexity.
Don't take my words for granted as I haven't read a lot of the firmware source code of open source hardware wallets. I'm just interpreting what I would do if I were a developer. Do not choose a hardware wallet with closed source firmware, thus Ledger should be out of competition for various reasons.
Try to get behind the security model of other hardware wallet competitors and how they cope with attack vectors and securing the important secrets of your wallet.
To understand HD wallets better, I recommend working through pages at https://learnmeabitcoin.com/technical/keys/hd-wallets/.