Only problem can happen if some hardware wallets are using specific derivation paths that is especially important if you are using bunch of altcoins.
For example, you can try recovering seed words you used for many coins on ledger to hardware wallet from other manufacturer that are popping up like mushrooms, in that case nobody can guarantee that you will be able to recover everything.
Walletsrecovery website is trying to collect all Bitcoin supported paths and BIP standards for wallets including hardware wallets:
https://walletsrecovery.org/