I think the fatal flaw in my understanding was that I thought the xpubs were generated solely by the hardware devices and were independent of the derivation path, and that the derivation path specified the receive and change keys from the child xpubs of the devices.
So the derivation path is used both in the generation of the xprvs/xpubs, and also in the generation of individual addresses from those xprvs/xpubs. Allow me to explain.
Take the following familiar derivation path: m/84'/0'/0'.
m stands for your master private key, which is generated from your seed phrase. Technically speaking, this is the only key which should ever be called your master private key, but a lot of people and software also use the term master private key to refer to your account extended private key (which I'll come to in a second).
The next three numbers are, in order, your purpose, coin type, and account. 84 is the number assigned to P2WPKH segwit addresses. 0 is the coin type for mainnet bitcoin. The next 0 simply means the first account. As I explained previously, that derivation path tells your wallet software to generate the extended key at the 85th hardened index,* use that to generate the extended key at the 1st hardened index, and then use that to generate the extended key at the 1st hardened index. The extended key you end up with after doing all that is your account extended private key. Account because it is at the account level, extended because you have an extra part which allows you to continue to derive at further levels. But as I said above, lots of places incorrectly refer to this key as your master private key. When you extract an xprv from a wallet, it is almost always this key that you are extracting - your account extended private key.
Now, to get actual addresses from that key, we need to derive through a further two levels. The derivation path of the very first address in your wallet will look like this: m/84'/0'/0'/0/0. The first extra zero is change (0 for not being a change address, 1 if it is a change address), and then the second extra zero is in the index of the individual address itself.
m/84'/0'/0'/0/0 - 1st non-change address
m/84'/0'/0'/0/1 - 2nd non-change address
m/84'/0'/0'/1/0 - 1st change address
m/84'/0'/0'/1/3 - 4th change address
And so on.
Note that multi-sig sometimes add in an additional field called the script type, so instead of something like m/84'/0'/0' you would have m/48'/0'/0'/2'. (48 is the purpose number for such multi-sig paths, and 2 is the script type for P2WSH.)
So, in the case of HD wallets, you can import keys in two ways. The first is that you can either import a seed phrase, and tell the wallet the exact derivation path you used to get the account extended private key. So for example I could give you the following seed phrase:
tray expect pact quantum junior chronic nation topple boy today maid syrup
And tell you I used the derivation path m/84'/0'/0', meaning you can calculate the following xprv:
xprv9zScb44VdRkzpdV8djUnNzD69QWRCA3iMvMm29UaouknnXxH3KRrnpN9QAyzEbqgMSVUVUavpkiaWuBhTBpefXXX5kg4tUvSQpd2dDHKFFX
The wallet then knows to start there and add on the extra /0/0 derivations to reach individual addresses.
The second option is I can just give the wallet the xprv directly, and not tell it about the seed phrase or the derivation path I used to get to the given xprv. The wallet will take the xprv and add on the extra /0/0 as above, without knowing the derivation path used to reach that xprv.
Now let's say, for example, I'm setting up a 2-of-3 multi-sig. In this particular multi-sig, I'm using one seed phrase on my computer, and two hardware wallets. Let's say the seed phrase on my computer is using the derivation path m/48'/0'/0'/2'. Great. So my software knows to use this deviation path to calculate the first xprv from the seed phrase it has, and the corresponding xpub. Now my two hardware wallets come along. For the sake of argument, the first uses m/84'/0'/0', and the second uses uses m/0'. What a nightmare! Each hardware wallets generates an xprv at the given derivation path, calculates the corresponding xpub for the xprv, and then feeds the xpub to the wallet software on my computer. Now, my wallet software has three xpubs, but it may or may not have any idea the derivation paths used to reach the two xpubs from the hardware wallets. But actually, it doesn't need to know. All it needs to do is calculate a child key at /0/0 for each one and combine them in to a multi-sig address.
So it can do seed phrase + m/48'/0'/0'/2'/0/0 to generate the first key itself, and then it can do hardware-wallet-xpubs/0/0 to generate the second and third keys. One it has all three keys at /0/0, it combines them to create an address. It does the same thing at /0/1 and combines the three keys to generate the second address, and so on.
*Hardened indices, indicated by the ' symbol, are actually the number you see plus 2
31, but the reasons behind that are not relevant for this.