It would seem that attempting to use multiple Ledger wallets for MultiSig is problematic if you're trying to set multiple hardware devices as a signer in the same wallet file. Also, it certainly seems crazy that you can't "save" the partially signed transaction, as what happens if your friend on the other side of the world is one of the signers for this multiSig and needs to use his Ledger to sign?
Wallet1: Ledger1 + Ledger2 + Ledger3 + Ledger4
Maybe if you use multiple wallet files, with 1 Ledger per wallet file and just the xpubs of the other hardware wallets as co-signer keys... You'd then have 4 wallets file created using 1 hardware device + 3 xpubs per file like this:
Wallet 1: Ledger1 + xpub2 + xpub3 + xpub4
Wallet 2: xpub1 + Ledger2 + xpub3 + xpub4
Wallet 3: xpub1 + xpub2 + Leger3 + xpub4
Wallet 4: xpub1 + xpub2 + xpub3 + Leger4
So when you want to send, you'd create a send in any given wallet... sign it with the appropriate Ledger, then pass that partially signed transaction to another wallet file to sign with the matching Ledger before broadcasting...
Slightly more complicated, but it might work around the issue?