-snip-
If none of these characters (*<>) are present, The seed is in the correct order and I can get the keys. Does anyone know what type of word mixing is done here?
The correct order relies on the "checksum" at the end of the bit-equivalent form of the mnemonic seed.
Here are the specs: github.com/bitcoin/bips/blob/master/bip-0039
But since we're talking about an existing mnemonic phrase here, just looking for combinations with a correct checksum is not enough
as it will likely produce valid but totally different mnemonic seed at approximate chance based from the checksum bit's size and the number of given words.
What BTCRecover does is to check a candidate valid combination if it will be able derive the given address/pubKey/xpub with the common or specified derivation paths.
If they do, BTCRecover says the words are in incorrect order.