Would it be obvious to a person if a couple of the characters in a private key were swapped?
No.
The problem is he get's an error message (I'm assuming he would either get an error message or the notification that there is no bitcoins to be swept) saying wrong private key or maybe he just gets a message saying there are no bitcoins.
Assuming your key is in Base58 Wallet Import Format, swapping two characters around will almost certainly generate an invalid key. The chance of the checksum still being valid after you change the address is 1 in 4.3 billion.
Could he at that point realizing I mixed up the private key figure out how to unscramble it?
He might realize you have scrambled the key in some way, but that will provide zero clues on how to unscramble it. All he can do is brute force a variety of combinations.
Would there be a way for them to determine that the key is good up through the first 12 characters?
No.
Is swapping just 2 characters around in a private key enough?
No. If someone was to try swapping every 2 character combination in a 52 digit WIF key, that is only in the region of 1300 possibilities. That could be brute forced in seconds.
Or should I swap 4 or 6?
Adding more swaps makes it harder, but still not hard enough to not be able to be brute forced relatively quickly. This is a very insecure method of storing your coins.
They wouldn't even know that it was almost correct would they?
Correct.
The private key exists in your notebook written down correctly so that if anybody is able to copy it they will be able to steal your bitcoin (assuming they get a copy of the address as well).
You do not need both the private key and the address to steal the coins. Having just the private key on its own is enough to control all the coins. The public address is derived from the private key, so importing the private key in to a wallet will automatically show you the relevant public address.
If you are this concerned about someone else being able to gain physical access to your private key, then storing it in plain text is not the method you should be choosing. Either store it somewhere secure that other people cannot access it, or look in to using an encrypted airgapped machine or hardware wallet.