It's the exchange you need to worry about.
Precisely.
I wouldn't care if a business I'm paying in bitcoin found out if the bitcoin came from Coinbase or Binance. There is essentially nothing they can do with that information; they can't link my output from the exchange to my input transactions.
I would care a whole lot more about an exchange I have potentially completed KYC procedures on knowing where I am spending my bitcoin.* Half the appeal of bitcoin is to move away from having some faceless central authority keeping a record of everything I buy, how much I spend, when I spend it, where I spend it, how I spend it, and so forth. You can bet your bottom
dollar satoshi that exchanges are tracking where and to whom you are sending any coins that leave your account. Much like with Amazon, Google, Facebook, etc., several big exchanges have already admitted to selling client data to third parties. Hence the appeal of mixers.
*I'm not doing anything illegal with my bitcoin, nor have I completed KYC on any exchange, but speaking hypothetically.