There is the assumption that anyone seeking privacy with coinjoin and mixers are criminals and drug dealers... maybe throw in terrorists, blackmailers and kidnappers. It's a wrong assumption, and seeking privacy does not amount to criminality... But you're really not eliminating that risk of mixing coins with undesirable activity if you also use, say exchanges, casinos or trade P2P.
I've done all of that so would be really surprised if none of my coins are thus tainted.
Besides, just because you don't mix don't prevent anyone from dusting you with dirty satoshi.
But think about it, tumblers and coinjoins are not cheap, and who would be willing to pay for them to keep their privacy?
I believe an ordinary user would be digging himself into a bigger hole if he mixes his coins with what usually would be the criminals' coins.
Unless you mined your coins yourself you likely already hold coins that are tainted in one way or another. Merely holding a couple of sats that are linked to illicit businesses won't implicate you in any crime unless law enforcement has additional reason to believe in your involvement. Same goes for using TOR, prepaid phones and other means of anonymization. Using these tools do not make you a criminal and do not build a case against you (unless you're involved in an illicit business to begin with, obviously).
In principal I get why tumblers have a bit of a bad rep, but in general I guess you could also see them as a bit of a workaround to "fix" Bitcoin's transparency. Bank account balances are not publicly accessible either, so why should Bitcoin balances be?
While coinjoins and tumblers are more work and require higher fees than regular Bitcoin transactions I wouldn't call them expensive either. Most Bitcoin ATMs and Localbitcoin traders charge higher fees than what coinjoins and tumblers entail.
That being said I fully agree that only few people are willing to pay for keeping their privacy. Heck, most people don't even bother rejecting 3rd party cookies when being presented with a GDPR prompt. That shouldn't prevent anyone from making privacy tools easily accessible though.