That's precisely what's going on currently, giving the dog a bad name so they hang it. Petty crimes have existed with humanity since and there's been different currencies and the eras they reigned yet it didn't stop criminality or these petty crimes so why lump all this responsibility in Bitcoin. It's just not fair to say the least.
Do you think that the reason why criminals and other people use bitcoin is that they can blame it and label it? Are you serious?
No one will be able to take down bitcoin, if the way of doing that is laebling it as a criminal currency. The criminals are doing that because it is efficient and easy for them because it has a feature a criminal can use, easy as that. Criminals don't care how they are using bitcoin or how people label it, they just care how the transaction goes and how bitcoin can make their identities safe.