Yes - but the adjustments don't happen immediately - and especially down adjustments will happen slowly.
This is only an issue if a large amount of mining power is switched off at the same time. If the mining power is halved at the start of a new difficulty-cycle, it'll take 4 weeks for a downward adjustment, which is long, but still reasonable. And an instant 50% reduction of mining power is absurd outside of catastrophic incidents.
Like I said before, existing miners won't be turned off anytime soon. And when they are finally turned off, old, less efficient miners, which will have a lower hashrate are turned off first, making any potential difficulty drop rather smooth.
Scrypt alt-coins are far more susceptible, since they're at the mercy of profit-switchers moving large amounts of hashing power from one coin to the other. SHA-256 alts can be merge-mined with BTC if they have been setup for it, so they're more protected.