Difficulty adjusts to the processing power available on the network. As long as so much as one person was mining blocks would still continue to be found regularly. The only problem with bitcoin's current setup is if a significant percentage of miners suddenly stopped mining (perhaps a very large scale disaster), difficulty only retargets after a set number of blocks. If half the people stopped mining, blocks would average 20 minutes until the next retarget. This effect gets worse with the more miners that suddenly drop off.
This is a problem that a lot of the derpcoin-of-the-week creators have had to deal with. Auto profit-switching pools would jump on their coin when difficulty drops, triple the difficulty and then leave until it comes back down again, leaving the remaining miners with slow block times and decreased mining income.