they control nothing, other non evil miners would step in and start mining, and things would continue, the lack of the other miners would reduce difficulty.
what your missing and not understanding is that before asics we had this issue on a daily basis. miners had a small profit margin and as the profit went to 0 for some of them they simply stopped mining. the rest who still saw profit kept mining. even now there are differences in profit for different miners. even if some are forced out there will be others who arnt. as miners leave the profit for others will go up. it ALWAYS reaches a natural balance.
The situation was very different when mining was done with one guy with one GPU that he switched to mining whenever he was not playing games on the computer. They could switch on and off at whim.
Things are developing fast into the direction that only profitable mining is going to be done in some few places on planet where electricity is cheap and they have the most bleeding edge HW running on huge datacentres. Whenever that becomes unprofitable there is nobody to step up in near term to fill the gap so that bitcoin economy as a whole that needs the mining can continue uninterrupted. There is going to be huge drop in profitability at the moment when block rewards cease to exist. At that moment all the mining gear at individual homes not in the datacentres has been obsoleted long ago and have been collecting dust at some attic for along time.
If somebody closes datacentres they need to be cleared from the equipment so that the facilities can be rented for some other use and without revenue from block rewards there is not enough money to go around for anybody rebuild the datacentres back up. They do not sit idle waiting if mining will become profitable in the future. They need to run constantly to pay the rent or be dismantled.
Or probably it actually works so that some will quit and some will stay and in this case mining will be concentrated to much fewer hands and getting the 51% will be very easy buying unprofitable mining datacentres from bankruptcies and the one reason to buy them is getting 51% to some one persons hands.
I believe keeping the mining profitable without any significant drop is actually very much in the interest of bitcoin economy as a whole to prevent any huge datacentres ever becoming available to the market for a ridiculously low price as that would be a disaster much bigger than anything else.