There's multiple possibilities if a super majority of miners continues backing Segwit2x and moves forward with the hard fork. Many supporters of legacy BTC have maintained that difficulty will eventually readjust on the legacy chain, even if the hash rate drops significantly. That's not necessarily true. It's entirely possible, if the vast majority of miners switch to Segwit2x, that difficulty will never readjust on the legacy chain.
In that case, those who do not wish to migrate to Segwit2x may be forced to switch to a hard fork of their own-- one which either has an emergency difficulty drop to get the network stable again, or a POW algorithm change.
That's the worst case scenario: a dead legacy chain and two altcoins.......
Remember, miners are heavily invested stakeholders. They have entrenched interests, so they will flex their muscles if different protocol changes will affect their prospective incomes.
However, they are also heavily invested in Bitcoin's success. I don't think many miners (if any) mine simply to dump for current market prices. They only sell to cover overhead and otherwise are investing for the long term.
So, miners don't want a split. The scenario above is a worst-case scenario. But it's also one that some Bitcoin developers (like Luke Dash Jr.) are already prepare for, code-wise. The problem is that no one can predict future demand for either chain. So if miners charge forward with a hard fork with bad information about user demand, they could end up causing a messy network split.