So, there seems to be two fairly obvious ways that the 2mb part of the implementation can fail. 1) fail to pass tests and 2) fail to achieve consensus ...
so we already got a lot of folks out there screaming that segwit2x is a kind of package, when the first part of segwit seems to be agreed upon; however, the second part of the deal (the 2mb aspect) seems to have contingencies. So, if it ends up that the second part does not go through, then then a large number of folks will be whining that Core broke the agreement, blah blah blah.. and the 2mb aspect was supposed to be "guaranteed", just like (and maybe even worse) they were mischaracterising and whining about the Hong Kong agreement.
Well assuming the segwit2x code is ready before mid July and the 80+% that are advertising NYA go ahead and adopt the code, then they will be actively dropping any blocks not signalling segwit at that time, and those running BIP141 nodes will automatically end up on the longest chain regardless since it will be >80% of the hashrate. Since all of the blocks on that branch must advertise segwit, it is guaranteed that segwit will activate after the two diff periods - one lock in period and then the one to activate it - since it will be 100% segwit signalling and >95% required by BIP141.
As for what happens after that, as you have correctly surmised, if core never adopts the 2MB hard fork code from segwit2x, then the bulk of the users using core's client will end up on a very low hashrate fork of bitcoin if the miners go ahead with their fork regardless. Claiming that "core broke the agreement" is likely on their part, but then - core never agreed to segwit2x in the first place! It is the segwit2x miners that agreed indirectly to activate segwit, not the other way around. Once segwit activates, there is also nothing preventing miners from going back to the core client as well and maintain the status quo, though the signed agreement is a contract of sorts.
What is likely to happen is after segwit gets activated, the pressure on transactions will no longer seem present; especially since there doesn't appear to truly be a transaction crisis at all since transaction spam suddenly and magically disappeared after the segwit2x agreement. If we reach a new deadlock there (and there's a good chance we will) then I suspect the transaction spam will magically appear again, larger than ever, and there will be cries that we desperately need the 2MB expansion as well. As I've said before, most of core
has said they'd support a block size increase but at a less frantic pace. I suspect the miners will maintain their pressure of their own hard fork claiming that core will backpedal on their agreement if they don't (as you said they claimed about the HK agreement - none of which actually happened.)