If only the exchanges changed to use XT (or whatever other client that implements a fork), then they would be losing business because few other users are also using XT so they don't make any money. If the users and exchanges were switched to XT and pushing for a for, then the miners would follow because that is where the money is. If miners and exchanges switch but users don't, then those miners and exchanges are incentivised to switch back because there is no money to be made if no one else is using that fork. In all of those cases, it is not possible for one group to coerce everyone else to switch, there has to be two out of three to do so otherwise there is no incentive for the other groups to switch.
Usually large mining pools have close relationship to exchanges, since they have plenty of liquidity to provide, so I guess they are more or less aligned. Average user however do not really have a choice in where bitcoin is going. So I think the concern of "the group of 12" manipulating bitcoin is real
Users are attracted by bitcoin's several properties:
1. Honest money creation and limited money supply, so its value will always go up long term against fiat money
2. Secure and censorship resistant, a safe medium for storage and transfer of value
3. Fast world wide transaction
When they face a change enforced by miners and exchanges, they have to decide the priority and evaluate what kind of feature they can sacrifice without totally giving up on bitcoin
Statistics showing that majority of people come to bitcoin because of 1 and 2. So any change impacting these property will heavily affect user adoption and drive new users away. 3 are less important. Why no one is too worried about 50% attack? Because 50% attack only affect the payment function of the system, coins in cold storage are not affected. Even chinese government took over all the chinese mining pools, they can't rob other people's coin
Currently there is a trend of advertising bitcoin as a fast and cheap payment method. However, if the new users you gained are just interested in fast and cheap transaction, then you will not have customer loyalty. Once more and more free and instant payment services appear, these customer will easily move away. Besides, this advertisement trend feels like finding innocent new users and dump the coins to them, this is not going to last long and hurt public image of bitcoin
Also, in your example, with such a fork, there is nothing that distinguishes a Core transaction from an XT transaction, so those XT holders who are sending core coins to that exchange are also sending their XT coins to that exchange. What you described is simply not possible.
After the fork the blocks on XT network will not be known to core network, vice versa, just like litecoin network is not aware of any bitcoin transactions since you are running different software
You brought a very interesting point: If XT somehow can destroy the corresponding coin when a pre-fork coin is spent on core chain, then we can eliminate this kind of dangerous attack. But nothing can enforce XT to do this, I guess that is not in the interest of XT, since they might want to destroy the competitor chain to make sure their success