This protocol is definitely not made under the assumption that all parties are acting honestly, it would not be considered trustless if that were the case. Not going through with orders is much less critical than other types of attacks, since no funds would be stolen, the only damage is that funds will be locked up for a few hours before the refund unlocks.
However, this attack is completely solved by using security deposits: traders pay a one-time deposit or fee when they first begin using the trading platform to initialize their trading identity (which is pseudonymous). If the trader using that identity fails to go through with trades (they could possibly be given 3 strikes), the orderbook server will ban that identity, and the trader will have lost their security deposit. This mechanism makes it unprofitable to back out of trades.
Front running can be solved by including signatures along with orders that prove they are actually backed by unspent outputs, and by communicating over a p2p gossip network to ensure the orderbook is valid. These things will be worked on in the future, and will essentially make this model fully decentralized (but are more complicated, so Mercury's federated model should work fine until then). They will also require full node verification (or UTXO set queries), so they are not yet viable with SPV clients.
