Hi, interesting project.
As Dzimbeck mentioned, I think transaction malleability is a serious problem.. one actor (perhaps a grumpy exchange pissed off that you are taking market shares) could cause some serious damage to any atomic cross chain transfer occurring on the blockchain with minimal costs. The new Bitcoin core makes it a bit difficult, but for someone motivated enough it is of no issue once they figure out what IPs the miners are listening on or they establish a working relationship with a miner. IMO, any project that relies on future tx contracts are a no go at this point and I think most of the core developers would tell you the same thing.
It also seems that order placement and matching happens on a centralized matching engine. It is somewhat misleading to label this project as a "decentralized exchange" when one of the primary component of an exchange is the order book and matching engine, although I do understand the need from a marketing viewpoint
I think users should be aware that DDOS to the server means no trading, and another worrying issues is that someone can easily pollute the order book with orders they don't intend to fill because obviously transactions don't happen until orders are matched (if there is no disincentive for a party to pull out from the trade, they will do so with impunity which means your order book is completely unreliable)
The transaction malleability vulnerability is an issue, but the cost of pulling off the attack is high enough that this early version of Mercury is OK as a proof-of-concept. The attack wouldn't be profitable unless people were making some really high-volume trades. And when OP_CHECKLOCKTIMEVERIFY gets adopted, the vulnerability will be completely fixed.
As for the centralized order matching, this part of the protocol does not require any trust, so it is still considered trustless. If the server gets DDoSed, traders can use other orderbook servers, or even find counterparties themselves through e.g. IRC and forums.
You are incorrect about there being no disincentive against backing out of a trade, this can be fixed with licensing. This is not implemented yet, but it would work to require traders to pay a one-time fee or security deposit when they begin using Mercury, and if they back out of trades (maybe if they get 3 strikes), their license will be revoked and they will lose their funds. If the fee is sufficiently large for the volume of trades they will be making, it will be unprofitable to cancel trades.