You don't have to trust the other party because the trade wont proceed if there's hiccup or even if it's at the final stage, you can refund your coins.
Those addresses where you should send your funds are designed to have a timelock, refund/spend conditions and a "secret" (it's also created using your own keys);
those are configured to be recoverable in case of a failed trade & the "to-be-traded" coins can only be spent by both parties unless the secret was revealed to finalize the trade.
The fee wont always be an issue either, a 1input-2outputs tx @50+sat/byte is still a lot cheaper than any Exchange's withdrawal fee or Instant Exchanges' trading fee.
One potential drawback for the fee is: if the either one of the final transaction got stuck unconfirmed due to a really low fee, the occasional "mempool clogging",
and the timelock for the refund transaction was due, then the other party can broadcast that refund tx (if it's RBF) or send it to a 'friend' solo-miner/pool to double-spend the inputs of the unconfirmed claim transaction.