It doesn't mean when you're use third party, you will be safe.
Middleman is always be a trusted, reputable and experienced, but if the middleman act unfair because the scammer bribed him, then what you can do? or maybe you have an evidence where you can prove if you're legit, but the scammer create a fake evidence in order to convince the middleman, if the middleman make a mistake by trusting the scammer, what you can do?
You made a valid point, because if the middleman is compromised then it becomes a double scam, where the victim becomes the overall loser. Therefore, one should not loose guard but be very cautious, in any financial dealings whether directly or using a middleman. This is why if someone must use middleman or exchanges as the case may be in bitcoin, to do any transaction, it is better to use the ones that have good track record. That is why reputable and trusted exchanges like Binance, who are very innovative, will stand the test of time, because trust is the key word in any financial matters.