I don't know, maybe if an official scammer put red or green trust on you it would maybe be removed after it has been publicly proven as a scammer, but else it's only a question of trust between members. If the guy has no link with you it means someone he trusts says he has been scammed by you. You'll have to convince (and probably prove) him that it's not the case.
As has been stated: Trust is not moderated. Either work it out with the person/s that left it or learn to deal with it.
If you are taking reputation/trust building loans, you will garner more red. I would advise anyone that does lend to you leave at best a Neutral.