You still have to "trust" the person who made the sworn statement. Do you trust an entity setup whose purpose is to defraud you by making false sworn statements?
You can't have a system interact with fiat without some degree of trust. That's just the nature of the game. You can, however, reduce the necessary trust down to something very basic, e.g. a sworn statement from a prestigious bank that has much more to lose from lying. If that doesn't satsify you, it should more than do to satisfy Lloyd's of London or some other insurance company which will happily insure those deposits against a bank theft.