I don't think that's exactly right. The lite clients need only check confirmations, miners validate transactions and only apply them to the blockchain if they make sense. Other miners only build on those blocks if they make sense so the network has protection from dodgy miners.
You need only trust the confirmation count.
Like I said good way to get robbed. How do you know the block you got from the network isn't malicious? You shouldn't be trusting any miner. The proof of work ensures miners can cheat among themselves however the client "verifies" all the work they do. If you aren't going to be using the blockchain that is fine.
You simply need to rely on someone else (blockchain server, e-wallet, etc) to verify the blockchain for you. However the entity doing the validation a
should be trusted and not some random node on the network there is no way to trust information you get from the network without validating it. You should assume that anything you receive on the network IS MALICOUS until verified that it isn't.
Assumming blocks are valid just because they are signed is a recipe for getting robbed. The network works under a "trust nobody" model. There isn't one piece of data your node gets from other nodes (who may not be trustworthy) that isn't validated. When your node is sent a transaction it verifies the transaction is properly signed and valid (by rules of the network). When your node gets a block it not only validated the signature (by hashing the header used by the miner and ensuring the hash matches and is below the target) it also verifies each transaction in the block and then reconstructs the merkle tree to ensure the merkle tree for the block is valid. Only once all transactions are known to be valid, and the merkle tree is valid, and the block is valid, and the hash for the block is valid does the client consider that block confirmed and increment the confirmation counters.
There is no shortcut. All that works needs to be completed in order to validate transactions. This doesn't mean YOU need to have the blockchain but you will need to rely on someone else who does have the blockchain. For example if you use Electrum client you are relying on the Electrum server to do the validation work for you. If you use the strongcoin ewallet you are relying on strongcoin to do the validation for you.
Simply trusting blocks as they appear on the network is completely insecure.