I think this has been discussed many times before. I thought a "block headers only" solution was in the works. In this case, a client only needs to download the block headers instead of all transactions. As new transactions are received, it would pull input transactions lazily out of the network to do verification. You need to take steps to ensure that there are still clients that maintain the full transaction history however.
So you receive an incoming transaction... How do you know if it's a double spend if all you have is block headers? If it has to ask the network about transactions, what happens if the network lies or the node being asked doesn't have a complete picture? The block headers allow you to prove that something
is on the block chain, but in this case, you are needing proof that something invalidating it
is not on the block chain to know it is any good. You can't do that without a complete copy, a trusted peer, or a trusted digest.
You don't have to trust any peers...transactions are identified by their hash...when you ask a peer for the transaction associated with a txid (i.e. you need to retrieve a transaction input), you validate what it returns by hashing it and verifying that it matches the txid. If that transaction is part of an earlier block, then to the extent you trust the block headers that you downloaded, you can trust that the input transaction was valid (you could make a rule that you only trust the headers that pre-date one of the hard coded blocks in the client...probably hundreds of other possibilities as well).
Creating a digest of current account balances is still a good idea though...in fact, it might be a good idea to include a full digest of all account balances with every block that is created (the block header would have a merkle hash that points to the digest and only addresses with a nonzero balance would be included). This way, no signatures or trust network is needed...the digest gets created as part of normal block creation. In fact, clients could choose not to verify the entire history, but simply start with the current block in the network and validate forward from there. They could even discard old transaction history this way (as long as you set a limit on how deep reorgs are allowed to happen). The validation might need to reach back into history prior to the earliest digest that the client is using, but it could pull those earlier transactions as needed from the network. I think you would just need to be verifying that input transactions did appear in an earlier block.
If these were done on an automated basis (e.g. every 1000 or 2016 blocks), then a client could feasibly only need the block headers from genesis to the last trusted digest. Miners would probably need to be forced not to "trust" the digest and would need to verify it in its entirety before building on it, until it's at least a certain age (e.g. 1000 blocks), and then clients would only treat a digest as trusted if it's more than 1000 blocks old. The digest is going to be huge (tens of MB or more) and can't reasonably be done every block of course, and the way I see it, is still probably a viable solution.
The digest for any block could be produced on demand by any node that has the full transaction history. If you did something like this, keep it simple...define a standard format for the digest and have miners always include the hash of the digest in every block. When a new block is found, other nodes would update their running digest with that block's transaction, then compute the hash of the digest and verify that it matches what is in the block.
Still, I'm struggling to think of a real benefit of such digests (it is redundant information after all)...using digests would require placing some trust in the information provided by peers (full validation of the transaction history and block chain is required to avoid trusting peers). It's possible this could provide some benefit to lighter weight peers (that don't have the storage, CPU or bandwidth needed for full validation), but then I think that problem is better solved by a lightweight client that just communicates with a full node that it trusts. In fact, as I write this, I'm wondering whether a "headers only" client fills an real need either.
Either you're trusting other nodes or your not...if you're not, then you need the full transaction history...I don't think you can escape that. If you are trusting other nodes, then just pull only the information you need from one or more trusted nodes when you need it and don't even try to connect directly to the p2p network.