Most people, including me, are unable to audit the source code. I trust the people holding the bitcoin.org (ie. Gavin and other bitcoin devs) so I'm happy to use the binary (also with the fact that I believe some other people will try to compile and compare the binary). But how do I know the PGP public key is legit at the first place, if it is not linked to bitcoin.org in any way?
The bottom line, IMO, is to run bitcoin.org over https, and offer the binary hashes. The binary itself may be transmitted with http. A better way is to offer a torrent for the reference client (through https, of course).
Then please read about PGP and check this page: all devs PGP keys are listed there
Edit: I updated the PGP description link to the more specific one, sorry for initial ambiguity
I know how PGP works. But I don't personally know any of the devs so it is impossible for me to verify the authenticity of the keys, and I don't have a web-of-trust that would lead me to them.