...
Don't forget to wear your tin hat whilst doing it!
Um, no. Assuming valuable files can be stolen from your computer is being street smart, not paranoid. Computers being "pwned" isn't something that "could happen", rather it "does happen" and "is happening right now".
The most popular piece of malware I've seen (fake antiviruses) routinely scape your computer (and e-mail them to the hacker) for saved eBay passwords, PayPal passwords, e-gold passwords (when that was around), the very second the malware makes it onto your machine, and Bitcoin wallets will be the very next thing on the list if they're not already.
Oh, hey, just so you know what happens when your wallet gets stolen by a hacker... your Bitcoin client will see the theft transaction in the block chain and will automatically display it for your convenience as the latest sent transaction in your wallet, in the form "To: (thief's Bitcoin address)", right under your new balance of 0.00... it will look exactly the same as if you sent the transaction yourself from your own computer.
If and when you see that, your coins are permanently gone. Watch for that in your Bitcoin client, next time you visit a website that suddenly causes popups that your computer is "infected with 37 trojans, scan for viruses now", which I'm certain you've seen a time or two yourself.