It turns out that it's possible to prove that you have the private key for a particular address, without revealing the private key itself. Proving that you have the private key for an address to the bitcoin network lets you spend any bitcoins sent to that address (this also means that you must NOT share your private key with anyone else, otherwise they will be able to spend all of your bitcoins). It's a good idea to encrypt your private keys with a strong password and then backup your private keys to several locations to ensure that your bitcoins are safe, even if your computer breaks.