The value of bitcoin is essentially controlled by digital signatures stored in a distributed database called the blockchain.
Your wallet is a file that contain private keys.
Your wallet can be stored on:
1. a hosted service.
2. a bitcoin client on your computer
3. Cold storage which is either a physical device not connected to the internet or a "paper wallet" (essentially the private written or printed on a piece of paper.
Whereever your private keys are stored they each have a corresponding public key (starting with the number 1) which is the "Bitcoin Address" we send bitcoin to and from.
You can send bitcoin to any one of your public keys regardless of where the corresponding private key is stored.
To buy stuff you essentially send your bitcoin to the public key of the person or business you are buying from. In the back ground your bitcoin client essentially creates a digital signature that say you own these bitcoin and that you are sending them to to bitcoin address indicated. Your client then broadcasts this transaction to the network and once it is confirmed by 6 nodes on the network (to avoid double spending) you transaction is complete and the control of the value of those bitcoin have effectively been passed from your address to the address you sent to.
You can keep you bitcoin on Mt. Gox however Mt gox is essentially a "hosted wallet" which is probably the least secure type of wallet. Hosted companies can get hacked. They can go out of business. They can disappear. It is best you move larger amounts of bitcoin to a more secure wallet.
If you store btc on you local computer. 1. be sure to encrypt your wallet. REMEMBER YOUR PASSWORD. And if you EVER loose the wallet.dat file from a corrupt or overwritten hard drive or any other issue, YOU BITCOINS ARE GONE FOREVER.
When you download the offical bitcoin qt client to you local computer you have to also down load the very large blockchain (in a process called syncing). You can not see or spend any bitcoins you receive at an address stored on your local machine until you client is "synced". The same is true if you turn your computer is off. You client will have to catch up with the ever growing blockchain although after the initial download, it takes significantly less time to sync depending on how long you have been on line.
Here are other ways to secure a bitcoin wallet
https://bitcointalksearch.org/topic/howto-create-a-100-secure-wallet-17240blockchain.info/wallet is a very popular and secure hosted wallet.
Good luck