This mean you can access your bitcoin thru your computer directly with the use of software like bitcoin core wallet, or other light wallet like electrum, multibitHD etc. And you can have the wallet that contain the private key and public key to that address in the blockchain. You can also back up those access keys and be able to transfer the control from one computer to another computer.
So logically you cannot store your bitcoin in your computer but can gain access and control to it thru desktop applications.
The private keys are mathematically related to all Bitcoin addresses generated for the wallet.
https://en.bitcoin.it/wiki/Private_key
Yes exactly. But most of the answers here show that most of them are clueless about how Bitcoin works or maybe they do not care. Read the answers right after my post. None of them care. They just say yes and tell you to download a piece of software so that you can "hold" Bitcoins in your computer.