The keys are stored on an external device and never leave this device.
Every time the private key is needed to sign something like a transaction for example the transaction is sent to the HSM and the signed transaction comes back.
I suspect this kind of thing will become more popular in Bitcoin.
HSM's can generally also be used to encrypt and sign other messages like emails, etc.
They're not in mainstream use right now. Not yet anyway. This is going to change in 2016.
The point is the key is never disclosed to the computer so it never enters the systems RAM, ever.
There's a little linux USB computer named the 'USB Armory' which could be used to create something like this.
Two words: hardware wallet.
One word: Trezor
Yeah, these always sounded cool but something that can be programmed to what you want sounds a lot more useful.
I got one of the USB Armory devices late last year and it's pretty good.