I ran into some of these questions with the system I am building.
One tentative solution my coworker and I have come up with is to keep the wallet on a separate server than the one that is user-facing (i.e. the website). I will call the web server WS and the separate machine that is interfacing with the Bitcoin network the transaction server, or TS.
In our proposed architecture the WS does not connect directly to the TS. This way, if a hacker gets into your WS, they do not have access to your wallet, or even know where your wallet is running. The TS does not even need to be accessible from the internet (does not need a public IP).
Instead, all communication between the WS and the TS happens via a message queue. The message broker can run on the same machine as the WS or on a third server. Anytime the WS requires information from the wallet (such as funds received at an address) it pushes a message on the queue and waits for a reply.
The TS listens to the message queue and processes any requests. Sensitive requests, such as sending funds to an external address, can go through a separate authorization process. For example, if users want to withdraw funds from their account, a request is pushed to the queue, but the TS does not immediately process it. Instead, it waits for a human administrator to approve the request. The human can either contact the user (via phone, email, etc.) to authenticate the request, or decide based on other criteria if the transfer should be authorized. In the future we may automate this (for example by introducing a two-factor authentication process) but for now we want to keep it simple.
Apart from securing your wallet you will have other issues to deal with. There are not many Bitcoin clients designed to work in a server environment and to deal with concerns such as disaster recovery, load balancing, etc. The only one I’ve come across is
https://bitsofproof.com, but I don’t know how good it is.
Some of the issues to think about:
- How to reliably and securely back up your wallet
- What happens to your system if you are servicing a large number of users simultaneously
- How to keep track of the balance in each user’s account
The latter is of particular concern for me. See this thread for more info:
https://bitcointalksearch.org/topic/bitcount-enterprise-grade-account-management-for-bitcoin-586013