We have off-site backups in a different DC. It's managed by Rackspace.
If the server crashed, we have no problems of recovering. There are a lot of backups of all our main servers. It's just that these backups were deleted by the hacker.
I meant data center of a different company (a different admin panel with a different password, and an append-only configuration). That's what provides a level of actual redudancy. Using a cloud service from one company protects you if a truck crashes into one of their data centers, but offers zero protection if someone gets your admin password!
I can't afford a scalable solution that gives me the same reliability for a bootstrapped startup.
Not anymore you can't!
You can make backups for cheap (backups aren't accessed, so they don't scale with the rest of the site).
All financial transactions will be handled by payment gateways who will be responsible for their own security. Apart from that, no money is involved so I just want to consider scalability, performance, availability and cost. For me, I think AWS's EC2 instance with Load Balancer handling SSL termination (can't be DDOS'd) + RDS with snapshots and binary logs recoverable to 5 minutes ago are more than enough for me.
Most people choose to outsource security, it's just that in Bitcoin world everything is DIY.
I'm not going to use your domain service or any other zhoutong "cloud" service because its clear that you don't have plan B (contingencies). What happens when someone deletes your AWS instances
and its snapshots and logs? Has the thought even crossed your mind to try and see if its possible on the AWS admin panel?
Its a truism that at the basic level, security can't be outsourced. You have to trust someone eventually (unfortunately for us, you happened to trust bitcoin consultancy), but catastrophes can only be averted by good planning.
I don't think I have to mention the common sense. Of course the database backups are going to be downloaded on a constant basis. Bitcoinica has accounting records, but they are not current enough to resume trading.
The requirement for currency isn't very important for a non-financial project if someone else (such as a domain registry) will keep the records anyway.
What I'm saying is that it's unreasonable to expect a mission-critical configuration for a project with limited financial interest involved. All possible efforts will be put into security for sure. The use of cloud services shouldn't be the argument here.
If I were to build another Bitcoin project, I will definitely consider the locked cages. But until I have the funds and the security expertise, I should probably stay away from money itself.
I recently got PCI Compliance for storing credit cards on the servers, and the certification has been recognized by a large business bank in Australia. AWS datacenters all have passed the physical security checks and required audits. But I still choose not to store any critical financial data at the moment. Instead I will use the vault services provided by reputable payment gateways.
For any of my commercial projects: Minute-interval binlogs and at least daily backups with weekly downloads. All firewalls configured properly. Two-factor auth on AWS accounts. No injection, XSS or CSRF possibilities.
No system is 100% secure. The uncompromised systems are the ones not being targeted. Outsourcing security to a responsible party will avoid the possibility of being a target as much as possible. We can already prove that Linode was not secure, but why only 8 accounts (all related to Bitcoin) were hacked? Bitcoin attracts cyber criminals and it's reasonable to expect a disproportionately frequent security attacks on Bitcoin-related projects.
However, I feel unfair for the reputation damage that wasn't even triggered by me. I'm always serious about security and even though I'm not a specialised security expert, I do have some knowledge and experience of maintaining a secure system that's enough for a SaaS project. You can criticise me for trusting 3rd parties too much, but it's still my belief that the so called 3rd parties have better security skills than me. It's just that they are being targeted. (Even Sony got hacked so many times.)