2) Email server security did not get the proper attention
Since administrative email accounts can receive password reset links from Rackspace, a breach of our administrative email server is as good as root access to web servers with Rackspace. Our administrative email server should have been protected with the same tight standards we apply to our other servers, and access to this email account should have remained more limited.
Rant Mode On. For the rant adverse scroll on past.
NO. NO. FUCKING NO.
There should be NO method to reset admin passwords from outside the system. Period. No exceptions, no ifs, no ands, no buts.How many thefts will it take for this to sink in? 1 (well we already know that isn't true)? 2 (based on this "response" that seems unlikely)? 3? 10? 100?
If an admin loses access due to lost credentials then one of the remaining admins (who already has internal access) restores access in a secure out of band method.
1) Contact "locked out admin" via predetermined method (i.e. specific contact phone #)
2) Verify "locked out admin" via predetermined challenge & response.
3) Reset "locked out admin" password to one time password and require password change on next login.
4) While still in out of band communication verify "locked out admin" is able to login, change password, log out, and login again.
5) Note the loss of access in employee records. If admin can't retain secure access reduce his access.
If somehow your admins are so incompetent that all simultaneously lose access to the server well then fire them, wipe the server, restore from backups, and deal with the PR fallout. It would still be cheaper than handing $300K to an attacker.
NO EXTERNAL ACCESS TO SECRET DATA (PRIVATE KEYS). PERIOD. External access to admin accounts still leaves you vulnerable to multiple attack vectors:
a) Rackspace could make a mistake and give an attacker access
b) Rackspace could have a flaw in their console which allows attacker to gain access
c) Rackspace could have internal employee who compromises the system
d) The email could be captured in-route.
e) The email server likely has the same backdoor so compromising that server provides indirect access to the primary server.
How fraking hard is this concept:
1) YOUR OWN HARDWARE in a locked co-location cage. A good provide should allow custom procedures to limit physical access.
3) No external admin/root access by third parties. Period. Access to the servers is granted internally by employees who already have access.
4) Use 2 factor authentication. Humans are fallable. Expect they will fail. Using a second factor provides hardening when that failure inevitably happens.
If you fail those three it doesn't matter how "secure" your code is.
You build Fort Knox on a foundation of sand.I mean for fuck's sake guys.
You just lost $300K in the span of a couple months. Go to a quiet place, sit down and meditate on that. You just lost $300K to hackers
who used the ability to externally reset admin credentials in two separate attacks. Does it even seem logical that the "solution" to ensure it doesn't happen again is to CONTINUE TO ALLOW EXTERNAL RESETS OF ADMIN CREDENTIALS and "try harder"?
