Author

Topic: OVH (host for Slush's pool and Bitcoin-Central.net) has been compromised (Read 67355 times)

legendary
Activity: 1204
Merit: 1015
This is the official update from ovh: http://forum.ovh.co.uk/showthread.php?t=6592

Quote
On April 26th, we detected an internal problem with the generation of the 21 characters. 2 out of the 3 random functions that we use in the code were not generating an authentic random sequence. It was possible to request a password change for a customer ID,
and then find the "unique" URL emailed to the customer by brute force. The problem was found by an internal developer on April 26th at 11:03:14 and it was fixed at 12:54:13. The cause of the problem was linked to the rand function used in this part of the code. It was not patched to the same extent as the rest of the code at the time of activating the script execution cache. We have replaced the old function of 3 sequences to generate 21 characters with 2 authentic random functions to generate 64 characters.

We then ran searches on our databases to verify whether the loophole had been exploited and if so, when. We tracked the log of password changes for your IDs for the last 3 years. We actually have authorisation from the CNIL (the French data protection authority) to archive and exploit all our back office logs for the last 10 years, precisely for this type of situation.

We detected three password changes carried out by brute force on 3 customers IDs with active services. These 3 cases involved an attack aimed at the "bitcoin" community that uses OVH services. The hacker seems to have found the loophole on April 23rd at
22:00 and ran a significant number of tests to develop their tactics over a period of 1 hour. At 23:00 it had been perfected and the 1st ID was hacked, followed by the other 2 the next day (all from the "bitcoin community"). We have been in contact with these customers but the quality of the exchanges prevented us from obtaining sufficient information to identify this loophole. Thanks to our internal developers, we have fixed the problem in a totally independent manner. Only then did we begin to make the connection between the loophole that we had just fixed annd these 3 customers. We have certainly learnt a lesson on how to communicate with clients in this type of situation.

If you still have 7 characters of entropy, that's 60^7 combinations. If the attacker hacked the account in one hour (as they claim), how did the attacker sent 777600000 requests per second for one hour without them noticing? One billion requests per second, that's not something you usually handle. Nor your servers, you would crash anytime.

Something is missing from this story, or i'm blinded and i'm missing something myself.

Nobody from OVH contacted me with any official statement, they even cannot blame for "not giving them enough information". Actually I called them many times and provided them many useful information for "debugging" the issue. So far the official statement from OVH is "not our problems, secure your email better". However, I'll call them tomorrow (again).

More discussion:
https://news.ycombinator.com/item?id=5624728
https://news.ycombinator.com/item?id=5632479
legendary
Activity: 1204
Merit: 1015
What about the user database?  Was it compromised?  I'd hate to see bitcoins sent to the wrong address.

I have a database snapshot taken before bad guys overtook the database. So there's no reason to think payout addresses have been modified. Any change of wallet on pool profile requires email confirmation by account owner so I think we're on safe side here.

Unfortunately the user database can be considered as compromised, so the attacker knows user's emails :-(.

If you have an account at OVH, IMMEDIATELY backup and FULLY WIPE any data you have there. It can be compromised at any second.
legendary
Activity: 1204
Merit: 1015
The pool has been hacked. Fortunately I noticed it fast enough, so I made database snapshot seconds before attackers overtake the database machine. I lost some amount of bitcoins, but I'll be able to recover it from my pocket. For now I'm evaluating what's next to do, because all machines in OVH has been compromised and they cannot be trusted anymore.

Full story:
Today at 3pm UTC I noticed that somebody succesfully resetted the password to OVH manager, the place where servers can be managed, restarted to rescue mode etc. I promptly resetted the password at OVH to something different and I also changed password on my email account and checked that there're no other active connections to my mailbox. I have to say that my mailbox is secured by OTP passwords and I take physical security very seriously, so nobody other had an access to my mailbox. I known that password-reset feature is quite popular attack vector, so I made everything possible to prevent it to happen.

By changing the password at OVH, all other sessions using the old credentials are automatically kicked from the Manager. I also cross-checked that nothing wrong happen to the servers at this time. Unfortunately I didn't find a way how the attackers got access to Manager, so I asked OVH support to provide some additional information and restrict Manager access to my IP range.

That's no surprise that OVH didn't respond to this ticket for hours, but at 11pm UTC I realized that there's another succesful password reset at OVH. This is complete mystery to me, because I'm aboslutely sure that nobody else had access to my mailbox and the email with reset link has been untouched (unread, not deleted). I'd say that attacker won't bother by changing status of the email to "unread", but he'd delete the email instead.

This time I realized that the attacker resetted the machine with the wallet to rescue mode, which means that I lost the control to this machine. I was still succesful by logging into the database and I took the snapshot of database and transferred it to safe location. Few seconds since the migration finished, attackers restarted all remaining machines to rescue mode.

So far it looks like yet another inside job, like Linode two years ago. Or attackers found some shortcut how to gain access to Manager without confirming the request from the email. I don't know what's worse option. I'll investigate this issue in detail later and I hope OVH won't close eyes to this.

I can recover the pool to the normal operation tomorrow.

Edit 01:38 UTC: Stratum servers are running on safe servers at Amazon. Mining works for now. I'll setup new database and webserver on trusted machines in few hours, so the pool will be back in full operation.
Quote
Down for Maintenance
Bitcoin-Central.net is temporarily down for maintenance.

We have been compromised, please do not send any funds to your wallet.

A few hundred BTC have been stolen, but the amount is small enough for us to cover it fully.

Please do not call, do not e-mail. We will give you additionnal information ASAP.nal information ASAP.

Fantastic.

https://bitcoin-central.net/
Jump to: