Why was there 600 in a hot wallet? Without cold storage you always take a big risk.
It went up to 600 very quickly. And I didn't want to move any coins offline because at this stage, if people couldn't withdraw for any reason, they would be very nervous (it already happened once).
After the transition it would be the case - ~90% in cold storage and 10% for daily operations. I'm even considering encrypting the private keys with user password, so the attacker can't just use them without user's credentials.
[Added on 17.02.2013, 08:10]Details about the hack follows:
On February 14th, 01:17:21 GMT+1, the attacker approached website's MySQL administration console, which can be seen on the log below:
178.177.206.245 - - [14/Feb/2013:04:17:21 +0400] "GET /adminer-iuqgs124.php HTTP/1.1" 200 2325 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:18.0) Gecko/20100101 Firefox/18.0"
The IP address 178.177.206.245 was used throughout the hack, and wasn't used on the site before. It does not look like a proxy server, and the address comes from Moscow, Russia. The script, which is a
well known MySQL administration toolkit was located at a randomly chosen (by me) filename (adminer-[eight random chars].php). The logs doesn't show any signs of guessing, he knew this filename somehow (more on this later).
Knowing the script's filename is not enough, because you still have to know the MySQL credentials. They are stored only in one place, in a config.php file, accessible only to website scripts. The password was 16 characters long and random: cVPzBh54N6bfdbmb (I've already changed it). Yet, the attacker somehow, again, without guessing, logged in to the console without a problem.
Having writeable access to the whole database he could do whatever he wanted really. He adjusted his account's record with new 'bitcoins' figure, and then even made a fictional transaction to point at his account (this was sloppy though, because he changed an existing one from last year, which was standing out, because this transaction seemed to be made before he even registered).
Then, he proceeded to the site itself. According to my logs, this is how it went:
- First, he tried to login to mralbi's account with a password that was supposedly leaked a year ago from other site. He failed at this.
- Then, he tried to access sonba's account (another high-profile user of Bitmarket, also known on the forums). He succeeded, but for some reason he didn't do anything there (just logged in, and two minutes later, logged out).
- Finally, he went to create a new account. The details are as follows:
[username] => chinabig01
[email] =>
[email protected] [password] => c....1
[country] => fi
- If you google this email address, you will notice that it's not a disposable address. It was used as early as 2009 on various sites (even one Bitcoin site - forbitcoin.com). Also, the username he chose is the same as on those sites. And the password seems to be his username (I don't store sent passwords in server logs, but for critical situations like this I leave first and last letter to prove that someone used a legitimate password for the owner. I know it lowers the password entropy, but if you use long password that you should use, it doesn't matter).
- He then activated his account using his email
- In account settings he set the withdrawal address to
1Lbcfpaw3uHs3iarBqZ12FYeD5vFwNvY49. Again, he confirmed this change by his email.
- Then he proceeded to withdraw the funds. He first withdrew 1 BTC (to test if it worked, I think), then 9 BTC. Finally, he withdrew 55.4561581 and 554.5438419 BTC, which all totals to 620 BTC you can see in the blockchain.
- After that, he went to transactions page to see if the transaction he made up is there, and logged out. I didn't saw this IP on the site since then.
Now, after reading this, there are some legitimate questions that one could ask himself: How in the earth this person could know filename of the script that wasn't posted ANYWHERE? How he knew the MySQL password? I don't know yet. I've asked those questions to my hosting provider and hope to get some answers. There are few possibilities, but at this point it's only guessing:
- there was a flaw in server software. Most critical parts are: Bitcoin client, Apache, PHP, MySQL. Bitcoin client at that time was at version 0.6.1 I believe. The reason for that was, when 0.7 came out, it didn't wanted to work with my wallet for some reason. I didn't want to risk any corruption, so I reverted to 0.6.x. The other bits were using fairly up-to-date software (not the latest point releases, but judging from the changelogs for these, there weren't any flaws in them fixed that could cause this).
- it was an inside job. Possible candidates are: someone from hosting company (which is hard to believe and literally can't be proven) or my old partner (which I don't believe, because he also had some Bitcoins on site when this happened and they were lost as well. He also didn't knew that this MySQL admin tool existed, I've installed this later).- someone hijacked my SSH details. I've looked at the auth.log and that doesn't seem the case.
- a flaw in website's code. I believe it's not the case, because remote code execution (and that's what was needed for this) is fairly easy to spot, and I looked again at the code yesterday and didn't find anything. The codebase is pretty small and straightforward. Also, I was looking through the apache access logs and didn't find any trace of this.
- something else that I didn't think ofSteps, that I've taken so far:
- gathered all necessary information and passed it to hosting company
- changed my SSH password and bitmarket's MySQL password (root password is totally separate, never used anywhere and not stored anywhere)
- removed all remote access to MySQL
- downloaded site logs for futher analysis.
[Added on 17.02.2013, 22:26]Sadly, it turned out that it was my SSH account that was hacked. I don't want to disclose any other information at the moment, because I'm still doing an investigation, that could lead me to the hacker. I will provide more information very soon.