Quote
On Monday March 17th, Blockchain services experienced a series of technical issues that originated from a database error and resulted in suspension of services. The issue was entirely unrelated to the earlier problems with the SharedCoin service; it was simply a timing coincidence. This was not the result of hacking or attacks, or of any suspicious activity. It was simply the result of an obscure bug and a combination of technical factors that expressed this bug under heavy load.
For tech savvy people, the specific problem that caused our database cluster to crash was MySQL Error code 625 and 2352. The error in our database affected several tables, across several replicated copies of our database system, specifically those tracking information that is also contained in the Bitcoin blockchain ledger. We were able to diagnose the problem quickly, but because of the very large size of the data sets and the extent of the problem, the fix was not easy. The two specific error codes are only sparsely documented on the web. We decided to pursue multiple methods to try to fix the problem, including re-compiling a custom version of MySQL to get around the error, while simultaneously restoring databases from our backup systems onto new servers. Because of the large dataset, each step in testing, restoring or copying data can take a very long time, making the entire process slow.
A few important facts: at no time were the customer funds or wallets at risk of theft or loss. Blockchain’s architecture ensures that we do not have access to the users’ unencrypted keys. Furthermore, our systems maintain multiple layers of replication and backup to ensure service continuity. During the outage, our customers were able to restore their wallets directly onto other Bitcoin clients, such as Multibit. Transactions and account balances were unaffected, because the customer funds are stored on the decentralized Bitcoin blockchain, not on our servers. Nevertheless, we recognize that this outage has caused concern and disrupted our customer’s use of the web wallet service, causing inconvenience. We sincerely apologize for this inconvenience.
Today, Tuesday March 18th, Blockchain’s services are partially restored and we continue to work hard to restore full services. The Blockchain explorer functionality, which allows people to search in the Bitcoin blockchain has been restored. We are still restoring databases and rebuilding services in a phased approach. Over the next 72 hours, we anticipate all services will return to normal, though we hope to have many restored much sooner than that. We are also adding new servers to support the higher-than-usual traffic we anticipate as services are restored. If you need urgent access to your funds, please follow the instructions here to restore to a desktop client or send us a support request here, if you don’t have a backup of your wallet. You can access your funds independently of our service.
Rest assured, your funds are safe, your wallet backups are safe, our services will soon resume without any loss other than to our pride. After this incident is over, we will conduct a full review; we will publish more information about exactly what happened and lessons learned. We will continue to improve our infrastructure to ensure we are much more resilient in the future and are ready to serve our 1.4 million and fast growing user base. We appreciate your support and patience and we promise you that we are working as hard as possible, around the clock and around the globe to fulfill our responsibilities to our users.
Thank you for your support and patience,
Blockchain Team
For tech savvy people, the specific problem that caused our database cluster to crash was MySQL Error code 625 and 2352. The error in our database affected several tables, across several replicated copies of our database system, specifically those tracking information that is also contained in the Bitcoin blockchain ledger. We were able to diagnose the problem quickly, but because of the very large size of the data sets and the extent of the problem, the fix was not easy. The two specific error codes are only sparsely documented on the web. We decided to pursue multiple methods to try to fix the problem, including re-compiling a custom version of MySQL to get around the error, while simultaneously restoring databases from our backup systems onto new servers. Because of the large dataset, each step in testing, restoring or copying data can take a very long time, making the entire process slow.
A few important facts: at no time were the customer funds or wallets at risk of theft or loss. Blockchain’s architecture ensures that we do not have access to the users’ unencrypted keys. Furthermore, our systems maintain multiple layers of replication and backup to ensure service continuity. During the outage, our customers were able to restore their wallets directly onto other Bitcoin clients, such as Multibit. Transactions and account balances were unaffected, because the customer funds are stored on the decentralized Bitcoin blockchain, not on our servers. Nevertheless, we recognize that this outage has caused concern and disrupted our customer’s use of the web wallet service, causing inconvenience. We sincerely apologize for this inconvenience.
Today, Tuesday March 18th, Blockchain’s services are partially restored and we continue to work hard to restore full services. The Blockchain explorer functionality, which allows people to search in the Bitcoin blockchain has been restored. We are still restoring databases and rebuilding services in a phased approach. Over the next 72 hours, we anticipate all services will return to normal, though we hope to have many restored much sooner than that. We are also adding new servers to support the higher-than-usual traffic we anticipate as services are restored. If you need urgent access to your funds, please follow the instructions here to restore to a desktop client or send us a support request here, if you don’t have a backup of your wallet. You can access your funds independently of our service.
Rest assured, your funds are safe, your wallet backups are safe, our services will soon resume without any loss other than to our pride. After this incident is over, we will conduct a full review; we will publish more information about exactly what happened and lessons learned. We will continue to improve our infrastructure to ensure we are much more resilient in the future and are ready to serve our 1.4 million and fast growing user base. We appreciate your support and patience and we promise you that we are working as hard as possible, around the clock and around the globe to fulfill our responsibilities to our users.
Thank you for your support and patience,
Blockchain Team