On Friday, April 13th, 2018, our team encountered a bug while bootstrapping that led to nodes failing to resolve forks correctly and keeping them from staying in sync. Our team immediately reached out to exchanges and advised them to close deposits and withdrawals until the problem was resolved. Once upgraded to version 12, exchanges will be able to reopen and resume trading as usual.
Upon diagnoses, IO thread starvation was causing network disconnections, which led to forks failing to resolve properly. IO threads are supposed to only queue long IO operations and then wait for more work. Instead, it was waiting for a database write lock to be available, which while bootstrapping could be many seconds or longer depending on scheduling.
While the appearance of this bug occurred during the same time-frame as the implementation of universal blocks, it was unrelated. The second canary is still on schedule to be released; this date will be provided in the upcoming days.
Our team has worked around the clock on this issue and putting a fix in place. The update to version 12 results in a more secure and stable Nano network.
Version 12 also de-peers nodes running versions 10 and below. These nodes will need to update to the latest version in order to re-sync with the Nano network, as they do not understand universal blocks.
We appreciate the support from exchanges and for their help, as well as the community’s patience and understanding. We would also like to thank the community members who assisted us with testing version 12.
The version 12 release can be downloaded here.
https://github.com/nanocurrency/raiblocks/releases/tag/V12.012.0 Release Notes
* De-peer nodes < v11 since they don’t understand universal blocks
* For the time being, v11 nodes will act as a bridge between old peers and new peers
* Prevent IO thread starvation
* Improved fork resolution process
* Priority queue for faster cleanup on rollbacks
* Confirmation history tracking
* Less log spam, various log cleanups and more accurate vote and rollback logging
* Answer with successor block on confirmation requests (#803)
* Local work can be disabled if peers are down (#791)
* Disallow account variations (#797)
* Fixed deadlock when shutting down the node
* Elections during bootstrap are more resilient (#810)
RPC
* State block RPC improvements (#785)
* Balance option for blocks_info (#794)
* Work and signature added to raw account_history
* Send callback contains actual amount (#796)