This might be a Bitcoin Cash only issue (or even Bitcoin Unlimited only issue).
On the other hand, it might be a source of intermittent/hard to track down blockchain stalls with all the other clients.
It
looks like the Bitcoin (Cash) Unlimited client manages the block files in an unexpected way. When I look at the block files, some earlier block files have modified times that are much more recent than they should be.
The
equivalent Bitcoin Core code seems to work the same way, but I haven't actually checked.
I did a full resync (deleted all the blk files and chainstate) and left the node running in the background. Once I noticed that it resynced, I restarted the node (before the 15:57 timestamps) and then started Armory after wiping the Armory databases.
Armory resynced and then stalled. It said connected to the node.
This is the end of "ls -lrt blk0*" and I assume the node completed resyncing around 6:45am.
-rw------- 1 tiern tiern 120188494 Nov 19 06:41 blk01037.dat
-rw------- 1 tiern tiern 134182052 Nov 19 06:41 blk01038.dat
-rw------- 1 tiern tiern 134024033 Nov 19 06:42 blk01039.dat
-rw------- 1 tiern tiern 127342200 Nov 19 06:44 blk01040.dat
-rw------- 1 tiern tiern 131259910 Nov 19 06:45 blk01041.dat
-rw------- 1 tiern tiern 83886080 Nov 19 13:14 blk01042.dat
-rw------- 1 tiern tiern 134215061 Nov 19 15:57 blk00001.dat
-rw------- 1 tiern tiern 134215184 Nov 19 15:57 blk00004.dat
-rw------- 1 tiern tiern 134216866 Nov 19 15:57 blk00006.dat
-rw------- 1 tiern tiern 134213458 Nov 19 15:57 blk00007.dat
-rw------- 1 tiern tiern 134198809 Nov 19 15:57 blk00008.dat
-rw------- 1 tiern tiern 134207767 Nov 19 15:57 blk00009.dat
Bitcoin node code was changed to this method pretty early on (in this
commit from Aug 2012).
When a new block is received from the network, it scans all the blk files and finds the first one that has enough space to store the new block.
This means that the file systems doesn't act as "append-only" overall. It is append-only for each file though.
It has a variable called nLastBlockFile. This variable acts as a lower limit on the scan. Once it writes to a blk file, it never checks files lower than that anymore. That is a RAM variable though, so it gets set back to zero each time a node is restarted.
I think the problem with Bitcoin Cash is that the blocks are highly variable in size. If the current blk file doesn't have enough to store a large block, the client will move on to the next blk index, leaving a large amount of free space.
The next time the node is started, a few 50kB blocks will probably fit in spaces like that. It highlights the issue, but I think it could happen on the Bitcoin Core client too.
I think this is not compatible with Armory? It assumes that the blk files are filled in order and there are no movements backwards.
A solution would be to keep a record of "last-checked time" and then re-scan any old blk files that have a modified time greater or equal to the last-checked time. This check could be run any time a new block notification is received. It should only be a few blk files at any one time.
Even better would be to record how much of each file was already processed, but that is probably to much hassle.
Would re-scanning already scanned blk files cause problems for Armory?
The workaround is to wipe the Armory database and let it re-scan everything if the block stalls but that is a sledge hammer.