For some of the stalled ones, I've tried the fix of deleting blkindex.dat, moving blk0001.dat and blk0002.dat to ~/, and running "./bitcoin-qt -loadblock=~/blk0001.dat -loadblock=~/blk0002.dat". But that just got me back to block 209,841.
I tried -rescan on one, and got that its wallet.dat was corrupted. So I lost a few Bitcoins.
At this point, it seems that my best bet is sending everything from the stalled clients elsewhere. Yes?
Thank you.
Edit: I just transferred balances out, nuked client and folder, and reinstalled. I'm still curious about what happened, though.
Edit2: Well, that didn't work! So far, two affected VMs with fresh bitcoin-qt installs have stalled at block 209,841 again.
Here's what I did:
sudo apt-get purge bitcoin-qt
sudo apt-get autoremove
rm -r .bitcoin
sudo apt-get update
sudo apt-get install bitcoin-qt
I did not retain/restore (AFAIK) anything from old bitcoin-qt installs. How can fresh bitcoin-qt installs inherit state from prior installs? Does the Bitcoin network somehow track machine name, IP address or identifying information from clients?
Edit3: I do know that clients go by IP address
Could the problem be that affected VMs have shared VPN exit IP addresses? Can that confuse peers about which client has which blocks? Can I fix that by manually changing client port number?
edit4: I've retried nuking and reinstalling bitcoin-qt on affected VMs, connecting through different VPN chains (exit IPs). But the clients still stalled at block 209,841.
So I'm just creating new VMs, transferring user data, and doing fresh bitcoin-qt installs. That seems to work. I would be totally gobsmacked if it didn't.
It's possible that my affected VMs contained bitcoin-qt "state-preservation" data outside ~/.bitcoin, which somehow tells peers to stop at block 209,841. Otherwise, it must be that peers are tracking the affected VMs, and stalling them at block 209,841.
Why would that have happened? What information is being used? How is it done?