@bible_pay - Good morning. Do you still think it would be best to have a few nodes reindex and resync this morning? If so could you tell me what the Windows command to do a reindex would be? Is it a launch option or a debug console command?
Yes its paramount.
Please Try these 3 tests after deleting debug.log:
biblepay-qt -reindex
biblepay-qt -rescan
biblepay-qt -zapwallettxes=1
Then take a look at the log and ensure there are no errors, and your block height remains the same and wallet balance is still full?
Then after all that you can delete: Blocks, Chainstate, Database folders (of course always keep a copy of wallet.dat!) and then resync from 0 and ensure no banning occurs and syncing does not stop?
Ill do this also asap today.
So doing a reindex had some odd results. It processes through the reindex all the way up until the progress meter says around 4 or 6 hours behind (I've done this twice, first time was 4 hours second time it said 6). In the debug log I see no errors but then tail of the file states the reindex completed:
2017-09-12 13:50:21 ProcessNewBlock : ACCEPTED
2017-09-12 13:50:21 7317.000000 189UpdateTip: new best=8b59e22885de070cb91790c279e66f0a4ec3d0d19d2a9203729d8e984a341945 height=7289 log2_work=46.895307 tx=11177 date=2017-09-12 13:25:24 progress=0.999845 cache=1.1MiB(4593tx)
2017-09-12 13:50:21 ProcessNewBlock : ACCEPTED
2017-09-12 13:50:21 7318.000000 189UpdateTip: new best=709b269c65b9a67926110eba5e677a523e8c06910cd22716651705eb9c78cff7 height=7290 log2_work=46.90914 tx=11178 date=2017-09-12 13:26:49 progress=0.999854 cache=1.1MiB(4594tx)
2017-09-12 13:50:21 ProcessNewBlock : ACCEPTED
2017-09-12 13:50:21 7319.000000 189UpdateTip: new best=f3e51b45b7bdad3a787ef5ff3c7e7524700c062654c4b150eeb27a5f0cafbaff height=7291 log2_work=46.928668 tx=11179 date=2017-09-12 13:29:43 progress=0.999872 cache=1.1MiB(4595tx)
2017-09-12 13:50:21 ProcessNewBlock : ACCEPTED
2017-09-12 13:50:21 7320.000000 Loaded 7320 blocks from external file in 37577ms
2017-09-12 13:50:21 Reindexing finished
What I find odd here is that I'm pretty sure we are in the 7280s..... not 7320 which makes me think when I reindexed that it followed the wrong chain.
If I attempt to rescan in this state when the core comes back up it can never find a source to sync.
This system was originally installed with 1.0.2.x and was upgraded with the existing blocks still in place.
Thoughts? Ideas?
Yeah, thats a good point but I believe I see the reason for this.
So first I just ran the commands, and dont see any problems with our zapwallettxes and rescans, all the coins are in the wallet and no orphan tx are present.
Moving on to reindex, the wallet erases the blocks file and loads the blockindexes back one by one, and counts how many block indexes exist vs how many blocks are in the chain. In this case, I also reindexed 7324 block indexes with a tip height of 7292. So we basically agree, the height is actually 7292 which agrees with the pool. The primary thing to look for is any corruption or business logic failure- It "appears" (God willing) that we are OK as far as algorithm logic, another words, there are no checkblock errors, and all the nodes agree on the 7292 without rejecting one in the middle and causing us to fork back to somewhere between the birth of F7000 and 7292. So thats good.
As far as explaining why we have 28 extra block indexes in memory, that can be explained and is possibly partially not even related to f7000 its probably partially related, but not entirely. The block index can be floating around the network due to blocks that fit on smaller chains, but dont fit in the tree. Another words, we have 28 blocks that were solved by miners that dont fit in the chain but do fit under other blocks. So that is Good also, another words the wallet is finding the longest chain and skipping by those dead ends.
They do get pruned eventually.
Im feeling better about it but not sure if we are out of the woods yet. I think we will need Wests pool for one. I sent him the word document on how to create a second pool. He is working on that.
Eventually the wallet caught up and I went through the rescan and zap and all appears good on this wallet. Hopefully the pool comes back the same way.
At some point when life slows down a bit can you post a debrief paper on what occurred? I'm legit interested in what caused the need for the fork.