Ok it makes a little more sense now, I believe I see the lions share of the problem (IE bigger than reorganizing).
Ive got two nodes that dont agree here, both consider one of the forks valid but node 2 is following a shorter chain.
Looking at the log on node2, it feels the list of sanctuaries is different than node1. Another words, it has a view of sanctuary payments that differs from other parts of our network.
So basically the sanctuary payment is enforced with a hard rule - and this can cause a fork. We need to either A) Sync all the sancs, or B) Temporarily not enforce that sancs get paid properly.
Ill work on B at a network level now.
Sancs: Please try to sync to the top so your data can synchronize.
Ok, I'm fully positive the problem is that some nodes are denying sanctuary payments, other nodes arent, because they have different views (confirmed in the logs).
So what this means is we either need 51% upgraded or we need to disable this hard check until we are all in agreement (this is sort of a chicken and egg problem).
I think the safer way is for me to disable this rule and then re-enable after we are fully synced. However, the problem is this rule switch-over to spork requires another upgrade.
Building...
P.S. The reason we did not have to go through this before is because at Christmas of 2017 we made our cutover to sanctuaries a mandatory all at the same height (and we only had a few sancs). Now we have 500 sancs that are completely in disagreement with each others height.