it is not coincidence...
I wonder if it's coincidence that a large proportion of the MN's have just just gone to WatchDog Expired states ??
NEW_START_REQUIRED 3
WATCHDOG_EXPIRED 83
PRE_ENABLED 0
UPDATE_REQUIRED 0
EXPIRED 0
Those might be running 1.0.8.2. If the node actually Stopped, their payments will stop.
Anyone who owns one of those 20 nodes, please come online and tell us what happened.
Either way you have to upgrade to 1.0.8.6 to stay up without a crash.
The problem was fixed in 1.0.8.6
On https://biblepay.eu/ we can see the version distribution which is actually protocol distribution, but as I see in GitHub commits, 70715 was introduced in 1.0.8.4? So they are probably not running 1.0.8.2 (right?), but also we don't know if they are running 1.0.8.4-5, or 1.0.8.6+. Maybe Virus can pull the version data to his site, if not, maybe protocol version can be increased with every version, if that makes sense? (I doubt it makes sense )
Either way, my MN went to EXPIRED state last night and it's on 1.0.8.7. When I checked it, the daemon had crashed. These are the last lines of debug.log before crashing:
2018-01-30 04:12:31 Disconnecting unauthorized peer with Network Time off by 7201.000000 seconds!^M
2018-01-30 04:14:37 Disconnecting unauthorized peer with Network Time off by 32401.000000 seconds!^M
2018-01-30 04:19:38 CGovernanceVote::IsValid -- VerifyMessage() failed, error: Keys don't match: pubkey=0fb4c558fb869b2fb813246c419180a01d316684, pubkeyFromSig=b2e987cffe06e8476276f7a88287362e18e60e74, strMessage=82e9a4def54f95dbc71e1fc2ed452c87e92cc44d004026b7774379d758b9e356-0|df3815d09091445c1ce839e7cd29f995abf117837d74525da866b51bbdb7bc16|1|1|1514729478, vchSig=GxNu58oFKlDbVCwpq6cFO0iIAm0GUXP7qsPtVDh5oGY5ErU9GC1SmamKLoaNXr0tI1CJbqavmyU2WKGkTIz1f/I=
2018-01-30 04:19:38 CGovernanceVote::IsValid -- VerifyMessage() failed, error: Keys don't match: pubkey=b2e987cffe06e8476276f7a88287362e18e60e74, pubkeyFromSig=0fb4c558fb869b2fb813246c419180a01d316684, strMessage=36472b9df495a7fabe447fa9ddafb5423af5e311166fc02b9a5a6b73e8fcfc39-1|df3815d09091445c1ce839e7cd29f995abf117837d74525da866b51bbdb7bc16|1|1|1514729478, vchSig=G4TVJF+oPO5QE6+5xFPWn7w7P25i0zSCgoTSEdq/2u1ZZbwZQPjFfK+G8wz58ULVldQkUAZCUpUvqOqVEfAdrt4=
2018-01-30 04:20:40 CGovernanceVote::IsValid -- VerifyMessage() failed, error: Keys don't match: pubkey=0fb4c558fb869b2fb813246c419180a01d316684, pubkeyFromSig=b2e987cffe06e8476276f7a88287362e18e60e74, strMessage=82e9a4def54f95dbc71e1fc2ed452c87e92cc44d004026b7774379d758b9e356-0|df3815d09091445c1ce839e7cd29f995abf117837d74525da866b51bbdb7bc16|1|1|1514729478, vchSig=GxNu58oFKlDbVCwpq6cFO0iIAm0GUXP7qsPtVDh5oGY5ErU9GC1SmamKLoaNXr0tI1CJbqavmyU2WKGkTIz1f/I=
2018-01-30 04:20:40 CGovernanceVote::IsValid -- VerifyMessage() failed, error: Keys don't match: pubkey=b2e987cffe06e8476276f7a88287362e18e60e74, pubkeyFromSig=0fb4c558fb869b2fb813246c419180a01d316684, strMessage=36472b9df495a7fabe447fa9ddafb5423af5e311166fc02b9a5a6b73e8fcfc39-1|df3815d09091445c1ce839e7cd29f995abf117837d74525da866b51bbdb7bc16|1|1|1514729478, vchSig=G4TVJF+oPO5QE6+5xFPWn7w7P25i0zSCgoTSEdq/2u1ZZbwZQPjFfK+G8wz58ULVldQkUAZCUpUvqOqVEfAdrt4=
2018-01-30 04:21:50 CGovernanceVote::IsValid -- VerifyMessage() failed, error: Keys don't match: pubkey=0fb4c558fb869b2fb813246c419180a01d316684, pubkeyFromSig=b2e987cffe06e8476276f7a88287362e18e60e74, strMessage=82e9a4def54f95dbc71e1fc2ed452c87e92cc44d004026b7774379d758b9e356-0|ed9ec4a65237b7d54f1766dffe39f1fb18cbc99783c3495fb927610f3c3620a1|1|1|1514729555, vchSig=HELzseIXLZTQQaCa2MqjUP9VpuCOlQ6zrgBrrRlmq6avSg1waraLOfbXYkYtLVs/c/T2xY6kwhuYN51+OuYOjjo=
2018-01-30 04:21:50 CGovernanceVote::IsValid -- VerifyMessage() failed, error: Keys don't match: pubkey=b2e987cffe06e8476276f7a88287362e18e60e74, pubkeyFromSig=0fb4c558fb869b2fb813246c419180a01d316684, strMessage=36472b9df495a7fabe447fa9ddafb5423af5e311166fc02b9a5a6b73e8fcfc39-1|ed9ec4a65237b7d54f1766dffe39f1fb18cbc99783c3495fb927610f3c3620a1|1|1|1514729555, vchSig=HKq4s+Yl2VQ42mCFLy84mHKXx0Mludl3BZ+kq2Pv9KUKF3qTUIWWjPA8Yb/8Fg96MzKUfl+HntxlAsW7Di6Eizs=
2018-01-30 04:22:28 CGovernanceVote::IsValid -- VerifyMessage() failed, error: Keys don't match: pubkey=0fb4c558fb869b2fb813246c419180a01d316684, pubkeyFromSig=b2e987cffe06e8476276f7a88287362e18e60e74, strMessage=82e9a4def54f95dbc71e1fc2ed452c87e92cc44d004026b7774379d758b9e356-0|ed9ec4a65237b7d54f1766dffe39f1fb18cbc99783c3495fb927610f3c3620a1|1|1|1514729555, vchSig=HELzseIXLZTQQaCa2MqjUP9VpuCOlQ6zrgBrrRlmq6avSg1waraLOfbXYkYtLVs/c/T2xY6kwhuYN51+OuYOjjo=
2018-01-30 04:22:28 CGovernanceVote::IsValid -- VerifyMessage() failed, error: Keys don't match: pubkey=b2e987cffe06e8476276f7a88287362e18e60e74, pubkeyFromSig=0fb4c558fb869b2fb813246c419180a01d316684, strMessage=36472b9df495a7fabe447fa9ddafb5423af5e311166fc02b9a5a6b73e8fcfc39-1|ed9ec4a65237b7d54f1766dffe39f1fb18cbc99783c3495fb927610f3c3620a1|1|1|1514729555, vchSig=HKq4s+Yl2VQ42mCFLy84mHKXx0Mludl3BZ+kq2Pv9KUKF3qTUIWWjPA8Yb/8Fg96MzKUfl+HntxlAsW7Di6Eizs=
2018-01-30 04:25:19 DSEG -- Sent 121 Masternode invs to peer 47696
2018-01-30 04:25:35 Disconnecting unauthorized peer with Network Time off by 14570.000000 seconds!^M
2018-01-30 04:27:14 CGovernanceVote::IsValid -- VerifyMessage() failed, error: Keys don't match: pubkey=0fb4c558fb869b2fb813246c419180a01d316684, pubkeyFromSig=b2e987cffe06e8476276f7a88287362e18e60e74, strMessage=82e9a4def54f95dbc71e1fc2ed452c87e92cc44d004026b7774379d758b9e356-0|df3815d09091445c1ce839e7cd29f995abf117837d74525da866b51bbdb7bc16|1|1|1514729478, vchSig=GxNu58oFKlDbVCwpq6cFO0iIAm0GUXP7qsPtVDh5oGY5ErU9GC1SmamKLoaNXr0tI1CJbqavmyU2WKGkTIz1f/I=
2018-01-30 04:27:14 CGovernanceVote::IsValid -- VerifyMessage() failed, error: Keys don't match: pubkey=b2e987cffe06e8476276f7a88287362e18e60e74, pubkeyFromSig=0fb4c558fb869b2fb813246c419180a01d316684, strMessage=36472b9df495a7fabe447fa9ddafb5423af5e311166fc02b9a5a6b73e8fcfc39-1|df3815d09091445c1ce839e7cd29f995abf117837d74525da866b51bbdb7bc16|1|1|1514729478, vchSig=G4TVJF+oPO5QE6+5xFPWn7w7P25i0zSCgoTSEdq/2u1ZZbwZQPjFfK+G8wz58ULVldQkUAZCUpUvqOqVEfAdrt4=
2018-01-30 04:27:39 Disconnecting unauthorized peer with Network Time off by 3575.000000 seconds!^M
2018-01-30 04:28:01 Disconnecting unauthorized peer with Network Time off by 7200.000000 seconds!
There is a lot more of these errors repeated, this is just a snippet of the last ones. I hope this can help.