Topic: Sentinel patch: a solution to the "WATCHDOG_EXPIRED" problem (Read 862 times)
Is there a windows build available?
hi, sorry to bother you, but..need help. I use Mac and saw the video and pdf from Vivo website regarding how to set up the master node. I followed up every step and now I m very confused. I sent the coins to the address got it on the Terminal (Mac) and server. same for the mastery. I sent the coin and they are gone from my wallet.. I can see the balance 1.000 coin when I connect to the server by terminal (Mac), but first...i can see any master code on the wallet...2..how can I send back from there to my wallet? I scare to loose the coin...compare to Innova master node..where i can see in the master node configuration file in the wallet..there is the third line with my alisa...ip...receiver address and masterkye..in the vivo wallet..there is no such information. do I have to write it...? Please..if you can help me. Thanks
P.s. by terminal, the master node look ok in the server..but on masternode.online they write in the monitoring that i should restart it.
P.s. by terminal, the master node look ok in the server..but on masternode.online they write in the monitoring that i should restart it.
Since Dash moved to a sentinel based gouvernance system, our master nodes have yet a new way to fail. Before we had "NEW_START_REQUIRED" from time to time, but now we have to deal with "WATCHDOG_EXPIRED".
This does not affect only Dash, it also affect any currency based on Dash 12.1+. E.G: vivo, sucr, etc.
For some coins, I see everyday a third of the MN that are not getting rewards because it decided to go "WATCHDOG_EXPIRED", EVEN WITH A WELL CONFIGURED RUNNING SENTINEL.
I've been trying to find the source of the problem. I don't understand fully the sentinel yet, but I discovered a bug in it a few days ago (https://github.com/dashpay/sentinel/pull/32). However, the solution I suggested has been rejected by the dev team. Indeed, they are working on a more general solution (https://github.com/dashpay/sentinel/pull/34) that they will probably publish on the next release.
But while waiting for the "clean" fix, our master nodes are still missing payouts, so I tried to implement it anyway, see if it would work.
I tried it on my nodes, then on some friends, then published it on the vivo discord channel. So far, I see a massive improvement of my node availability and I've got good feedback from the other users.
So I forked sentinel and added my patch to it, and you can find it here: https://github.com/Tetricoins/sentinel/tree/watchdogfix
(You can see the changes here: https://github.com/Tetricoins/sentinel/commit/48fbf36433a4a81e452db032ea9e88be985861c7)
If you look at the commit, you'll see it's a very small change, most of the sentinel works the same.
You are free to use it, and report back here to let me know if it helped you.
This is NOT a permanent solution. Once the Dash team release their own fix, you should switch to their version. I don't think I have the time and resource to maintain a fork for months and months and keep it in sync.
But for now, it's good enough.
How to use it ?
===============
I assume you already have a sentinel working and that you know how to install a one. If you don't, don't ask for help here. It's not a tutorial.
You can just clone my sentinel fork next to the official one realdy on your server. It doesn't matter which coin your are using, sentinels all work the same for all the coins.
Once you have cloned my fork, switch to the watchdogfix branch.
Then you can remove the cron with your old sentinel, and replace it with the new one. It's exactly the same command to use, except YOU NEED TO USE AN ENVIRONEMENT VARIABLE TO DECLARE THE PATH TO YOUR sentinel.conf.
E.G:
* * * * * SENTINEL_CONFIG=/path/to/sentinel.conf && /path/to/virtualenv/bin/python /path/to/bin/sentinel.py 2>&1 >> sentinel-cron.log
With my setup, I this for vivo:
* * * * * cd /home/coins/.vivocore/sentinel; SENTINEL_CONFIG=.vivocore/sentinel/sentinel.conf && ./venv/bin/python bin/sentinel.py 2>&1 >> sentinel-cron.log
So basically, move to the sentinel directory, point to the sentinel configuration file with the SENTINEL_CONFIG variable and run the sentinel from the virtualenv.
It's also very important that the configuration file of your master node (the one you mention in sentinel.conf) is named after your coin. If you use vivo, it must be named "vivo.conf", if you use sucr, it's "sucr.conf", etc.
How to know if it works
========================
Wait a day, and see if you node gets into "WATCHDOG_EXPIRED". That's pretty much it.
If it doesn't suit you, you can always go back to the previous sentinel version.
If you want to learn what's going on, you can activate the sentinel debug mode by setting the SENTINEL_DEBUG to 1 and running the sentinel manually (not in a cron).
E.G:
SENTINEL_CONFIG=.vivocore/sentinel/sentinel.conf SENTINEL_DEBUG=1 && ./venv/bin/python bin/sentinel.py
You'll notice that the sentinel is trying to vote, but that there is a timer for that. If the timer is not up, the sentinel won't do anything.
You can force the sentinel to ignore the timer and vote anyway with the -b option:
E.G:
SENTINEL_CONFIG=.vivocore/sentinel/sentinel.conf && ./venv/bin/python bin/sentinel.py -b
Combined with SENTINEL_DEBUG, it's an nice way of trying to understand what's going on. Careful though, don't use -b too much since it can get you banned.
This does not affect only Dash, it also affect any currency based on Dash 12.1+. E.G: vivo, sucr, etc.
For some coins, I see everyday a third of the MN that are not getting rewards because it decided to go "WATCHDOG_EXPIRED", EVEN WITH A WELL CONFIGURED RUNNING SENTINEL.
I've been trying to find the source of the problem. I don't understand fully the sentinel yet, but I discovered a bug in it a few days ago (https://github.com/dashpay/sentinel/pull/32). However, the solution I suggested has been rejected by the dev team. Indeed, they are working on a more general solution (https://github.com/dashpay/sentinel/pull/34) that they will probably publish on the next release.
But while waiting for the "clean" fix, our master nodes are still missing payouts, so I tried to implement it anyway, see if it would work.
I tried it on my nodes, then on some friends, then published it on the vivo discord channel. So far, I see a massive improvement of my node availability and I've got good feedback from the other users.
So I forked sentinel and added my patch to it, and you can find it here: https://github.com/Tetricoins/sentinel/tree/watchdogfix
(You can see the changes here: https://github.com/Tetricoins/sentinel/commit/48fbf36433a4a81e452db032ea9e88be985861c7)
If you look at the commit, you'll see it's a very small change, most of the sentinel works the same.
You are free to use it, and report back here to let me know if it helped you.
This is NOT a permanent solution. Once the Dash team release their own fix, you should switch to their version. I don't think I have the time and resource to maintain a fork for months and months and keep it in sync.
But for now, it's good enough.
How to use it ?
===============
I assume you already have a sentinel working and that you know how to install a one. If you don't, don't ask for help here. It's not a tutorial.
You can just clone my sentinel fork next to the official one realdy on your server. It doesn't matter which coin your are using, sentinels all work the same for all the coins.
Once you have cloned my fork, switch to the watchdogfix branch.
Then you can remove the cron with your old sentinel, and replace it with the new one. It's exactly the same command to use, except YOU NEED TO USE AN ENVIRONEMENT VARIABLE TO DECLARE THE PATH TO YOUR sentinel.conf.
E.G:
* * * * * SENTINEL_CONFIG=/path/to/sentinel.conf && /path/to/virtualenv/bin/python /path/to/bin/sentinel.py 2>&1 >> sentinel-cron.log
With my setup, I this for vivo:
* * * * * cd /home/coins/.vivocore/sentinel; SENTINEL_CONFIG=.vivocore/sentinel/sentinel.conf && ./venv/bin/python bin/sentinel.py 2>&1 >> sentinel-cron.log
So basically, move to the sentinel directory, point to the sentinel configuration file with the SENTINEL_CONFIG variable and run the sentinel from the virtualenv.
It's also very important that the configuration file of your master node (the one you mention in sentinel.conf) is named after your coin. If you use vivo, it must be named "vivo.conf", if you use sucr, it's "sucr.conf", etc.
How to know if it works
========================
Wait a day, and see if you node gets into "WATCHDOG_EXPIRED". That's pretty much it.
If it doesn't suit you, you can always go back to the previous sentinel version.
If you want to learn what's going on, you can activate the sentinel debug mode by setting the SENTINEL_DEBUG to 1 and running the sentinel manually (not in a cron).
E.G:
SENTINEL_CONFIG=.vivocore/sentinel/sentinel.conf SENTINEL_DEBUG=1 && ./venv/bin/python bin/sentinel.py
You'll notice that the sentinel is trying to vote, but that there is a timer for that. If the timer is not up, the sentinel won't do anything.
You can force the sentinel to ignore the timer and vote anyway with the -b option:
E.G:
SENTINEL_CONFIG=.vivocore/sentinel/sentinel.conf && ./venv/bin/python bin/sentinel.py -b
Combined with SENTINEL_DEBUG, it's an nice way of trying to understand what's going on. Careful though, don't use -b too much since it can get you banned.
Jump to: