How to use Monit to Monitor your Node and Restart znnd when memory usage gets too high

Several node and pillar operators have reported high memory usage for the znnd service. I personally run a node with 64g of ram and it rarely runs out of memory. However, on smaller systems with 16G of ram I regularly have issues with the znnd process consuming 100% of the ram and then crashing. Users have resorted to setting up a cron job to reboot the service or server all together. I found a better way to manage this situation with the use on monit.

Monit is an open source tool that can monitor running services, memory usage, cpu usage and many other things. With monit you can setup a Process ID file to monitor the znnd service. When memory usage of znnd reaches a user defined threshold, you can setup an action to restart the service or send an email. That way you can proactively monitor the znnd process and automatically take action when it hits certain thresholds. The instructions below explain my basic setup for a Linux Machine running Ubuntu 22.04. Over time I will expand these instructions to include a telegram notification bot.

Create a .pid file for Monit to Monitor

This assume you have installed the znn-controller software.

sudo nano /etc/systemd/system/go-zenon.service

add the following line to the go-zenon.service file

ExecStartPost=/bin/sh -c "echo $MAINPID > /var/run/"

The service file should look like this when done.

    Description=znnd service
    SuccessExitStatus=SIGKILL 9
    ExecStop=/usr/bin/pkill -9 znnd
    ExecStartPost=/bin/sh -c "echo $MAINPID > /var/run/"

In order to reload the changes to the systemd unit file type the following command.

sudo systemctl daemon-reload

Install Monit on Ubuntu 22.04

sudo apt update && sudo apt upgrade

After running the system update command, install Monit Monitoring on Ubuntu 22.04. It is available to install through the default system repository. Hence, no need to look for some third-party repository for future updates.

sudo apt install monit

Check service Status & Verison of Monit

Once the installation is completed, check whether its service is running without any error in the background.

sudo systemctl status monit --no-pager -l

Service Status

If it is not running, start it using:

sudo systemctl start monit

To check the version:

sudo monit --version

Configuration File

The Monit program can be configured using the /etc/monit/monitrc file. There are numerous sample settings, some of which are commented out, which are self-explanatory, or whose comments contain help texts. I recommend that you do not make your own settings directly in this file. It is better to create a new one for the desired settings.

sudo nano /etc/monit/conf.d/znnd.conf

Paste in the following setting and modify them for your specific settings.

set mailserver [MAIL SERVER] port [PORT] #change MAIL SERVER and PORT
    username "USERNAME" password "PASSWORD" #change USERNAME and PASSWORS
    using tls #change tls to the correct protocol or leave as is

set alert #change email address

check system $HOST
    if loadavg (1min) per core > 2 for 5 cycles then alert
    if loadavg (5min) per core > 1.5 for 10 cycles then alert
    if cpu usage > 95% for 10 cycles then alert
    if memory usage > 75% then alert
    if swap usage > 25% then alert

check process znnd with pidfile /var/run/
    start program = "/usr/bin/systemctl start go-zenon"
    stop program  = "/usr/bin/systemctl stop go-zenon"
    if cpu > 60% for 2 cycles then alert
    if cpu > 80% for 5 cycles then restart
    #if memory usage > 75% then alert
    #if memory usage > 90% then restart
    if totalmem > 32000.0 MB for 5 cycles then alert #adjust for your situation
    if totalmem > 40000.0 MB for 5 cycles then restart #adjust for your situation

Enable m/Monit httpd port on Ubuntu

By default, port 2812 to communicate Monit will be disabled and has to be enabled by editing its configuration file.

sudo nano /etc/monit/monitrc

Find the line: set httpd port 2812

There remove the # tag for the following lines. Also, replace the allow and use address value from localhost to if you want to access the Monit web interface remotely as shown in the screenshot.

You can also change the default password for the admin that is monit.

set httpd port 2812 and
use address # only accept connection from localhost (drop if you use M/M>
allow # allow localhost to connect to the server and
allow admin:monit # require user 'admin' with password 'monit'

Check and Load the Changes

To check configuration files are without any errors use the following command:

sudo monit -t

# If there are no errors, you will receive the following feedback:
# Control file syntax OK

Reload Monit in order to load the changes

sudo systemctl restart monti