BLOG

Alerting with ilert and Pandora FMS

Sancho Lerena

October 19, 2023

Table of Contents:

This post introduces the Pandora FMS monitoring solution and how to integrate it with ilert to establish reliable alerting. The guest post is written by Sancho Lerena, the CEO of Pandora FMS.

Why use ilert with Pandora FMS?

There are notification options in a monitoring application as flexible as Pandora FMS, but regardless of how many options it includes, it is not a specific application to cover the complexity of the notification flows of a complex organization, which requires integrating many systems that generate notifications. It is not the same—detecting a process down, receiving an asynchronous event through a WEB API, or processing an email, just to mention three possible alert sources in a complex environment. Pandora FMS is good at monitoring, and there are hundreds of plugins published in Pandora FMS public module library.

‍

In addition, receiving an email or SMS is easy today; the complex thing is to provide each user with their favorite notification system (SMS, email, voice call, Whatsapp, Telegram) and for the configuration to be fast, easy, and centralized. Anything is possible, but it takes time and is subject to failure. Do you want the notification to fail because an API has changed? Can you really afford not to receive a notification?

‍

That's why ilert makes the most sense to you.

‍

It is not because Pandora FMS cannot send notifications directly; it is because sometimes you just need to implement a convergence of notifications from other sources, and sometimes you do not have time to implement the complexity of a complex notification workflow that includes push technologies such as Whatsapp or Telegram, and you may want to delegate this to a specialized cloud provider, such as ilert.

‍

For all purposes, ilert works as a notification funnel that collects all notifications from different sources; once received, and thanks to a complete scaling system, workflow rules, group/user system, and notifications, it develops all necessary actions, regardless of the notification sources, their format, frequency or structure.

‍

ilert can also be integrated with your ITSM or used to enrich alerts. It can also be integrated with Pandora ITSM.

Setup in 5 minutes

Signing up for ilert takes less than 30 seconds. The next step is to configure users to receive different types of notifications: voice calls, SMS, WhatsApp, Telegram, or email. From this account, you may receive notifications from Pandora FMS, a script, another application, or workflows as manual and wide-ranging as receiving an email. The API in ilert is flexible, simple, and allows you to get your project up and running in minutes.

Setting up the user (and how you want to receive notifications)

The first thing is to configure a user (if you already have one created, configure/enable the different notification methods). Let's look at this example, where email sending (by default), SMS, voice calls, and WhatsApp are configured:

‍

To activate each of the systems, it will ask you to activate it. To do it through WhatsApp, it notifies you through your cell phone to ask for confirmation, and you just have to click. The same goes for voice calls, SMS, email, etc. You don't have to install or do anything.

Each user has their own notification system. User groups (teams) can be created, and groups can be notified. There is the possibility of creating rules for notification, scaling, etc. The power of ilert lies in how it manages notifications from third parties.

‍

The notification "order" or priority is established in the notification screen of the configuration of each user:

Configuring notification escalation

If you edit the values that come by default, you will see that it is a set of simple rules. It allows you to define the alert escalation flow based on people. Each person/user has already configured their notification system (SMS, call, etc.).

‍

Since all the scaling logic (time windows, repetitions, workflows) is in ilert, do not forget that the "alert sources" send you multiple alerts, ilert will help you filter and sort out things.

Creating notification sources

‍

Here is where the interesting begins. Let's define how you may add alerts to ilert. To show the power of ilert, we are going to integrate Pandora FMS and integrate two other very different sources: generic emails and another for use in an API from a script.

‍

ilert alerts using an email as a source

This is one of the easiest. As simple as sending an email to a specific address, and that will launch an alert. To that end, go to Alert sources > Create new alert source and then filter the results by "mail":

Use that source (email) and fill in some data:

If you have not yet created a Team, you may create it later. Accept the following screen with default values, and finally configure the email to which you will send the notifications:

Click continue to finish the setup:

Now, you just need to send an email from any application to "pandorafms@pandorafms.ilertnow.com" to enter your notification funnel.

‍

ilert alerts using Pandora FMS as a source

‍

In a very similar way, create a "Pandora FMS" data source, which comes by default in the ilert alert source library. After following all the steps, it will return some data that you will need to integrate the alert with Pandora FMS:

Now it's time to "connect" your Pandora FMS with ilert; for that you will need the API Key. You may use ilert in any Pandora FMS version (even the older ones), but since version 775, it already comes "as standard" integrated into the action system. We are going to describe how to implement it from scratch, perfect if your version is older. If you already use a Pandora FMS 775 or higher, you may skip almost all of the following steps (command creation, action creation), and you may directly edit the action that comes by default with Pandora FMS for ilert to set the API Key generated in the previous step.

‍

To create an ilert notification with Pandora FMS, follow these three steps: Create a command, an action, and associate an alert template with that action.

‍

Create the command in Pandora FMS to call ilert

Download the script from Pandora FMS's own plugin library at https://pandorafms.com/library/ilert-integration/. In Pandora FMS 775 and later, it is already installed as standard; you do not have to follow this step (we also use our own version of the integration, which is a little more complete). To see how to install/use the ilert tool itself, just keep reading.

For that, you have to enter the Pandora server by shell, download the file, decompress it, and add it to the indicated path. It is summarized in the following commands (executed as root).


cd /tmp
wget https://pandorafms.com/library/wp-content/uploads/2022/11/pandorafms_ilert.sh.zip

unzip pandorafms_ilert.sh.zip
mv pandorafms_ilert.sh /usr/share/pandora_server/util/pandorafms_ilert.sh
chmod 755 /usr/share/pandora_server/util/pandorafms_ilert.sh

In the Pandora console, add an Alert Command (In the sidebar, go to Alerts -> Commands and click Create).
Enter a Name for the command.
In the Command field enter:


 \
sh /usr/share/pandora_server/util/pandorafms_ilert.sh "_field1_" "_field2_" "_id_alert_" "_field3_" "_field4_" "_timestamp_" "_alert_text_severity_" "_agentname_" "_module_" "_data_"

In the Description field, enter:


_field1_ : ilert alert source API key. _field2_ : Type of the created event. Can be "alert" or "resolved". _field3_ : Title of the event. _field4_ : Description of the event.

‍

Once created, it will be available as a command:

Creating an action using the ilert command

We will create an action, which is like the alert; it will execute the command (with certain parameters) for each case. We are only going to create a certain case (action), but you could define several if you needed to. In there, we will configure only field 1 (API Key), copied from the ilert configuration, and field 2 (alert type), specifying "Alert" when it is triggered and "Resolved" when it is recovered.

That way, any use of the "ilert General" action now will already contain the APIKey, and you will not need to specify it.

‍

Creating an alert using the ilert action

‍

Create an alert about a module (in this example, a disk one) that uses your command through the newly created action:

ilert Alerts using a generic external API as a source

Imagine that Pandora FMS process itself can not be activated because the database failed; how would you notify it? Well, also with ilert!

‍

For this purpose, create a hook via API so that it can be called from any script.

‍

First, create an API as an alert source in ilert. When you finish configuring it, it will return a reference to be able to use it:

To enter a notification into ilert, just run this command from any shell:

‍


curl -X POST -H "Content-Type: application/json" \
    -d '{"apiKey": "il1api10230c21209d93294006a2fa47528fae069bc8f2f0820e","eventType": "ALERT","alertKey": "Pandora General Alert","summary": "Something goes wrong"}' https://api.ilert.com/api/events

‍‍
‍The only thing you will have to replace is apiKey on your own. Only with this execution, you may launch an alert to ilert from any script. It is a perfect way to ensure, for example, that if everything fails, you will at least know that everything failed. If you do not know how to make that script, you may check Pandora FMS community; in addition to being experts in monitoring, we master Linux.

What does an ilert alert look like?

By e-mail:

By SMS:

By WhatsApp:

And of course, in the ilert control interface where you may do many things with the alert:

Advanced uses of ilert: event correlation with Pandora FMS

‍

With Pandora FMS, one can monitor 20 servers or 2000. It is easy to imagine that if you happen to have dozens of metrics on each server, assigning an alert to each and every module can be a burden. We have tools like policies or the bulk operations manager to assign hundreds of alerts in one go, but even so, it's complex and cumbersome.

The perfect solution in these cases is to assign alerts by "concepts," let's see some cases:

Assign an alert when MySQL processes fail without looking at which machines they are on. To do this, you would assign alerts on any failure event of a specific module, regardless of the agent.
Assign an alert when something happens in modules marked with a label, for example, "Critical Infrastructure."
Assign an alert when something happens in any data source of a group of systems that belong to a group, for example, "Production environment."
Assign an alert when complex conditions are met, for example: "Failure of any MySQL process" in "the group of production machines."
Assign more complex conditions, such as the previous ones, using logical operators (AND, OR, NOT, XOR...) in time windows.

For this, there are event alerts or event correlation alerts. They work on the events generated by modules, not on the modules but on their data. This allows operating on more generic concepts, not on specific data, since a fail event of a MySQL process is the same in one machine as in another (if the module is called the same), the same as a fail event is a fail event, no matter the agent or no matter the module. Therefore, applying rules on events is much more efficient and intuitive than doing it on modules.

‍

Example of defining an event correlation alert.

If you also add ilert to the event alerts, you can further optimize the process because ilert has additional logic that you may use together with that of Pandora FMS. With Pandora FMS, it is possible to set special day calendars, scheduled downtimes, or time slots, you may also set escalation rules and many other things, but ilert allows you to do so on alerts from other data sources that are not integrated with Pandora FMS.

‍

You can read more about event correlation in Pandora FMS in Pandora FMS online documentation.

You may also get help online in support forums and in Pandora FMS Discord community.