At iLert we are big friends of dogfooding as we love to take our features and check out any way possible to hack or integrate them.
We have launched the auto-raise feature for support hours in early 2020 and they really are a fancy way to setup wake-up calls in the morning.
In case you already know iLert, you can skip this part and start right at The wake-up call.
Alert sources are the basis of any kind of interaction in iLert. An alert source is a self-service inbound connection to your preferred monitoring software or other tool.
Support hours help you manage the priority of incoming alerts for an alert source depending on the current time and day. E.g. from Monday to Friday between 8 am and 5 pm you want to be immediately alerted and outside of these hours the alerts should not keep you from enjoying your free time.
But what if you want to be immediately informed in case there have been any alerts during your well deserved free time after coming back into the office on Monday 8 am?
We got you covered with the auto-raise feature for support hours. All you have to do is enable the “Raise priority” checkbox and we will raise the incident’s priority for the alert that came up during your off-time as soon as your next support hour start.
We have multiple small check applications in different zones across cloud providers to validate the response and uptime of our service. These small apps are actually Rust binaries which constantly poll our API and run through test operations e.g. create and update an escalation policy.
In case any of these checks fail they create an incident right in iLert - but additionally once a day they create a sum-up incident with the response statistics of the last 24 hours.
We are going to use this daily sum-up event to wake us up in the morning with a call informing us about the latest statistics to start the day right.
We want the new alert source to be of integration type: ‘API’ with incident creation enabled.
Choosing ‘High during support hours’ helps us to receive our sum-up updates all day in iLert (except for our active time from 8 - 9 am) with a low setting (only sending us an email per default). But once we reach a new support hour window at 8 am, the event will be risen and we will receive a voice call with its details (as configured in our profile above).
We are using the iLert Rust client to send the event to iLert’s API. Make sure to replace the placerholder with the API key exposed by your alert source.
You can also find the API documentation here for your client or simply try out the curl command below :)
Now we just have to make sure that the event is triggered every 24 hours. (The Rust excerpt below is just a mockup, I would not recommend using this in production environments.)
When the event is received outside of the support window, a low priority incident is created.
But as the support hour window is approached it will be automatically risen and become a high priority incident, which will cause an outbound voice call that wakes us up with the incident summary in the morning.
After listening to the wake up call we can straight up answer by pressing ‘1’ on our phone to resolve the incident.
Surely in 99% of the cases you would only ever sent events that actually evolved out of some kind of issue or problem of which you want to be informed about.
However it is always good feeling to start the day knowing that everything ran smoothly while you were asleep.
A better way to be on-call for your team.