Top on-call practices for SREs (by SREs)

This September, we hosted a few of our customers at the ilert headquarters in Cologne. The plan was to share where we're heading, have real-life conversations, and give them more room to ask us questions and share their thoughts.

‍

One part of the day was a discussion about on-call culture with some of the greatest minds in SRE (source: they chose ilert as their incident management tool). We asked our customers: what are your top practices for on-call? Everyone shared their thoughts and voted for the ones they agreed with most. We've kept the answers in their own words.

1. Be ready and reachable

The worst thing that can happen when an alert fires is that your team can't reach you. So the most practical answers of the day went straight to the top:

• “Have the ilert mobile app installed”
• “Keep your phone charged and reachable”

It sounds obvious, and it is. But being reachable is the foundation everything else in on-call depends on. If an alert can’t reach the right person, nothing else in the response matters. With the ilert app, critical alerts get through to you, even when your phone is on Do Not Disturb.

2. Build a positive team culture

The group had a lot to say about team culture and mindset. The consensus was that a positive mindset matters:

• "Good team spirit helps you get through difficult situations."
• "Be kind, stay cool, go on."
• "Always keep cool 😎"
• "Blameless."

Many mentioned some version of "be good to each other." Tools make sure you get paged, but people are what get you through the night. Being blameless matters most once the incident is over: the postmortem should be about the system, not about who pushed the button. 

If you want to see how others do it, browse our Postmortem Library.

3. Only page for what matters

This is the practice teams most often get wrong, and the group had strong opinions on it:

• "If a page can't be acted on at 3am, it shouldn't page."
• "Separate product support from SLA incidents: only report and resolve SLA-relevant incidents."
• "Empower your support agents to solve the culprit of alerts rather than teaching them how to snooze in less than 3 seconds."

We had a laugh about the snooze speedrun, but the point is serious. Every alert should be something a person can act on. If it isn't, it needs to be tuned, grouped or moved to business hours.

4. Communicate openly and make escalation easy

When an incident hits, how you talk about it matters almost as much as how you fix it. The group agreed on two things:

• "Keep the communication public (company-wide), not in private channels or chats."
• "Fallbacks and easy ways to escalate."

Incident context gets lost in DMs. When updates go into a shared channel or onto a status page, the next responder, your support team and your manager can all see what's going on without asking. Fallbacks mean nobody becomes a single point of failure. If the first responder doesn't acknowledge an alert, an escalation policy should do the chasing so no human has to.

5. Make on-call visible to management

One practice was aimed less at the engineers on call and more at the people who plan their time:

• "Make capacity going into on-call and support duties transparent to management – nothing comes for free."

It wasn't the loudest practice of the day, but it may be the most important sentence. On-call is work. It uses up hours, focus and sleep. If it doesn't show up in capacity planning, it will show up later as burnout.

Bonus: It was always DNS

It didn't top the list. Probably not because anyone disagreed, but because nobody needed convincing.

What we're taking home

When you bring together the people who run on-call teams, you might expect a list of tools and processes. Instead, the practices that rose to the top were "charge your phone" and "be kind." The basics still win. The teams that get on-call right are the ones that make it predictable, actionable and humane, and that takes support from the whole organization, not just whoever is holding the pager tonight.

‍

Want to make "ready and reachable" the easy part? Try ilert for free.