Prioritize Incident Notifications

Mean Time To Response (MTTR) measures how long it takes a first responder to acknowledge an incident since it has occurred. There’s a lot of talk on minimizing MTTR. But take a look at the following data set. Which scenario do you prefer?

MTTR Scenarios in Table Format

At first glance, Scenario 1 seems preferable with an average MTTR more than 2x faster. But when you break it down by incident urgency, suddenly we’re talking about a very different story.

MTTR Scenario 1 Breakdown Graph

MTTR Scenario 2 Breakdown Graph

Incident Urgency Matters

Remember the Boy Who Cried Wolf? He’s the shepherd boy who keeps waking up the whole town just for fun. And the one time his sheep really are in danger, no one responds because they’re tired of his antics. It’s a simple child’s tale about false alarms. But it hits close to home for those of us on-call.

Boy who cried wolf cartoon

Whether we’re being rudely awaken in the middle of the night, stepping away from a family dinner, or pulling the car over to check an alert, our phones blow up with every minor incident. But there’s a world of difference between a momentary spike in load times compared to a full on outage!

Simply put: urgency matters. If every incident is critical, then no incident is critical. That’s how you end up in scenario 1, where every incident gets treated exactly the same way. Sure, you’re MTTR across the board looks great. But when critical incidents have the same MTTR as low urgency incidents, you’re doing something wrong. The fact is, people get desensitized with too many false alarms. They suffer from alert fatigue, and when a critical incident actually strikes, they’re going to operate at the same, single speed they’ve grown accustomed to.

Design Smart Notifications

Enter PagerTree’s configurable notification settings. PagerTree’s notification settings allow you to specify notification preferences based on the following criteria:

  • Urgency
  • Time of Day
  • Day of the Week
  • Key Words in the Title and/or Description

Let’s take a look at a few real-world situations.

Rule #1

In my first rule here, I’m going to ignore any low urgency incidents that occur after hours. There’s no point infringing on family time for something that can surely wait until the next morning. My team and I are on the same page here. If it’s low, let it go. We’ll see the incident first thing in the morning, and we can figure it out then.

Rule #2

In Rule #2, I’m using a tiered notification method. For medium and high urgency incidents, I definitely want to hear about it. But it’s probably not worth waking up the kids over. I’ve set my preferences to start with a text message so it’s subtle enough, but if I don’t get to it in one minute, go ahead and call me.

Rule #3

Rule #3 deals with critical alerts. And when I say critical, I mean critical. As you can see here, I’ve got my settings set to blow up my phone with simultaneous Push, SMS, and Voice notifications. And if I somehow miss the first round of obnoxious noises, it’s set to repeat again and again.

Takeaway

Notification rules give you the flexibility to respond with the appropriate sense of urgency. There’s no need to call out the fire brigade for nothing. But when things get critical, you’ve still got all hands on deck.

The customizations are endless, so don’t let your setup become the boy who cried wolf. Empower your employees, reduce noise, and swarm on critical issues with PagerTree’s notification rules.