Alert fatigue: why 1,000 alerts a day is worse than 10
More alerts do not mean more safety. A framework for precision, recall, and the human cost of noise—usable in any NOC, with any tool.
The wrong goal
“Alert on everything that changes” feels thorough. It produces:
- High volume
- Low actionability
- Slow mean time to understand (not just mean time to notify)
A healthier goal: every page should imply a decision. If nobody would change their behavior when the alert fires, it should be a dashboard metric—not a pager event.
Precision and recall (without the ML lecture)
Borrow two ideas from detection theory:
| Idea | In ops language | Failure mode |
|---|---|---|
| Precision | Of the alerts we fire, how many were worth acting on? | Noise; fatigue |
| Recall | Of the incidents that mattered, how many did we catch? | Missed outages |
You can raise recall by alerting on everything—and destroy precision. You can raise precision by alerting on almost nothing—and destroy recall. Mature operations pick a few critical paths and tune both.
A practical triage rule
Before promoting a condition to an alert, ask:
- Who owns the response at 02:00? If the answer is “someone,” it is not ready.
- What is the first action? If the first action is “look at more graphs,” you wrote a ticket to yourself, not an alert.
- What is the cost of a false positive? If it wakes people weekly, the bar must be higher.
- What is the cost of a miss? Customer-facing paths deserve more recall than niche lab circuits.
Severity is a contract
Severity levels only work when they mean the same thing every week:
- Critical — customer impact now, or imminent; drop everything
- High — degraded path or capacity risk; respond in business hours with urgency
- Info / watch — trend, not a page
If “critical” is used for “disk 80% on a non-customer box,” you have trained the org to ignore critical.
Signal hygiene checklist
- Prefer symptoms customers feel (reachability of the service path) over raw vendor counters alone
- Deduplicate flapping: one incident, one thread
- Expire stale alerts automatically
- Review the top 20 noisiest rules monthly—delete or demote without guilt
- Measure acknowledge time and ignore rate, not just alert count