Alert fatigue: what causes it and how to fix it

Ask an on-call engineer what they fear and the honest answer is not the 3 a.m. page — it is the channel with four hundred unread alerts where one of them matters. Alert fatigue is not a comfort problem. A team trained to skim is a team that will miss the real incident, because skimming is precisely the skill their alert volume taught them.

Fatigue is always a symptom of specific, fixable causes. Here are the five that account for nearly all of it.

The root confusion is treating every alert as an interrupt. An alert is a page only if it needs a human now, out of hours, at interrupt priority. Everything else — degradations with headroom, batch-job failures with a morning deadline, warnings — belongs in a queue, handled at queue priority.

The fix is severity-gating: only P1/P2 conditions page; everything else files a ticket. The pager going quiet except when it matters is what rebuilds the reflex that a page means "move now."

One root cause rarely produces one alert. A bad deploy or an expiring certificate fires across every service that touches it, and each firing becomes its own notification or ticket. The queue fills with forty copies of one problem, and the on-call spends the incident closing duplicates instead of fixing the cause.

The fix is deduplication at ingestion — and it has to be semantic, not string-matching, because the same root cause phrases itself differently in every service. Collapse the storm into one ticket with an occurrence count. This is exactly what FlowTux does with error-tracker and CI signals: repeat occurrences fold into the original ticket instead of multiplying it.

Thresholds set once and never revisited drift out of sync with the system they watch. Traffic doubles, a 70% CPU alert becomes background noise; a dataset grows, the job-duration alert fires every night. Each stale threshold contributes a trickle of noise, and the trickles add up to a flood.

The fix is a review habit: any alert that fired without producing action in the last month gets retuned, downgraded to ticket-only, or deleted. An alert that never leads to action is not monitoring — it is decoration with a notification sound.

An alert that says "error rate elevated" hands the responder a research project. At 3 a.m., the gap between "something is wrong" and "here is what to check" is twenty minutes of dashboard archaeology — and after enough of those, responders stop opening alerts promptly at all.

The fix: every page carries context — what fired, what it means, what to check first, and a link to the runbook if one exists. If nobody can say what a responder should do when an alert fires, that alert has not earned the right to page anyone.

Broadcast alerting — one channel, whole team — guarantees that most alerts each person sees are not theirs. Relevance is the strongest predictor of whether an alert gets read, and broadcast destroys it. The bystander effect does the rest: an alert that is everyone’s responsibility is nobody’s.

The fix is routing by ownership: alerts land with the person or rotation that owns the failing component. That requires knowing what each signal belongs to — which is why triage grounded in the codebase helps here too. When the system can tell which module an error came from, it can page the person who owns that module and no one else.

The fixes above decay without maintenance, so institutionalize one habit: a short weekly review of what paged, what was actionable, and what was noise. Retune or retire the noise on the spot. Teams that run this review keep their pager trustworthy; teams that skip it rebuild alert fatigue within a quarter.