In this guide 6 sections
Sort every alert into three bins: wake someone now, ticket for tomorrow, or delete. Alert on what customers feel, such as failed checkouts or a growing queue, not on CPU spikes. Give each paging alert a short runbook link so the person woken knows the first step.
Why a noisy pager hides the real outage
Alert fatigue is simple. When the phone buzzes forty times a week and thirty-nine are harmless, the fortieth gets muted. Real outages then run for hours before anybody looks.
Noise usually grows by accident. Someone adds a disk alert after one bad night, another adds a memory alert, and nobody removes either. Each rule felt reasonable alone.
Sort every alert into three bins
Export the last month of alerts from your tool, whether that is Prometheus Alertmanager, Datadog, CloudWatch or plain email. Read each one and ask a single question. Did a human need to act, and how soon?
The answers fall into three bins. Wake someone now means customers are affected at this moment. Ticket for tomorrow means something is degrading but can wait for office hours. Delete means nobody ever acted on it. Be strict about that last bin, because most alerts land there.
- Wake now: site down, payments failing, data not saving
- Ticket: disk filling slowly, certificate expiring in two weeks
- Delete: a CPU spike that cleared itself, repeat warnings
- Anything unowned gets an owner or gets removed
Alert on user impact, not machine twitches
A server at high CPU for ninety seconds may be doing its job. A checkout failing for a customer is never fine. Move thresholds toward symptoms people feel: failed checkouts per minute, login errors, page response time, an order queue that keeps growing, a payment webhook that stopped arriving.
Add a duration to each rule. An error rate that stays high for five minutes deserves a call. One blip does not. Pair the symptom alert with a quiet dashboard of causes such as CPU, memory and database connections, which the responder opens only after being paged.
Make each alert say what to do next
Attach a runbook link to every paging alert. A runbook is a short page. Ten lines at most. It names the likely causes, the first command or dashboard to check, who to call next and how to roll back the last release.
Write it from the actual last incident. A half-asleep engineer at 3am follows steps, not theory. Review the runbook after each page and fix any line that confused somebody.
Also set a clear route. Severe alerts go to the on-call phone, the rest to a ticket queue or a chat channel that nobody is forced to read at night.
Keep the system quiet afterwards
Hold a short alert review each month. Count pages per week, list the ones that led to no action and delete or downgrade them. The target is that every page means real work.
Finally, test your quietest alerts. Stop a service in staging and confirm the right person is paged. Cutting noise must never mean cutting coverage.
Alert triage worksheet
A one-page table listing each alert in rows. Columns hold alert name, source tool, times fired last month, times a human acted, owner, bin chosen (page, ticket or delete), runbook link and review date. A footer box records pages per week before and after the clean-up.
Open a blank worksheet to printQuestions owners ask
How many alerts should page a person at night?
Few enough that each one means real work. If most pages end with no action, the rules are too loose and need tightening or removing.
Should we alert on CPU at all?
Rarely as a page. Keep CPU on a dashboard for diagnosis, and page on symptoms like errors, slow responses or a stuck queue that customers actually notice.
What is a runbook?
A short page linked from an alert. It lists likely causes, the first checks, contact names and the rollback step, so the responder does not have to guess.
How do we avoid missing an outage after deleting alerts?
Keep an outside check that loads your login and checkout pages every minute, and test the paging path in staging after each change.