In this guide 6 sections
Watch journeys, not just servers. Combine metrics, logs and scripted checks of checkout and login, then route every alert to a named on-call person with a severity level. A dashboard showing only that the machine is running misses the failures that cost customers their orders and staff their afternoons, so a healthy server proves little.
Starting from the journeys that matter to the business
List the journeys first. Checkout, a staff form that triggers downstream processing, and the integration syncing orders to accounting each deserve a measurable reliability target, known as a Service Level Objective.
Most setups go wrong at this step. It is tempting to monitor what is easy, such as server uptime and CPU use, instead of asking whether a customer can finish a purchase.
Consider an online ordering app for a restaurant group whose server reported healthy while the kitchen-printer queue backed up for two hours. Orders were charged but never printed. Nobody noticed until a branch manager phoned head office to ask why a normally busy Saturday evening had gone so quiet while customers waited for food nobody was cooking.
Metrics, logs and synthetic checks together
Metrics give the current state in numbers. Checkout latency, error counts and connection-pool usage suit trend-spotting and alert thresholds, whereas logs hold the detail behind a spike: which request failed, for which customer, at what moment. A team reads logs to diagnose an incident rather than merely learn that one is happening, so keep them searchable for long enough to cover a slow-burning fault.
Synthetic checks are scripted transactions. They run on a timer and walk a real journey end to end. Examples are logging in, adding a product to a basket, paying with a test card and calling the courier-booking endpoint. Metrics miss these failures. The server reports healthy while a third-party payment gateway rejects every card, or a canary release broke one step in a longer flow that no Grafana panel would flag.
Alerts with a named owner, severity and escalation
Unread alerts are no alerts. Every alert needs a named owner, a severity saying whether it needs attention now or can wait until morning, and an escalation path if the first person stays silent. Route to an SMS or pager call, not a shared mailbox.
Match urgency to business impact. A brief slowdown on an admin page does not deserve the wake-up that checkout errors in trading hours do, and treating them alike teaches people to mute everything.
Runbooks instead of tribal knowledge
A runbook is a short document for one known type of incident, recording what the alert means, the first three checks, who to escalate to and any safe mitigation. Speed under pressure is the value. Someone paged at 2am should not have to rebuild context that a colleague already worked out last time and left buried in a chat thread.
Write runbooks from real incidents. After each outage, add or amend the matching runbook with what the team learned, so the same fault never starts debugging from zero a second time. Hypothetical runbooks gather dust.
Tuning noisy alerts and reviewing what monitoring missed
Too many low-value alerts breed alert fatigue. That is worse than silence, because it creates false confidence that somebody is watching while the team has quietly learned to ignore the pager. Each month, export the pager log, list which alerts fired, which were acted on and which were muted, then retire the useless ones and raise thresholds on noisy ones.
Ask why after every miss. When an incident was not caught, or was caught late, treat it as a coverage gap and close that gap with a new check, rather than adding more alerts.
Monitoring-to-response responsibility map
A table listing each critical journey, naming the metric or synthetic check that watches it, the alert threshold, the on-call owner and the escalation path. It settles who responds.
Open a blank worksheet to printQuestions owners ask
What should we monitor beyond server uptime?
Monitor the transactions the business lives on: whether checkout completes, whether the order sync to accounting is moving data, and whether the overnight cron backup ran. Uptime only shows that the machine is switched on, not that the work customers and staff depend on is getting done each day.
Who responds to an alert?
The on-call owner named for that specific alert, chosen in advance and written on the responsibility map, so nobody has to decide who goes during a night incident. Each alert maps to one person or rotation, and the escalation path takes over if nobody acknowledges it within the agreed window.
How often should runbooks be updated?
Update after every incident. A runbook never tested against a real outage tends to describe a system that was replaced two releases ago, usually discovered by whoever is on call that night.