In this guide 6 sections
An average blends fast and slow requests into one comfortable number. Percentiles split them apart. The p95 is the time that 95 of every 100 requests beat, so it shows what your slowest users feel. Alert on p95 or p99, and judge changes with the same load.
Why the average looks fine while users complain
Take ten requests. Nine finish in a quarter of a second and one takes twelve seconds. The average is under a second and a half, and the dashboard stays green. The person who waited twelve seconds is already on the phone to your support desk.
Slow requests are not rare accidents. They cluster around specific causes: a cold cache after a deploy, a lock held by a nightly job, a garbage collection pause, a customer with ten thousand orders. The average dilutes all of them.
Heavy users also hit the tail more often, because each page load makes many calls.
How to read p50, p95 and p99
Sort every request by duration. The p50, or median, is the middle one: half were faster. The p95 sits near the slow end, and the p99 is closer still. A report showing p50 at 200 milliseconds and p99 at nine seconds describes two very different services.
Look at the gap between them. A narrow gap means consistent behaviour. A wide gap means something intermittent, such as a lock wait, a retry or a slow third party API.
Maximum values are noisy. One stuck request sets the max, so use p99 for the extreme end.
- p50 shows the typical experience
- p95 shows what one user in twenty meets
- p99 exposes locks, retries and timeouts
- Max is mostly noise from a single stuck call
Choose the percentile for your alert
Alert on p95 for pages people use all day, such as order entry or a dispatch board. Choose p99 for payment calls and anything with a hard timeout, since one failed request there costs a sale. Keep the average on a chart for context only.
Set the threshold from a baseline. Observe a normal fortnight, note the usual p95 for each endpoint and alert when it stays above that for several minutes. A single spike should not wake anyone.
Alert per endpoint. A fast login page can mask a slow report export when everything is pooled into one figure.
- Separate thresholds for each endpoint
- A duration condition, not a single sample
- Error rate beside latency
- A named owner for every alert
Compare before and after without fooling yourself
A fix is only proven if the comparison is fair. Use the same endpoint, the same hours of the week and similar traffic. Comparing a quiet Sunday with a busy Monday proves nothing.
Percentiles cannot be averaged across servers or days. Taking the mean of three p95 values gives a number with no meaning. Store the raw histogram buckets, or compute the percentile over the combined data.
Check the sample size too. A p99 drawn from forty requests is guesswork, so widen the window until you have thousands.
Put it into your next monitoring review
Ask your team for p50, p95 and p99 per endpoint, over the last thirty days, in one table. Highlight any row where p99 is more than ten times the p50. Those are your candidates for a closer look.
Then trace one slow request from end to end. Find where its time went: database, queue, outside API or rendering. The answer tells you which fix to buy.
Start this week with your three busiest endpoints.
Latency review table
A one-page table listing every endpoint on its own line. Columns are request count, p50, p95, p99, p99 to p50 ratio, alert percentile chosen, threshold, owner and last reviewed date. A second block holds the before and after comparison: date range, traffic level, sample size and result.
Open a blank worksheet to printQuestions owners ask
What is a good p95 response time?
It depends on the task. A screen a clerk uses all day should feel instant, while a large report can take longer. Set targets with the people who use each page.
Is p99 better than p95?
Neither is better. Use p95 for everyday pages and p99 where one failure is costly, such as payments or a request with a hard timeout.
Can I average p95 across servers?
No. Percentiles do not combine that way. Merge the raw measurements or histogram buckets first, then calculate the percentile on the full set.
Why is the average still reported?
It is easy to compute and useful for trends in total load. Keep it as context, but never as the only measure of user experience.