Application slow only at peak hours: what that pattern points to

A system that is quick in the morning has no basic fault, because its code and query plans are acceptable at low load. Something shared runs out when more people arrive. Your task is to find what.

That shared thing might be database connections, row locks, memory, a disk queue or a single worker. Find which limit is hit first.

Write down when the slowness starts and stops, to the minute, for a full week of working days. A start at exactly noon points to a scheduled task. A gradual slide through the morning points to a pool, a cache or memory.

When one transaction holds a lock on a row, every other user who needs that row must wait. A long invoice posting, or a stock update inside a transaction that was left open, can freeze a screen for every cashier at the counter.

In PostgreSQL, query pg_stat_activity and look for sessions waiting on a Lock event. In MySQL, read the InnoDB status. Note which table keeps showing up.

The usual cure is a shorter transaction. Do the slow work, such as a PDF or an email, after the commit.

Applications borrow database connections from a small pool. If the pool holds twenty and twenty requests are slow, the twenty-first waits for a free connection. The page seems dead while the server shows little load.

Check the pool metrics: connections in use, requests waiting, time spent waiting. If waiting climbs while the database itself sits idle, the pool is your bottleneck.

Raising the pool size can backfire. The database then fights over its own memory. Ask first why connections are held so long, because a leak, where code never hands a connection back after an error, is a common cause.

Read the scheduler. A report email, a sync with the accounts package or a backup that starts at twelve will compete with the lunchtime rush for the same database. Moving it to two in the morning costs nothing.

In a multi-tenant product, one large customer can slow everyone else, usually through a report or an import that scans a huge table. Group the slow query log by tenant id. If one account fills most of the lines, you have your culprit.

A rate limit, a separate queue for that tenant, or its own database may fix it. All three cost less than doubling the whole fleet.

Set up basic monitoring before touching the system. You need response time per page, pool use, lock waits, CPU and memory, drawn on one shared timeline with the busy hour marked. Cause and effect then line up.

Change one thing at a time. Watch the next peak. Three changes at once teach you nothing about which worked.

Consider a bigger server only when the evidence points at raw CPU or memory. By then you can say how much bigger it must be, and why.