In this guide 6 sections
Slow builds and flaky builds are two separate problems. Slowness comes from cold dependency installs, missing caches, serial test runs and heavy images. Flakiness comes from shared test data, timing assumptions and live network calls. Time every stage from the pipeline logs, rerun failed jobs on the same commit, then change one thing.
Split the complaint into two problems
Developers say the pipeline is bad, but they mean two things. One group waits too long for a green tick, while another sees red on code that did not change. These need different fixes.
Start with the history your CI tool already keeps, because GitHub Actions, GitLab CI and Jenkins all store run durations and outcomes for every job. Export a few weeks of runs for the main branch into a sheet and sort them by duration and by result.
A slow build is consistent: the same job takes a long time on every run. A flaky build is erratic. It passes on a retry, with no code change at all. Mixing the two leads teams to buy bigger runners for a problem that bigger runners cannot touch, which wastes the budget and leaves the red builds exactly as red.
- Median and worst duration per job
- Share of runs that passed only after a rerun
- Which test files fail on rerun
- Queue time before a runner picks the job up
- Branches or times of day where failures cluster
Time each stage before touching anything
Open the log of a slow run and read the timestamps rather than the output, because the gaps between lines are where the minutes go. Note when checkout ended, when the dependency install ended, when compilation ended and when the tests began, then subtract each from the next to get stage durations. Four numbers usually point at one dominant stage.
A cold install is the usual culprit. If npm ci, pip install or a Maven resolve downloads the same packages on every run, the cache is missing or its key changes on each commit. A cache keyed on the lockfile hash, such as package-lock.json or poetry.lock, fixes that in an afternoon and trims minutes from every run on every branch.
Queue time is easy to miss. A build that waits ten minutes for a free runner looks slow but is really short of capacity.
- Checkout and submodule fetch
- Dependency install and cache hit or miss
- Compile or bundle step
- Docker image build and push
- Test run, split by suite
Why tests run slowly even on good hardware
Many suites run every test file one after another on a single process, so unit tests, API tests and browser tests all queue behind each other. Splitting them into parallel jobs, or sharding one large suite across four runners, cuts wall time without touching the code.
Oversized images hurt too. A Dockerfile that copies the whole repository before installing packages invalidates its layer cache on every edit, so each small commit rebuilds every dependency from nothing. Ordering the steps so that the package manifest is copied first keeps the heavy layers cached, and only the application code layer rebuilds on a normal commit.
Check what the pipeline does on a pull request that nobody needs: full browser suites, security scans and image pushes can often wait for the merge to main.
What makes a test flaky
Rerun a failed job three times on the same commit, and if it passes, you have a flaky test whose file name appears in the failing log. Collect those names in a shared sheet for a week and a short list of repeat offenders usually appears.
The usual causes are few. Tests share a database and one leaves rows behind. A test sleeps for two seconds and hopes the queue worker is done, while another calls a live payment sandbox that times out now and then. Another depends on the system clock, the time zone or the order in which files run, so it passes on a laptop and fails on the shared runner.
Quarantine the worst offenders into a separate job that does not block merges, and give each one an owner and a review date. A skipped test with no owner is just a hidden bug.
- Shared database rows or fixtures
- Fixed sleeps instead of waiting for a condition
- Live calls to external APIs
- Dates, time zones and random seeds
- Tests that pass only in a certain order
Change one thing and measure again
Pick the stage that costs the most minutes, fix it and compare the next twenty runs against the old median. Do not stack five changes in one pull request. You will not learn which one worked.
Write the before and after numbers into the pipeline repository README, so that next quarter, when someone adds a slow step, the baseline is there to compare against.
This week, export the last fifty runs of your main pipeline and mark each as slow, flaky or healthy. That one sheet tells you where to begin.
CI diagnosis worksheet
Two tables share a single page. The first lists each pipeline stage with columns for median minutes, worst minutes, cache hit or miss and owner. The second lists failing tests with columns for file name, rerun result, suspected cause (shared data, timing, network call, clock) and fix status.
Open a blank worksheet to printQuestions owners ask
Should we just buy bigger CI runners?
Only after you know where the time goes. Larger runners help compile and test steps limited by CPU, but they do nothing for a missing dependency cache, long queue times or tests that fail on timing.
How many reruns prove a test is flaky?
Three reruns of the same commit is a reasonable start. A test that fails once and then passes twice, with no code change, is flaky, so keep a tally across several weeks to catch the rarer ones.
Is it safe to retry failed tests automatically?
Automatic retries hide flakiness and let it grow. Use them as a short bridge while you fix the cause, and report every retried test so the list stays visible.
Who should own the pipeline?
One named person or small group, usually in the platform or DevOps role, who can approve new steps and retire old ones. Without an owner, every team adds steps and nobody removes them.