How to Cut Your Test Suite Runtime by 70%

A standup in March, payments company, about thirty engineers. Someone said they'd merge the invoice-copy change and "let CI catch up." Nobody pushed back. The suite took 74 minutes. By the time it finished, three newer commits were already on the branch, so the result described a tree that no longer existed.

We got the pull-request path down to 19 minutes over five weeks. I am not going to pretend that was a framework swap. It was a cache key, a smaller job on the PR, and a willingness to stop rerunning tests we already knew were junk.

Most of the time wasn't tests

We timed one worker with a stopwatch because the CI UI rounded everything into "install" and "test" in a way that hid the truth. Eleven minutes of setup. Nine minutes of specs. Eight workers each did their own npm ci and their own Postgres image pull. The cache key included a timestamp a contractor had left in the Dockerfile in 2024, with a comment that said "debug". Every build was a cache miss. Every build.

When the key became the lockfile hash plus the base image digest, warm setup dropped to about 90 seconds. That's the part people skip because it doesn't feel like a QA project. It isn't. If the machine spends longer preparing than asserting, you don't have a slow test suite. You have a build that refuses to remember yesterday.

A pull request is not a release

On a PR we started running two things: specs that imported a file in the diff, and a smoke list nobody was allowed to bikeshed. Login. Create an invoice. Pay it. Refund it. The full pack still ran on main, and again on the release branch.

The first week felt loose. A class rename skipped a visual spec we hadn't mapped. Main went red 40 minutes later, before prod. We added that spec to the smoke list. Two misses in five weeks, both caught on main. The alternative we'd been living with was a 74-minute job people didn't wait for, which is a polite way of skipping every test.

If your impact map is wrong you'll skip something that mattered. Own that. Then compare it to a suite that finishes after the author has context-switched into a different ticket. I'll take the imperfect map.

Quarantine with a date, or don't bother

Eleven tests failed more than twice a week with no product change behind them. One was a date-picker spec racing an animation. The reruns were what made the bad days ugly, not the median. We pulled those eleven off the blocking job and onto a nightly. Each one got a name and a 14-day clock. Two got fixed. Three got deleted because they asserted a tooltip product had killed in January. The rest were sleep() with a paycheck.

Quarantine without an owner is a drawer. We put the nightly failure count on the same dashboard as the PR job so it couldn't become folklore.

The slow shard is the runtime

Eight workers, and one of them always finished last because it owned the entire billing folder. Billing specs averaged around 40 seconds. The other workers were done in six minutes and idle. We split by recorded duration, not by directory. The slowest shard went from 22 minutes to 9. Wall clock is that shard. Average is a number you use to feel better.

A few specs wrote the same customer row and couldn't share a database. They got their own shard. Calling that "parallelism" when they stomp on each other is just a slower flake.

Fail the likely stuff first

We sorted the PR job by how often each spec had failed in the past 30 days. A query, not a platform. The checkout spec that had failed nine times that month ran before the settings-page spec that hadn't failed since winter. People found out they were wrong around minute four instead of minute eighteen. Total time barely moved. The time until a useful red did, and that's the wait that makes someone click "merge anyway."

What the 70% actually was

74 minutes to 19 on the pull-request path. Main still sat around 28 once caching and the shard split landed, because main runs the whole pack. I won't quote 19 as if we threw regression away. We stopped charging every copy tweak for it.

If your suite is slow, time one job before you rewrite the runner. You'll probably find a cache key that can't hit, a shard hoarding one folder, and a handful of tests you rerun because deleting them feels like a moral event. Fix those. Then talk about hardware.