Flaky Tests Are a Tax on Every Deploy. Here's What Paying It Actually Costs You

There's a spec on a team of about forty that fails when the build agent is busy. Not always. Often enough that the Slack reaction is a rerun emoji. Last quarter I asked them to count it instead of joking about it. Nineteen reruns in ten working days, just for that one spec. Each rerun is a few minutes of CI. The expensive part was the ten minutes before the rerun, while someone decided whether the failure was the date picker or their actual change. Nineteen times ten minutes is a short work week, gone, and that's one test.

The tax is the doubt

People price flakes as machine time. Machine time is the cheap line. The costly line is the moment "CI is red" stops meaning "stop typing." Once the team believes red is probably noise, a real red gets the emoji too. I watched that happen with a permissions check. It failed for a reason. It got rerun twice, passed on the third try because the fixture raced, and the hole shipped. The flake didn't just waste the afternoon. It laundered a bug.

At forty people you can still see it. At a bigger org it becomes culture, and culture is hard to put in a spreadsheet. New hires learn the emoji before they learn the product.

The suite rots because the product moves

Most of these tests didn't start flaky. A selector changed. A modal grew a step. An API field flipped from a number to a string and the test still parsed the old shape on Tuesdays when the stub won the race. The product moved and the test stayed, so the test stopped describing behavior and started describing a memory of the UI. Keeping up by hand means every change owes the suite a visit. That visit loses to the feature. It always does. Then you "stabilize" in a sprint that gets cut, and the emoji stays.

Deleting the test feels like giving up coverage. Leaving it in the blocking job taxes every deploy. Both feel bad. Only one of them is honest. If a test doesn't fail when the behavior breaks, it isn't coverage. It's a toll booth.

What we did instead of a motivation speech

We took any spec that failed twice in a week without a product change off the merge path. Not into a void. Onto a list with an owner and a date. Fourteen days. Fix it, rewrite it, or delete it in the review. The date-picker spec was a sleep waiting out an animation. We waited on the element. The failure stopped. Another spec asserted a tooltip the designer had removed. We deleted it in the same PR as the admission that we were afraid to. The merge job got quieter in a way people noticed by Wednesday, because red meant their diff again.

The maintenance didn't vanish. It got a boundary. The person on call for flakes wasn't also expected to invent coverage for the new checkout. Those were separate hours. Mixing them is how the flake list becomes permanent.

A red build has to be allowed to hurt

If you can merge through a failure because "it's probably that one," you don't have a signal. You have a suggestion. I'd rather run fewer tests that the team believes than a thousand the team has learned to negotiate with. Belief is the whole point of the job. When it goes, QA is a department that produces badges, and engineering ships on vibes with extra steps.

I don't buy the idea that this is a discipline problem, as in "if people cared they'd update the tests." They care. They're also late for a demo. A system that relies on guilt will lose to the demo every week. The list with a date works because it doesn't ask anyone to be a hero. It asks the merge job to stop pretending a known-bad spec is information.

Count the reruns for a week before you buy a new runner. If one spec shows up more than twice, it doesn't get to sit on the merge button while you "get to it." Give it a name and a deadline. The tax is already being collected. You might as well see the bill.