Why Your CI/CD Pipeline is Lying to You

The build was green for eleven days. Not "green if you reran it." Green. We shipped a tax change on day three of that streak and felt fine about it. On day twelve a bookkeeper wrote in and said the receipt was a rupee off the filing export. The test that was supposed to catch that had been edited six weeks earlier. Someone changed the assertion from the exact paise amount to toBeTruthy() because a fixture was annoying and the release was that afternoon. The pipeline did its job. It ran the test we left it.

Passing means the script didn't throw

That's the whole contract. The scenarios you wrote didn't fail. It does not mean you wrote the scenarios that matter. It does not mean the mock still looks like the partner's API. It does not mean two features, each correct alone, still behave when a user does them in the order a tired person does them on a phone. The bugs I remember didn't hide in a red job. They hid in a green one that had slowly agreed to less.

Three lies I keep seeing

The first is the flake you've made friends with. A spec fails, you rerun, it passes, you go to lunch. After a month of that, red stops meaning "stop." It means "spin again." Then a real failure gets the same treatment and ships. The pipeline isn't neutral at that point. It's training the team.

The second is a test that passes while checking the wrong thing. The tax spec is the clean example, but the usual one is a mock. The client starts returning amounts as strings. The mock still returns numbers. The test is faithful to the mock and a stranger to production. Or a selector gets a fallback so wide it matches the footer and the test "finds" the button. Green. Useless.

The third is the coverage percentage in the README. Ninety percent means lines ran. It doesn't mean the assertion at the end of those lines could fail for a reason you'd care about. I've watched a file sit at 100% coverage with assertions that checked the function returned something. Something. That's not a safety net. That's a participation trophy the build hangs on the wall.

Why we keep trusting it

Because a lying pipeline is still a pipeline, and the alternative in a lot of companies is a spreadsheet and a prayer. Because fixing the tax assertion is nobody's OKR. Because when the badge is green, the incident feels like it belongs to whoever was on call, not to the job that smiled at you. Trust leaks slowly. Engineers stop blocking on red. QA starts clicking the same flows by hand "just in case," and the release gets longer because everyone is nervous and nobody can say why, since the build is green.

What I'd demand before I believe a badge

A red build has to be rarer and sharper. If a spec flakes, it doesn't get a quiet rerun as policy. It gets a ticket with a name, or it leaves the blocking path. We did that with four specs that failed on shared staging data. After they were gone, people argued about reds again. That argument is the product.

Assertions have to be allowed to be specific. The toBeTruthy() edit should have been a review comment, not a favor. If the fixture is painful, fix the fixture. Weakening the check to save an afternoon is how you buy the bookkeeper's email.

And once a quarter, pick a green test and try to break the feature without the test going red. If you can, the test is decoration. We did this with the tax spec after the incident. The broken version was still green. That was the most useful hour the team spent that month.

Don't ask the badge to be a conscience

Your pipeline isn't scheming. It's running what you gave it, including the shortcut from six weeks ago. Green means the script was satisfied. If you need it to mean "a customer can file this receipt," you have to put that sentence in the test and defend it when it gets in the way of a release. Otherwise the badge will keep telling you a comforting version of the truth, and the bookkeeper will send the other one.