Why Do Tests Pass on Retry but Fail the First Time

A test that fails the first time and passes on retry is almost always flaky, not fixed. The retry did not correct anything in your code or your test; it just rolled the dice again and got a different result, which is the definition of a flaky test. The reason this matters is that automatic retries, the ones built into most test runners, quietly convert that flakiness into a green result, so the run looks clean while the underlying instability is still there. This post covers what a pass-on-retry is actually telling you, why retries hide the problem instead of solving it, and how to see the pattern that a single retried run conceals.

What does passing on retry actually mean?

A test has one job: given the same code, produce the same result. When a test fails and then passes on an immediate retry, with nothing changed in between, it has failed that job. The result depended on something other than the code under test, timing, ordering, a race, a slow response, available resources, and the retry simply caught a luckier roll. That is not the test being wrong once and right the second time. It is the test being non-deterministic, which is exactly what a flaky test is.

This is worth stating plainly because the retry creates a comforting story: it failed, I ran it again, it passed, so it is fine now. It is not fine. Nothing was fixed. The same test, run a third or fourth time, could fail again. A pass-on-retry is not a resolution, it is a symptom, and the symptom is flakiness.

Why do automatic retries hide flakiness instead of fixing it?

Most test runners can retry a failed test automatically and report the final outcome. Configure two retries, and a test that fails once then passes is reported as a pass. On the surface this is helpful, it keeps a flaky test from breaking your build over a failure that is not a real regression. Underneath, it is hiding information you need.

The problem is that the run-level result now says pass, and the fact that it took two attempts to get there is buried or discarded. Your CI dashboard is green. Your pass rate looks healthy. And the flaky test sits there untouched, because nothing surfaced it as a problem. Retries are a reasonable tactic for not letting flakiness block a deploy, but used alone they are a way of not looking at the flakiness, and a flaky test you are not looking at does not get fixed. It accumulates, alongside all the others being silently retried into green, until the day a genuine failure hides among them and the retries paper over that too.

So retries and flaky detection are not the same thing and should not be confused. A retry keeps the build moving. Detection is what stops the flaky test from disappearing. You want both: retry so a flake does not block you, and detection so it does not vanish.

Why can't a single run tell you a retried test is flaky?

Even without automatic retries, one run is not enough to identify flakiness, because flakiness is by definition variation across runs, and a single run shows no variation. With retries in the picture it is worse: the one run you are looking at reports a clean pass, so there is not even a failure visible to prompt suspicion. The evidence that this test is unreliable does not exist in that run at all. It exists only in the comparison across many runs, where the same test can be seen passing, failing, passing, failing, regardless of what the code did.

That is why flakiness that hides behind retries is so persistent. The information needed to catch it, the run-to-run history, is exactly the information a single green run throws away. To see it you have to retain the result of every run and look at how each test behaved over time, rather than trusting the final status of the run in front of you.

How do you see the pattern across runs?

The fix is to keep the history and let something compute across it rather than relying on any one run's status. Tesults retains every run pushed to it, so each test accumulates a record of how it behaved over time. Against that history, a test is flagged as flaky when its result alternates between pass and fail more than twice across the runs analysed, a deterministic reading of the record rather than a judgement about any single run. Flagged tests are collected in a dedicated flaky section in the supplemental analysis view and marked with a snowflake symbol wherever they appear, so a test that has been alternating is visible as flaky even when the latest run looks fine.

For a specific test you are unsure about, its stability can be examined over recent runs and returned with a classification and the number of runs analysed, so you can tell the difference between a test that genuinely stabilised and one that just happened to pass the last couple of times. That is the distinction a pass-on-retry hides and that cross-run history restores: not what happened in this run, but what has been happening across all of them. The broader approach to finding and handling these is covered in how to detect and handle flaky tests.

What should you do when a test passes on retry?

Treat it as a flaky test to track, not an incident that resolved itself. Let the retry do its job of keeping the build moving, but record that the test needed a retry, so it enters a list of known-flaky tests rather than vanishing into a green run. From there the work is the same as for any flaky test: quarantine it from anything that gates a deploy so a known flake does not block you, investigate the cause, timing, ordering, shared state, fix the non-determinism, and confirm from the history that it has genuinely stabilised rather than just passed a few times by chance. The point is that a pass-on-retry should add the test to a backlog, not remove it from your attention.

A test passing on retry is not a test that fixed itself, it is a flaky test caught in the act, and automatic retries, useful as they are for keeping a build moving, will hide that fact unless something is retaining the history and surfacing the pattern. Keep the run-to-run record, let flaky detection read across it, and a pass-on-retry becomes what it should be, a flagged test to stabilise rather than a green result you were relieved to see. The detection views described here are documented under supplemental analysis.