Why Do Flaky Tests Only Fail in CI

Tests flake in CI but not locally because CI is a harsher, more variable environment than your machine, and flakiness is caused by exactly the variability CI adds. CI machines are slower and shared, they run more tests in parallel, they start from clean state, and they run under load that changes from one run to the next. A test with a hidden dependency on timing, ordering, or shared state passes reliably on your fast, quiet laptop and then flips between pass and fail in CI, because CI is where the non-determinism finally gets exercised. This post covers the specific CI conditions that produce flakiness, and how to confirm a CI-only failure is a flake rather than a real bug you should fix.

Why does CI produce flakiness that your machine does not?

The difference is not that your code behaves differently in CI. It is that CI exposes non-determinism your local environment happens to hide. Your laptop is fast, lightly loaded, and consistent from run to run, so a test that depends on something finishing quickly, or on tests running in a particular order, gets the conditions it needs every time and passes. CI removes those comfortable conditions, and it removes them inconsistently, which is what turns a latent weakness into visible flakiness.

The key word is inconsistently. A consistent difference between local and CI produces a consistent failure, a test that fails every CI run, which is a different problem. Flakiness is when the same test passes some CI runs and fails others with no code change, and that happens because CI's conditions vary between runs: how loaded the runner is, which tests happen to run alongside it, how long a response took that particular time. The test is sitting on the edge of passing, and CI's variability pushes it over the line on some runs and not others.

Which CI conditions cause it?

Slower, shared machines. A hosted runner has a fraction of your laptop's CPU and memory, shared with the browser, your application, and everything else in the job. A test that waits a fixed time, or checks for something a moment after triggering it, gets away with it locally where the thing is always ready in time. In CI the timing is tighter and varies with load, so sometimes the wait is enough and sometimes it is not.

Parallelism. CI usually runs more tests at once than you do locally, and often shards them across machines. Which tests land together changes between runs, so a test that shares state with a neighbour, the same user, record, or fixture, interferes with that neighbour only on the runs where they happen to run concurrently. Same test, different neighbours, different result.

Clean and varying state. CI starts from nothing, no warm cache, no leftover data, no browser profile. A test that quietly depended on state from an earlier run passes locally, where that state tends to persist, and behaves unpredictably in CI depending on what has and has not been set up on that particular run.

Resource contention. Memory and CPU limits that you never approach locally get hit intermittently in CI when a run is heavier than usual, producing failures that look random because they track the load rather than the code.

Every one of these has the same signature: a test on the edge, pushed over by conditions that vary run to run. That variation is why the failure is flaky rather than constant.

Is it flaky, or is it a real bug that only CI reveals?

This is the distinction that matters before you spend time on it, because the response is opposite for each. If a test fails every CI run but passes locally, that is not flakiness, it is a consistent environmental difference, a real gap between your machine and CI that you need to close (a missing environment variable, a headless rendering difference, a service that is not running). If a test fails some CI runs and passes others with no code change, that is flakiness, and the fix is stabilising the test rather than hunting a bug that is not consistently there.

The trap is treating an intermittent CI failure as a bug and changing application code to chase it, when the test is simply non-deterministic and the code is fine. You cannot tell the two apart from the one run you happen to be looking at, because a single CI failure looks identical whether it is consistent or intermittent. The evidence that separates them is how the test has behaved across many CI runs: consistently failing points at a real difference, alternating points at a flake.

How do you confirm a CI-only failure is flaky?

You confirm it from the run history, not from the failing run in front of you. That means retaining the result of every CI run so each test carries a record of how it has behaved, rather than looking at each run in isolation and guessing. Tesults keeps every run pushed to it, so a CI failure can be read in the context of that test's recent history. When a test alternates between pass and fail more than twice across the runs analysed, Tesults flags it as flaky and collects it in a dedicated flaky section in the supplemental analysis view, marked with a snowflake symbol, so a test that is flaky in CI is surfaced as flaky rather than looking like any other failure.

That turns the question from a guess into a reading. Instead of wondering whether this CI failure is real, you can see that the test has failed four times in the last forty CI runs, always with the run passing on either side, which is the signature of a flake and tells you to stabilise the test rather than change the code. A test that instead started failing on every run from a specific build is telling you the opposite, that something real changed. The history is what makes the difference legible. The broader approach to finding and handling these is covered in how to detect and handle flaky tests, and this is closely related to why a test passing on retry is usually flaky rather than fixed.

Flaky tests concentrate in CI because CI is where the variability that causes flakiness actually lives, slower shared machines, shifting parallelism, clean and changing state. A test on the edge of passing gets pushed over on some runs and not others, which is flakiness by definition. Before you treat a CI-only failure as a bug, check whether it fails consistently or intermittently across runs, because only the run-to-run history can tell you which one you are looking at, and only one of them is a bug. The detection views described here are documented under supplemental analysis.