Why do tests flake, how do you find and fix them, and what makes a CI test run hermetic?

22.7.1 · L5 · Interview

Predict first

Two tests share a global. Run in the order “fill the cache, read the cache”, then in the reverse order. What happens?

var shared = 0

struct T { let name: String; let run: () -> Bool }
let setUp = T(name: "fills the cache") { shared = 1; return true }
let uses = T(name: "reads the cache") { shared == 1 }

func suite(_ order: [T]) -> String {
    shared = 0
    return order.map { "\($0.name): \($0.run() ? "pass" : "fail")" }.joined(separator: ", ")
}
print(suite([setUp, uses]))
print(suite([uses, setUp]))

pass pass in the first order; the reader fails in the reverse order
both orders pass
both orders fail

Predict first

Which xcodebuild options for repeating and retrying tests exist?

// === dummy ===

-test-iterations, -retry-tests-on-failure and -run-tests-until-failure all exist
only -retry-tests-on-failure exists
none: you must script repeats yourself

Mental model. A smoke alarm that sometimes shrieks for toast is worse than none: people unplug it, and then the real fire is not noticed.

A flaky test passes and fails on the same code. The causes are usually one of a few: time (real sleeps, timeouts too tight, dates and time zones), ordering and shared state (a test relies on a global, a singleton, UserDefaults or a file that another test sets up, so the result depends on the order: the check below shows the same two tests passing in one order and failing in the other), concurrency races (assertions before an async result arrives), external dependencies (network, a shared server, the keychain), environment (simulator state, leftover app data, OS or device differences, a slow shared CI machine), and UI timing (animations, taps before an element exists). Find them with data, not guesses: run a test many times and in random order to expose it (xcodebuild has -test-iterations, -run-tests-until-failure and -retry-tests-on-failure, and test plans can randomise order and repeat), track failure rates per test over time, and bisect by running the failing test alone and with its neighbours.

Then fix the cause. Make each test hermetic: it creates what it needs and leaves nothing behind, uses its own temporary directory and an isolated defaults suite, gets time, randomness and network as inputs (see the determinism question), and does not depend on other tests. On CI pin the Xcode version, simulator model and OS, boot a clean simulator (or erase it) per run, avoid a shared mutable machine state, and use build-for-testing then test-without-building so the build is separated from the run. Retries (-retry-tests-on-failure) are a diagnostic and a stop-gap, not a fix: they keep the build green while hiding a flaky test, so surface every retry that was needed and quarantine repeat offenders (move them to a non-blocking job with an owner and a deadline) instead of ignoring them. A red build that “usually passes on rerun” trains everyone to rerun, and then real failures slip through.

The reading test only passes because another test ran first. That hidden coupling is why you run in random order.

Xcode can repeat tests a set number of times, retry only the failures, or run until one fails, which are the tools for finding flakes.

Common traps

Retrying until green without recording it. The flake stays in the suite and its failure rate quietly grows.
Tests that need another test to run first. They pass in order and fail when run alone, in parallel or shuffled.

They might follow up with

How do you find an order-dependent test? Run in random order and in isolation, and compare the results.
What is quarantine? Moving a flaky test out of the blocking path, tracked with an owner and a fix date, so it does not erode trust or get forgotten.

Why do tests flake, how do you find and fix them, and what makes a CI test run hermetic?

Predict first

Predict first

Common traps

They might follow up with

Related questions