How do you decide whether a phased release is healthy: why is a crash-free percentage on a small sample misleading, and how does a confidence interval help?

25.8.4 · L5 · Interview

Predict first

The Wilson interval for the crash rate is computed for three samples and used against a 2% budget. What does it print?

import Foundation

func wilson(crashed: Int, users: Int, z: Double = 1.96) -> (low: Double, high: Double) {
    let n = Double(users), p = Double(crashed) / n, z2 = z * z
    let centre = (p + z2 / (2 * n)) / (1 + z2 / n)
    let half = z * (p * (1 - p) / n + z2 / (4 * n * n)).squareRoot() / (1 + z2 / n)
    return (centre - half, centre + half)
}
func pct(_ x: Double) -> String { String(format: "%.2f%%", x * 100) }

for (crashed, users) in [(0, 50), (4, 200), (400, 20_000)] {
    let w = wilson(crashed: crashed, users: users)
    print("\(crashed)/\(users) crashed: crash-free \(pct(1 - Double(crashed) / Double(users))), crash rate 95% range \(pct(w.low))-\(pct(w.high))")
}
let budget = 0.02
for (crashed, users) in [(5, 200), (500, 20_000), (300, 20_000)] {
    let w = wilson(crashed: crashed, users: users)
    print("\(users) users:", w.low > budget ? "halt (clearly over budget)" : w.high < budget ? "proceed (clearly under budget)" : "keep waiting")
}

Ranges 0.00-7.14%, 0.78-5.03%, 1.81-2.20%; then keep waiting, halt and proceed for the three decision cases
The same interval for 200 and 20,000 users
Halt for the 200-user case

Mental model. Judging a restaurant from two reviews: a perfect score, but you would not bet dinner on it. With two thousand reviews the same average means something.

A phased release exposes a small fraction of users first, so the early numbers come from few users. A crash-free rate of 98 percent is the same figure for 4 crashed out of 200 users and 400 out of 20,000, but the evidence is very different, and a gate that compares it to a threshold treats them alike. Put a confidence interval around the crash rate (the Wilson interval is a good default for proportions, since it behaves sensibly near zero and for small samples). The check prints the 95 percent range: 0 crashes among 50 users is still consistent with up to a 7.14 percent crash rate; 4 of 200 spans 0.78 to 5.03 percent; 400 of 20,000 narrows to 1.81 to 2.20 percent. A decision rule that uses the interval has three outcomes instead of two: halt when even the low end exceeds your budget, proceed when even the high end is under it, and keep waiting otherwise. With a 2 percent budget the check halts for 500 of 20,000 (2.29 to 2.73 percent), proceeds for 300 of 20,000, and waits for 5 of 200.

Practical points: count users or sessions consistently (the denominator must match the numerator), compare against the previous version over the same period instead of an absolute number, split by OS version and device because one cohort can hide inside the total, watch other signals as well (hangs, launch time, errors in the backend, support contacts), and let a kill switch or a server-side change do the fast response while you wait for the store (Q485). Phased release in App Store Connect rolls out over seven days at fixed percentages and can be paused; do not assume the percentages equal your own user counts. Treat the interval as a guide for the decision, not a guarantee: crashes are not independent when many come from one device or user, and the math was run here on invented numbers, not on a real rollout.

More users narrow the interval, so the same percentage supports a firmer decision.

Common traps

Halting or approving on a single percentage from a 1 percent rollout. The sample is too small to separate noise from a real regression.
Using a different denominator than the numerator. Crashed sessions divided by users gives a number that means nothing.

They might follow up with

Why not the simple normal approximation? It gives intervals below zero or too narrow when the rate is close to 0 and the sample is small.
Can the gate be automated? Yes; compute it from the crash reporter’s counts and block the next phase in the pipeline (Q432, Q462).

How do you decide whether a phased release is healthy: why is a crash-free percentage on a small sample misleading, and how does a confidence interval help?

Predict first

Common traps

They might follow up with

Related questions