What happens when ten thousand clients retry after an outage: compare fixed, exponential, full-jitter and decorrelated backoff.

26.7.2 · L5 · Interview

Predict first

Ten thousand clients retry after a 20 second outage and the server then accepts 1,500 requests a second. Which policy finishes last?

import Foundation
struct Rng { var s: UInt64; mutating func unit() -> Double { s = s &* 6364136223846793005 &+ 1442695040888963407; return Double(s >> 11) / Double(1 << 53) } }
enum Policy: String, CaseIterable { case fixed, exponential, fullJitter, decorrelated }
let clients = 10_000, outageEnd = 20.0, capacity = 1500   // server accepts at most 1500 requests per second once healthy
func run(_ p: Policy) -> (peak: Int, total: Int, done: Double) {
    var rng = Rng(s: 99)
    var next = [Double](repeating: 0, count: clients), attempt = [Int](repeating: 0, count: clients), prev = [Double](repeating: 1, count: clients)
    var active = Set(0..<clients); var total = 0, peak = 0; var t = 0
    while !active.isEmpty && t < 600 {
        var due = active.filter { next[$0] < Double(t + 1) }.sorted()
        if t >= 5 { peak = max(peak, due.count) }; total += due.count
        var served = 0
        for c in due {
            let healthy = Double(t) >= outageEnd
            if healthy && served < capacity { served += 1; active.remove(c); continue }
            attempt[c] += 1
            let cap = min(32.0, pow(2.0, Double(attempt[c] - 1)))
            let d: Double
            switch p {
            case .fixed: d = 1
            case .exponential: d = cap
            case .fullJitter: d = rng.unit() * cap
            case .decorrelated: d = min(32, 1 + rng.unit() * (prev[c] * 3 - 1)); prev[c] = d
            }
            next[c] = Double(t) + max(d, 0.05)
        }
        due.removeAll(); t += 1
    }
    return (peak, total, Double(t))
}
for p in Policy.allCases { let r = run(p); print(p.rawValue, "peak req/s from t = 5 s", r.peak, " total requests", r.total, " all served by t =", Int(r.done)) }

Exponential without jitter (t = 224); full jitter finishes by 51 and decorrelated by 52
Fixed 1 s, because it sends the most requests
They all finish together

Mental model. A crowd at a locked door that all pushes again after exactly the same pause. Each person waiting a random extra moment lets the doorman admit them one by one.

Backoff delays a retry; jitter spreads the retries of many clients so they do not arrive together. The check simulates 10,000 clients that all fail at second 0, a server that comes back at second 20 and then accepts only 1,500 requests a second. Fixed 1-second retries are the worst: 10,000 requests every second (238,500 in total), although everyone is served by second 27 because the server is simply hammered. Exponential without jitter sends far fewer requests (88,500), but all clients keep the same schedule: they arrive in synchronised waves at 1, 3, 7, 15 and 31 seconds, and each wave of 10,000 overruns the 1,500 capacity, so the last client succeeds only at second 224. Full jitter (a random delay up to the exponential cap) spreads the waves, peaks at about 4,000 requests a second after the fifth second and finishes by second 51. Decorrelated jitter does slightly better: peak about 2,800 and the fewest requests (64,766), finished at 52.

So jitter is not decoration: without it, clients that fail together stay synchronised and retry together. The wider policy (Q324): retry only idempotent or keyed requests, cap the delay and the total attempts, honour a server Retry-After, add a circuit breaker so a long outage stops generating traffic at all (Q494), and spread scheduled work (background refreshes at a fixed minute past the hour) with random offsets. Servers help with admission control and load shedding, but the client is the part you control. The numbers are from a seeded simulation with simplified timing in one-second buckets; the ranking of the policies is the robust result, the exact figures are specific to these parameters.

Synchronised waves overrun the server again and again; jitter spreads them.

Common traps

Exponential backoff with no jitter. It looks polite but keeps the whole fleet in lockstep.
Retrying forever. During a long outage the retries themselves become the load; stop at a limit or a breaker.

They might follow up with

Full or decorrelated jitter? Both are fine; full jitter is simplest and already removes most of the synchronisation.
What about a push-triggered refresh from millions of devices? Add a random delay on the client and send the trigger in waves from the server.

What happens when ten thousand clients retry after an outage: compare fixed, exponential, full-jitter and decorrelated backoff.

Predict first

Common traps

They might follow up with

Related questions