How do you design a client for reliability: deadlines, circuit breakers, degradation tiers, bulkheads, kill switches and the signals that tell you it works?

26.2.1 · L5 · Interview

Predict first

A breaker opens after three failures with a ten-second cooldown. What do the states and allowed calls look like?

struct Breaker {
    enum State: String { case closed, open, halfOpen }
    var state = State.closed
    var failures = 0
    var openedAt = 0.0
    let threshold = 3, cooldown = 10.0

    mutating func allow(now: Double) -> Bool {
        if state == .open, now - openedAt >= cooldown { state = .halfOpen }
        return state != .open
    }
    mutating func record(success: Bool, now: Double) {
        if success { failures = 0; state = .closed; return }
        failures += 1
        if state == .halfOpen || failures >= threshold { state = .open; openedAt = now }
    }
}

var breaker = Breaker()
for time in [0.0, 1.0, 2.0] { if breaker.allow(now: time) { breaker.record(success: false, now: time) } }
print("after 3 failures:", breaker.state.rawValue)
print("call at t=5 allowed:", breaker.allow(now: 5))
print("call at t=13 allowed:", breaker.allow(now: 13), "state", breaker.state.rawValue)
breaker.record(success: true, now: 13)
print("after a success:", breaker.state.rawValue)

open after 3 failures; t=5 not allowed; t=13 allowed in half-open; closed after a success
It stays closed forever
It reopens at t=5

Mental model. A ship has watertight compartments, a bilge pump and lifeboats. Reliability is not preventing every leak; it is making sure one leak does not sink the ship.

Reliability on a phone means the app stays useful while the network, the backend and its dependencies misbehave. Start with deadlines: give each user action a total time budget and make every layer’s timeout shorter than the one above, so a slow dependency cannot hold the screen hostage (timeoutIntervalForRequest for stalls and timeoutIntervalForResource for the whole transfer, Q314). Retries (Q324) handle brief blips; they must be bounded, jittered and idempotent, otherwise an outage becomes a retry storm. A circuit breaker protects you and the backend when failures persist: after N consecutive failures it opens and calls fail fast or use a fallback; after a cooldown it goes half-open, lets one trial through, and closes on success or opens again on failure. The check below walks that state machine with a clock you control (Q445). Add bulkheads: separate queues or sessions for critical traffic (checkout, auth) and background traffic (analytics, prefetch), so a flood in one cannot starve the other, and shed load deliberately, dropping or sampling analytics first.

Plan degradation tiers per feature: full experience, then cached or stale data with a visible “last updated”, then a reduced mode, then a clear error with retry, and never a blank spinner. Kill switches (remote flags, Q486) let you turn off an expensive or failing feature, switch a provider or lower a quality tier within minutes, so they should exist for every dependency you do not control (a payment provider, an ad SDK, a recommendation service), and the app’s behaviour with the flag service itself unreachable must be defined (safe defaults, Q486). Handle partial failure in lists and screens independently, so one failed widget does not blank the page, and make state restoration cheap so a kill or crash mid-flow resumes. Measure with journey-level signals, not only crashes: success rate and latency of the critical user journeys (sign in, send, pay), reconnect rate, time to first content, crash-free sessions and hang rate (Q432), each with a target (an SLO) and an alert, and review them in incidents (see the incident question). Test failure on purpose: network conditioner, fault injection in stubs, and dependency-down drills (Q464).

Fail fast while open, trial after the cooldown, and close only when a trial succeeds.

Common traps

Unbounded retries with no breaker. When the backend is struggling, every client piles on and keeps it down.
One shared queue or session for everything. A burst of low-priority traffic delays the payment call.

They might follow up with

What does a half-open breaker do? Lets a single trial through and closes if it succeeds, or opens again if it fails.
How do you decide a timeout? From the user’s patience and the journey’s deadline, with each inner layer shorter than the outer one.

How do you design a client for reliability: deadlines, circuit breakers, degradation tiers, bulkheads, kill switches and the signals that tell you it works?

Predict first

Common traps

They might follow up with

Related questions