How do you design analytics and experimentation that respect privacy: event schemas, consent, minimisation, stable assignment, exposure logging and sample ratio checks?

26.2.3 · L5 · Interview

Predict first

A 50/50 test shows 5000 versus 5100 users in one run and 5000 versus 5400 in another. Which should be investigated?

func chiSquare(_ a: Double, _ b: Double) -> Double {
    let expected = (a + b) / 2
    return (a - expected) * (a - expected) / expected + (b - expected) * (b - expected) / expected
}
let fine = chiSquare(5000, 5100)
let broken = chiSquare(5000, 5400)
print(String(format: "%.2f %.2f", fine, broken))
print("5000 vs 5100 is a plausible 50/50 split:", fine < 3.84)
print("5000 vs 5400 signals an assignment bug:", broken > 3.84)

5000 vs 5400 (chi-square 15.38, above 3.84)
5000 vs 5100 (chi-square 0.99)
Neither

Mental model. Good analytics is a doctor’s chart: record the vital signs that guide treatment, not everything about the patient’s life, and make sure the thermometer works before trusting the reading.

Measurement is a product feature with a cost, and the design goal is the least data that answers the question. Define an event schema (name, version, typed properties, owner, purpose) in a registry reviewed like an API, and generate the client code from it so events are consistent and rename-safe. Collect behaviour needed to run the product (funnel steps, errors, latency), avoid personal data and free-text, use a random installation or session identifier rather than a persistent cross-app one, aggregate or sample where you can, set retention limits, and honour opt-outs. The platform rules matter: tracking users across other companies’ apps and sites needs App Tracking Transparency consent (Q346), the App Privacy labels and privacy manifests must match what the app and its SDKs do (Q345, Q484), and third-party analytics SDKs are your responsibility. Batch and compress events, send them on a low-priority session so they never delay the product (see the reliability design), and tolerate loss (analytics is best effort).

Experiments add correctness rules. Stable assignment: hash a stable id with the experiment name so a user stays in one variant (Q486). Exposure logging: record that a user actually saw the variant at the moment they saw it, otherwise you compare people who never reached the feature. Check for a sample ratio mismatch before trusting any result: if a 50/50 experiment shows 5000 versus 5400, a chi-square test gives 15.38, far above the 3.84 threshold for a 5% significance, which signals an assignment or logging bug (in the check below 5000 versus 5100 is a plausible split at 0.99). Predefine the primary metric and guardrails (crash-free sessions, latency, unsubscribes, accessibility) so a win on one number cannot hide harm elsewhere, run long enough to cover weekly cycles, do not peek and stop on the first good result, and clean up the code path when the experiment ends. Keep experiments away from safety-critical paths such as payment and authentication unless they are server-controlled and tested. The staff-level point is governance: who may add an event, how long data lives and who can see it.

A large chi-square for a split that should be even means the groups are not randomised as intended.

Common traps

Counting every assigned user as exposed. People who never saw the feature dilute the effect and can reverse the conclusion.
Ignoring a sample ratio mismatch. It means the groups are not comparable, so the result may be an artefact.

They might follow up with

What is exposure logging? Recording the moment a user actually experiences the variant, not when they are assigned.
When does a feature need App Tracking Transparency? When it tracks users across apps and websites owned by other companies, or shares data with data brokers.

How do you design analytics and experimentation that respect privacy: event schemas, consent, minimisation, stable assignment, exposure logging and sample ratio checks?

Predict first

Common traps

They might follow up with

Related questions