How do you build a production diagnostics pipeline: collection, symbolication, grouping, prioritisation, breadcrumbs and the response?
Predict first
A new release shows a crash on the payment screen affecting 0.3% of users, and a rare background crash with a higher raw count. Which do you fix first?
// A: the background crash, because its count is higher
// B: the payment crash, because of the affected flow and users, and pause the rollout if it is a regression
// C: neither, wait for more data- A
- B
- C
Mental model. A fire service does not chase every alarm equally. It has detectors in every building, a dispatcher who groups calls, a priority rule, and a review after each fire.
One crash is a debugging task; ten thousand are a system. Design it in stages. Collect from several sources because each sees different things: the platform’s crash and metric data (Xcode Organizer, MetricKit diagnostic payloads for hangs, CPU and disk-write exceptions, Q431), a third-party or in-house crash reporter for faster and richer reports, and your own logs. Symbolicate automatically: upload the dSYM from every release build in CI, fail the release if it is missing, and verify by UUID that the uploaded symbols match the shipped binary. Group crashes by a stable signature (the top frames in your code, the exception and trap message) so thousands of reports become dozens of issues, and track each issue’s count, affected users, versions and devices, and whether it regressed.
Prioritise by user impact, not by count alone: a crash on launch or in a payment flow outranks a rare background crash; look at crash-free sessions and users per version, and treat hangs, launch time and out-of-memory kills (which produce no crash report, Q227) as part of the same picture. Add context so a report can be reproduced: breadcrumbs (a ring buffer of recent screens, requests and log lines kept in memory and attached to the report), the app’s feature-flag and configuration state, OS, device, memory and locale, and a non-identifying session id; keep personal data out (Q347). Close the loop: an owner triages new issues daily, a fix ships behind a staged rollout with automatic pause on a regression in crash-free rate, a kill switch or remote flag can disable a crashing feature without a release, and each fix adds a regression test (Q357). Track the pipeline’s health too: symbolication rate, time to detect and time to fix. The senior answer ties it together: detect fast, group sensibly, fix what hurts most, and learn from every escape.
Impact on a critical flow outweighs a bigger count of harmless background failures.
Common traps
- Counting crashes by raw volume. One noisy rare crash can hide a smaller one on the sign-in screen that costs far more.
- Shipping without uploading dSYMs. The reports arrive but are unreadable until it is too late to match them.
They might follow up with
- How do you decide what to fix first? By affected users and the importance of the flow, with regressions and launch crashes first.
- How do you limit damage from a bad release? Staged rollout, automatic pause on a metric regression, and a remote kill switch.
Related questions
- 23.5.1 How do you read a crash report, what do the exception types and Swift runtime messages mean, and how does…
- 23.6.1 What is a sysdiagnose, how do you use Console.app and the log command, and what should you ask a user or…
- 21.3.1 What does MetricKit give you in production, how do you subscribe, and what is the difference between metric…
- 21.3.2 How do you build a performance culture on a team: budgets, regression gates, ownership and release guardrails?
- 6.6.2 What is jetsam, and how does it differ from a crash?
← How do you read a crash report, what do the exception types and Swift…