How do you run an engineering incident, hold a blameless review, and mentor and hire well: roles, mitigation first, rubrics and calibration?

26.4.3 · L8 · Interview

Predict first

A new release crashes for 5% of users and the team starts reading logs to find the root cause. What should happen first?

// A: finish the investigation, then act
// B: pause the rollout and switch off the feature with a flag now, then investigate and ship a fix
// C: announce it is not serious

Mental model. A good incident response is a well-drilled crew: one person steers, others bail and patch, and afterwards they study the chart to learn how the ship hit the rock, not to blame the person at the wheel.

Incident response. The priority is to stop the harm, not to find the culprit. Define severities by user impact, and have named roles: an incident commander who coordinates and decides (and does not debug), people investigating and fixing, a communications owner for stakeholders and status updates, and a scribe who keeps a timeline. Mitigate before you diagnose: pause the phased release, flip a flag or kill switch, roll the server back, or ship a hotfix (Q485, Q486); a mobile team cannot roll back installed apps, so the levers must exist before the incident. Communicate on a regular cadence, even if the message is “no change”. Afterwards run a blameless review: build a timeline from logs and chat, ask how the system and process allowed the error to reach users (detection, testing, review, rollout, alerting) instead of who made it, find contributing causes rather than one “root cause”, and finish with a short list of action items with owners and dates that are actually tracked (a regression test, an alert, a safer default). Share the review widely; teams that hide incidents repeat them.

Mentoring and the hiring bar are the same craft: raising the level of the people around you. Mentoring means giving real, stretching work with a safety net, feedback that is specific, timely and about behaviour and impact, sponsoring people for opportunities, and teaching how to think (how to find the cause, weigh a trade-off, write a design) rather than handing out answers. Delegate decisions along with the work, review for learning and not only correctness, and make your own reasoning visible. For hiring, use a structured process: a written rubric of what each level must demonstrate (design, coding, debugging, collaboration), the same questions for every candidate, independent written feedback before discussion, a calibrated bar enforced by a trained panel, and a focus on evidence of what the candidate did, not pedigree or confidence. Treat the interview as a work sample of the job (a code review, a design discussion, debugging a real bug) and give candidates a good experience. The staff-level expectation is to be a force multiplier: faster recoveries, clearer decisions and more capable engineers because you were on the team.

Mitigation stops the harm quickly; the investigation continues while users are protected.

Common traps

Debugging the root cause while users are still affected. Mitigate first (flag, pause, rollback), then investigate.
A review that ends in “be more careful”. Without a system change and an owner, the same failure recurs.

They might follow up with

What does the incident commander do? Coordinates, assigns roles, decides and communicates, without being the main debugger.
How do you keep a hiring bar consistent? A written rubric, structured interviews, independent feedback and calibration across interviewers.

How do you run an engineering incident, hold a blameless review, and mentor and hire well: roles, mitigation first, rubrics and calibration?

Predict first

Common traps

They might follow up with

Related questions