How do you design a quality system for a large team: test tiers, ownership, coverage, flake budgets and release gates?
Predict first
A pull-request job takes 55 minutes, and developers batch their changes. What is the best first move?
// A: require it for every commit and wait
// B: split into a fast pre-merge tier and slower post-merge and nightly tiers, selecting tests by impact
// C: skip tests on busy days- A
- B
- C
Mental model. A hospital does not rely on one big exam at discharge. It has quick checks at every step, specialists for the hard cases, monitors on the ward, and a review of every incident.
At scale testing is a system with goals, owners and feedback loops, not a folder of tests. Start from what the team needs: fast feedback to developers, trust in the main branch, and confidence to ship often. Organise the suite in tiers by speed and scope: a pre-commit and pre-merge tier of unit, integration and snapshot tests that finishes in minutes (use test impact analysis or module-level builds to run only affected tests, Q373), a post-merge tier with broader integration and a small set of UI journeys, and nightly or pre-release runs for the full device matrix, localisation, upgrade and accessibility checks. Keep the blocking path small and reliable; push slow or inherently noisy checks to non-blocking jobs that still report.
Make it owned and measured. Each module or feature team owns its tests and their health; a dashboard tracks test duration, flake rate and quarantine count, and a flake budget (a maximum fraction of builds that may fail for non-code reasons) triggers work when exceeded. Treat coverage as a map of untested risk, used per change (“new and changed code is covered”) rather than a global number to chase. Use contract tests at module and service boundaries (an API schema, the data shared with extensions, Q413) so teams can change independently, and keep a characterization-test habit before refactoring legacy code (Q357). Add release gates that combine tests with field signals (crash-free rate, hang and launch metrics, Q432) and stage the rollout. Write down the strategy: what each layer is for, what not to test, how to add a test, and who to ask. The L8 answer is the loop: prevent with design and fast tests, detect with CI tiers and field data, and learn by fixing the cause of each escape, not only the instance.
Fast, trusted feedback on every change is the goal; broader checks run later and still report.
Common traps
- One giant blocking CI job. Slow feedback pushes people to batch changes and skip waiting, so failures get harder to localise.
- A mandatory global coverage number. It rewards trivial tests and ignores the risky code.
They might follow up with
- How do you decide what runs on every pull request? The fast, reliable, relevant tier, selected by impact, with the rest running after merge or nightly.
- How do you keep the suite healthy over years? Track flakes and duration, quarantine with deadlines, delete tests that no longer protect anything, and review the strategy periodically.
Related questions
- 22.7.1 Why do tests flake, how do you find and fix them, and what makes a CI test run hermetic?
- 22.1.1 What is the test pyramid for an iOS app, and what does each layer cost and catch?
- 22.1.2 What should you not test, and how do you decide whether a test is worth having?
- 16.6.1 A CI pipeline takes hours on a large modular app. How do you analyse and cut it: affected builds, caching…
- 19.7.2 How do you treat extensions, widgets and intents as a public API of the product: shared modules, data…
- 21.3.2 How do you build a performance culture on a team: budgets, regression gates, ownership and release guardrails?
← Why do tests flake, how do you find and fix them, and what makes a CI…