25.8 · Pipelines that lie, build numbers, hotfix branches and rollout statistics

Most release incidents are not clever bugs: a step reported success when it failed, a build number went backwards, a hotfix missed a branch, or a rollout was judged on too little data.

Mental model. Make each pipeline step fail loudly, derive identifiers from complete history, track fixes by patch rather than by hope, and decide on rollouts with an interval, not a single percentage.

Honest pipelines

Exit statuses and build numbers that mean what you think.

  1. Why can a CI job pass when the build failed, and how do pipes, pipefail, set -e and PIPESTATUS change that? 25.8.1 · L3
    In a shell pipeline the exit status is by default the status of the last command. The classic iOS case is xcodebuild … | xcpretty or | tail: the formatter…
  2. Can you use the git commit count as the build number, and what breaks it: shallow clones, squashes, rebases and multiple branches? 25.8.2 · L3
    The commit count (git rev-list --count HEAD) is a tempting build number: it needs no server state and increases with every commit. It is only as trustworthy as…

Hotfixes and rollout decisions

Moving a fix between branches and judging a phased release.

  1. How do tags, release branches and cherry-picks work together for a hotfix, and how do you check that a fix reached every branch that needs it? 25.8.3 · L5
    Cut a tag on the exact commit you shipped (annotated, such as v1.0.0) and keep a release branch from it if you may need to patch that version. A hotfix is then…
  2. How do you decide whether a phased release is healthy: why is a crash-free percentage on a small sample misleading, and how does a confidence interval help? 25.8.4 · L5
    A phased release exposes a small fraction of users first, so the early numbers come from few users. A crash-free rate of 98 percent is the same figure for 4…

More in App Store & release engineering