21.8 · Writing benchmarks you can believe: the optimiser, build settings and growth tests

A number from a benchmark is only evidence if the code under test actually ran, in the build users get, at a size where the trend is visible. Most disappointing numbers are an artefact of the test, not the code.

Mental model. Distrust a result until you know the optimiser did not delete the work, the build matches Release, and the cost grows as you think.

Harness traps

Dead-code elimination and the cost of an unoptimised build.

  1. Your micro-benchmark reports an absurdly fast time. What did the optimiser probably do, and how do you write a benchmark that cannot be fooled? 21.8.1 · L3
    In an optimised build the compiler deletes work whose result is never used. A loop that computes a value and throws it away has no side effects, so it can be…
  2. How much slower is a Debug (-Onone) build than Release (-O), why, and what does that mean for profiling and for what you ship? 21.8.2 · L3
    Debug builds optimise for compile speed and debuggability: no inlining, no specialisation of generics, no removal of retain/release pairs, extra checks, and…

Complexity by experiment

Finding the real growth rate by measuring n and 4n.

  1. How do you find out how an operation really scales, and which Swift operations that look cheap are quadratic? 21.8.3 · L3
    Do not guess the complexity of a call; measure at n and 4n. If the time grows about four-fold the loop is linear, if it grows about sixteen-fold it is…

More in Performance & energy