How do signposts and Points of Interest work, and how do you turn a measurement into an XCTest performance test with metrics and baselines?

21.2.2 · L5 · Interview

Predict first

Does using a signposter without Instruments attached work?

import os

let signposter = OSSignposter(subsystem: "com.example.app", category: .pointsOfInterest)
let id = signposter.makeSignpostID()
let state = signposter.beginInterval("load-feed", id: id)
signposter.emitEvent("cache-miss", id: id)
signposter.endInterval("load-feed", state)
print("intervals recorded without error, id created:", id != .null)

It works: the calls are safe even when nothing is recording
It crashes without Instruments attached
It does not compile outside an app

Predict first

How many times does the measured block run when iterationCount is 3?

// === Package.swift ===
// swift-tools-version:5.9
import PackageDescription
let package = Package(name: "Lib", targets: [.target(name: "Lib"), .testTarget(name: "LibTests", dependencies: ["Lib"])])
// === Sources/Lib/Lib.swift ===
public func work() -> Int { (0..<1000).reduce(0, +) }
// === Tests/LibTests/T.swift ===
import XCTest
@testable import Lib

final class T: XCTestCase {
    func testPerf() {
        var runs = 0
        let options = XCTMeasureOptions()
        options.iterationCount = 3
        measure(metrics: [XCTClockMetric(), XCTCPUMetric(), XCTMemoryMetric()], options: options) {
            runs += 1
            _ = work()
        }
        print("BLOCK_RAN \(runs)")
    }
}

3
4: one warm-up that is not recorded plus 3 measured
10

Mental model. Signposts are numbered mile markers painted on the road of your code, so the stopwatch can say what the runners did between marker 3 and marker 4.

A signpost is a cheap, named mark in your code that Instruments draws on its timeline. Create an OSSignposter(subsystem:category:) (the category .pointsOfInterest shows up in the Points of Interest track), call beginInterval(_:id:) to start a named interval and endInterval(_:_:) with the returned state to end it, and emitEvent for a one-off point. They are designed to be nearly free when nobody is recording, so you can leave them in shipping code. Wrap the phases you care about (parse feed, build snapshot, decode images) and the trace answers “which of my phases took the time” instead of showing anonymous stacks. The older C function os_signpost does the same job.

The same intervals can be tested. In an XCTest case, measure(metrics:options:) runs a block repeatedly and records each metric: XCTClockMetric (wall time), XCTCPUMetric (CPU time, cycles, instructions), XCTMemoryMetric (physical memory, peak), XCTStorageMetric (disk writes), XCTApplicationLaunchMetric (launch, in UI tests) and XCTOSSignpostMetric (the duration of your own signposted interval). The block runs one extra time first as a warm-up that is not recorded, then iterationCount times. In Xcode you can set a baseline and a maximum allowed deviation, and the test fails when the average regresses beyond it. That is the building block of a performance gate in CI (see the culture question), with the caveat that baselines are tied to a device model and machine, so compare like with like.

Signposting is an ordinary API available everywhere. Whether anyone is listening only affects whether the marks are kept.

XCTest runs the block once before measuring so one-off costs do not pollute the numbers, then runs the configured iterations.

Common traps

Signposting every loop iteration. Cheap is not free; mark phases, not inner loops.
Setting a baseline on one machine and running it on another. Baselines are per device configuration, so the test fails or hides regressions.

They might follow up with

Why use XCTOSSignpostMetric? It measures a precise interval inside your app (for example a feed render) rather than the whole test body.
Do signposts cost anything in production? Very little when nothing is recording, which is why they can stay in.

How do signposts and Points of Interest work, and how do you turn a measurement into an XCTest performance test with metrics and baselines?

Predict first

Predict first

Common traps

They might follow up with

Related questions