How do build-for-testing and test-without-building split a test run, what is an xctestrun file, and how do you read results from the .xcresult bundle?

25.7.4 · L5 · Interview

Predict first

Tests are built once, the sources are deleted, and the tests run from the xctestrun. What happens?

// === run.sh ===
cp -R "$FIXTURE"/. .
DEST='platform=iOS Simulator,name=iPhone 18 Pro'
xcodebuild build-for-testing -project Demo.xcodeproj -scheme Demo -destination "$DEST" -derivedDataPath dd -quiet >/dev/null 2>&1
echo "xctestrun files: $(ls dd/Build/Products/*.xctestrun | wc -l | tr -d ' ')"
echo "relocatable test root: $(grep -c __TESTROOT__ dd/Build/Products/*.xctestrun | head -1)"
rm -rf Demo.xcodeproj Demo DemoTests
xcodebuild test-without-building -xctestrun dd/Build/Products/*.xctestrun -destination "$DEST" -resultBundlePath r.xcresult >/dev/null 2>&1
echo "test-without-building exit: $? (project sources deleted first)"
xcrun xcresulttool get test-results summary --path r.xcresult | grep -E '"(result|passedTests|failedTests)"' | tr -d ' \n'; echo
xcodebuild -help 2>&1 | grep -oE -e '-(only-testing|skip-testing|retry-tests-on-failure|test-iterations|parallel-testing-enabled|resultBundlePath|xctestrun)' | sort -u | tr '\n' ' '; echo

One xctestrun with a relocatable root; the run exits 0 and the summary shows Passed with 1 passed test; the retry and sharding options exist
The run fails because the sources are missing
No xctestrun file is produced

Mental model. Build-for-testing is baking the cakes; test-without-building is tasting them in any kitchen. The result bundle is the tasting notes with photos.

xcodebuild build-for-testing compiles the app and the test bundles and writes an .xctestrun file next to the products; test-without-building -xctestrun … then runs them without compiling anything. In the check, the xctestrun path variables use __TESTROOT__, so the products folder can be copied to another machine, and the run still passes after the project and source files were deleted, proving the second step needs only the built products. That split is the foundation of faster CI: build once on a strong runner, upload the products, then run shards in parallel on several simulators or machines with -only-testing:Target/Class/method or -skip-testing, and rerun failures with -retry-tests-on-failure and -test-iterations (the check confirms those options exist in this Xcode). Keep the flake budget honest (Q453): a retry that turns red into green still belongs in a report.

Always pass -resultBundlePath. The .xcresult bundle holds results, logs, screenshots, attachments, coverage and the failure details; read it with xcrun xcresulttool get test-results summary --path r.xcresult (JSON with result, passedTests, failedTests) and the sibling subcommands for tests, details and metrics (the older get --format json form is deprecated). Upload the bundle as a CI artefact on failure, convert the summary into the format your CI understands rather than parsing console output, and merge result bundles from shards with xcresulttool merge before reporting. Caveats: the destination must exist on the machine running the tests (same OS version family), a test run reuses the xctestrun’s configuration so changing a test plan means rebuilding, and UI tests need a signed or properly configured runner on devices. The check ran on one simulator only; multi-machine sharding was not exercised.

Test execution needs the built products and the xctestrun, not the project.

Common traps

Parsing console output to decide pass or fail. Use the exit status and the result bundle; the log format changes.
Retrying until green without recording the retries. Flaky tests then look healthy and the budget is never spent down.

They might follow up with

Why can sharding be slower? Each shard pays simulator boot and install cost; split by measured test time, not by file count.
Can a test plan replace the xctestrun? Test plans define configurations (languages, environment); the xctestrun is the built, runnable form of them.

How do build-for-testing and test-without-building split a test run, what is an xctestrun file, and how do you read results from the .xcresult bundle?

Predict first

Common traps

They might follow up with

Related questions