With @Test(arguments:), how are several argument collections combined, and how is a failure reported?

22.8.1 · L3 · Interview

Predict first

Two collections, a zip and one failing argument. What is reported?

// === Package.swift ===
// swift-tools-version:6.0
import PackageDescription
let package = Package(name: "Lib", targets: [.target(name: "Lib"), .testTarget(name: "LibTests", dependencies: ["Lib"])])
// === Sources/Lib/Lib.swift ===
public func square(_ x: Int) -> Int { x * x }
// === Tests/LibTests/T.swift ===
import Testing
@testable import Lib

@Test(arguments: [1, 2], ["a", "b", "c"])
func product(_ n: Int, _ s: String) { #expect(square(n) > 0, "\(s)") }

@Test(arguments: zip([1, 2], ["a", "b"]))
func zipped(_ n: Int, _ s: String) { #expect(square(n) > 0, "\(s)") }

@Test(arguments: [1, 2, 3, 4])
func oneFails(_ n: Int) { #expect(n != 3) }
  • 6 cases for the product, 2 for the zip, and only argument 3 fails
  • 2 cases for the product
  • The whole test stops at the first failing argument

Mental model. A loop in one test is one exam paper marked once. Arguments are one paper per student: each gets its own mark, and one failing student does not hide the rest.

One argument collection runs the body once per element. Two collections give the Cartesian product: arguments: [1, 2], ["a", "b", "c"] runs six cases, which the check confirms (“with 6 test cases passed”). To pair elements instead, pass zip(a, b), which gives two cases from two-element collections. Each case is its own test case with its arguments in the name, so a failure points at the exact input: in the check, only n → 3 of four cases records an issue, the test as a whole is reported failed, and the other three cases still run and pass. Arguments must be Sendable, and cases run in parallel, so the body must not share mutable state between cases.

Prefer this to a for loop inside one test: a loop stops being informative after the first failure (or hides which element failed), cannot be rerun for one input, and gives one result. Prefer a small named type for the inputs when there are several fields (a struct with an expected value and a description), and keep the product small, because it grows multiplicatively: three collections of ten are a thousand cases. Parameterised tests are also the right tool for table-driven tests such as parsing and formatting edge cases. XCTest has no equivalent; people write loops, or generate one method per input.

Collections multiply, zip pairs, and each argument is judged on its own.

Common traps

  • Passing two collections when you wanted pairs. You get every combination, not element-wise pairs; use zip.
  • Giant products. The case count multiplies, so a modest-looking declaration can mean thousands of runs.

They might follow up with

  • Can one case be rerun alone? Yes, cases have identities from their arguments, so tools can run a single one.
  • Do the cases run in order? No; they may run in parallel.

Related questions

  • 22.8.2 What are traits in Swift Testing, and how do you skip, tag, bound and annotate tests with them?
  • 22.8.5 Swift Testing runs tests in parallel by default. What does .serialized change, and what does it not fix?
  • 22.2.1 How do XCTest and Swift Testing differ, and when do you still use XCTest?

What are traits in Swift Testing, and how do you skip, tag, bound and… →