How do you reproduce a bug reliably: environment, feature flags, seeded data, launch arguments, simulator tools and network conditions?

23.6.2 · L3 · Interview

Predict first

Which simctl subcommands exist for pushing notifications, privacy permissions, location, deep links, status bar overrides and erasing a device?

// === dummy ===
  • All of: erase, location, openurl, privacy, push, status_bar
  • Only openurl and erase
  • None: you must use the Simulator menus

Mental model. Reproducing a bug is setting up the crime scene exactly as it was: the same room, the same lights, the same weather. Only then can you test what happened.

Start by writing down what could differ between your machine and the failure: the build (version, configuration, which commit), OS and device, the user’s account and data, settings (language, region, time zone, Dynamic Type, dark mode, accessibility), permissions granted or denied, network quality and the server environment, feature flags and remote config, and timing and ordering. Then make each of those controllable. Use launch arguments and environment variables (set in the scheme, or by a UI test, Q449) to switch on a debug mode, choose a stub server or force a flag; ship a hidden debug menu in internal builds to set flags, switch environments and load a seeded data set (a known account, a fixture database) instead of clicking through setup; and keep the production data’s shape in an anonymised fixture so you can replay the case.

The simulator is a control panel: xcrun simctl can push a notification payload, change privacy permissions (grant, revoke or reset), set a location, openurl for deep links and universal links, override the status_bar for screenshots, and erase a device to a clean state, as the check below lists. Combine that with the Network Link Conditioner or a proxy for slow, lossy or offline conditions, the scheme’s language and region options, a changed system time zone, and Low Power Mode. If it only fails on a device, attach the debugger wirelessly and use logging. Once you can reproduce it, shrink it: remove steps until you have the smallest scenario, then turn it into an automated test so it can never return (Q357). If you cannot reproduce it, add logging and breadcrumbs for the next occurrence rather than guessing at a fix.

simctl is the scriptable side of the simulator, so a bug scenario can be set up from a script or CI step.

Common traps

  • Testing with your own warm, fully set-up account. Many bugs appear only on a first launch, an empty state or an old data version.
  • “Fixing” a bug you could not reproduce. Without a reproduction you cannot know the fix worked; add logging instead.

They might follow up with

  • How do you test a deep link or push notification without a server? simctl openurl and simctl push with a payload file.
  • What do you do once you can reproduce it? Shrink it to a minimal case and add an automated regression test.

Related questions

  • 23.6.1 What is a sysdiagnose, how do you use Console.app and the log command, and what should you ask a user or…
  • 23.2.1 How do you debug layout and views with the view debugger, and how do you inspect what your app really sends…
  • 23.7.1 What does an Xcode scheme control, how do test plans and derived data fit in, and how do you use xcodebuild…
  • 22.5.2 How do you write UI tests that stay stable: accessibility identifiers, page objects, launch arguments…
  • 22.4.2 What makes a test non-deterministic, and how do you control time zones, locales, randomness, UUIDs and the…

← What is a sysdiagnose, how do you use Console.app and the log…