How does AVCaptureSession work: devices, inputs, outputs, presets, permissions and interruptions?

18.3.1 · L5 · Interview

Predict first

In the simulator, is there a default camera, and what is the default preset?

import AVFoundation

print(AVCaptureDevice.default(for: .video) == nil)
print(AVCaptureSession.Preset.photo.rawValue, AVCaptureSession().sessionPreset.rawValue)
  • true, then AVCaptureSessionPresetPhoto AVCaptureSessionPresetHigh
  • false, then AVCaptureSessionPresetHigh AVCaptureSessionPresetPhoto
  • true, then AVCaptureSessionPresetHigh AVCaptureSessionPresetHigh

Mental model. A capture session is a tiny TV studio: cameras and microphones in, a few outputs (monitor, recorder, a frame grabber) wired by a technician (the session), and a rule that you only rewire between takes (begin and commit configuration).

An AVCaptureSession connects inputs (an AVCaptureDeviceInput made from a camera or microphone AVCaptureDevice) to outputs (AVCaptureVideoDataOutput for raw frames, AVCapturePhotoOutput for stills, AVCaptureMovieFileOutput for recordings, AVCaptureAudioDataOutput, metadata outputs for barcodes and faces) and a preview (AVCaptureVideoPreviewLayer), via connections the session builds. The setup recipe: ask for permission (AVCaptureDevice.requestAccess(for: .video); the Info.plist keys NSCameraUsageDescription and NSMicrophoneUsageDescription are mandatory, and a missing one crashes at the request), find a device (AVCaptureDevice.default(.builtInWideAngleCamera, for: .video, position: .back), or a DiscoverySession to list cameras, which returns nil in the simulator since it has no camera), then inside beginConfiguration() / commitConfiguration() add the input and outputs (checking canAddInput and canAddOutput) and choose a sessionPreset (the default is .high; .photo for full-resolution stills) or select a specific activeFormat for exact resolution, frame rate and HDR. Call startRunning() on a background queue: it is a blocking call that can take hundreds of milliseconds, and doing it on the main thread freezes the UI. Configure the device’s focus, exposure, white balance, zoom and frame-rate range inside lockForConfiguration() and unlock when done.

Plan for the unhappy paths. Interruptions: the session is interrupted by a phone call, another app using the camera, split-screen multitasking on iPad, or the system’s camera being in use (AVCaptureSession.wasInterruptedNotification with a reason such as videoDeviceNotAvailableInBackground or audioDeviceInUseByAnotherClient), and interruptionEndedNotification tells you when to resume; show a message instead of a frozen preview. Runtime errors (runtimeErrorNotification, for example media services were reset) need a restart. Permissions: handle denied and restricted with an explanation and a link to Settings, and re-check on returning to the foreground. Threading: use a dedicated serial queue for session changes and a separate one for the sample-buffer delegate callbacks; keep the callback fast, because if it is slow frames are dropped (captureOutput(_:didDrop:from:)), and set alwaysDiscardsLateVideoFrames true for live processing. Resources: stop the session when the view disappears (the camera light and battery), and use AVCaptureMultiCamSession only on supported devices. Newer APIs add AVCaptureDevice.RotationCoordinator for orientation, and the Camera Control and Deferred Photo Processing features. Test on a real device, with each permission state, interruptions and low light, since none of it appears in the simulator.

The simulator has no camera, so there is no default video device. A new session starts with the high preset.

Common traps

  • Calling startRunning() on the main thread. It blocks and freezes the UI; start it on a background queue.
  • Doing slow work in the sample-buffer callback. Frames are dropped; keep it fast or hand the buffer to another queue.

They might follow up with

  • Why wrap changes in beginConfiguration and commitConfiguration? The session applies them atomically instead of reconfiguring after each change.
  • What does the simulator return for a camera device? nil, since it has no camera; test capture on a real device.

Related questions

  • 18.3.2 What are CVPixelBuffer, CMSampleBuffer and IOSurface, and how do you process frames without copying them?
  • 18.2.1 What is the AVAudioSession, how do categories and modes decide behaviour, and how do you handle interruptions…
  • 18.5.2 How do Core Image and Metal relate, why is Core Image lazy, and how do you build a smooth image-processing…

What are CVPixelBuffer, CMSampleBuffer and IOSurface, and how do you… →