18.3 · AVCaptureSession, and zero-copy pixel buffers

Capturing video means a stream of large frames at 30 or 60 per second. The craft is making the camera work without ever copying a frame more than you must.

Mental model. A capture session is a small graph of inputs and outputs. A frame is one block of memory that the whole pipeline should share, not copy.

The capture session

Devices, inputs, outputs, permissions and interruptions.

1. How does AVCaptureSession work: devices, inputs, outputs, presets, permissions and interruptions? 18.3.1 · L5
   An AVCaptureSession connects inputs (an AVCaptureDeviceInput made from a camera or microphone AVCaptureDevice) to outputs (AVCaptureVideoDataOutput for raw…

Frames in memory

CVPixelBuffer, IOSurface, pools and zero-copy.

1. What are CVPixelBuffer, CMSampleBuffer and IOSurface, and how do you process frames without copying them? 18.3.2 · L5
   A video frame at 1080p BGRA is about 8 MB, and at 60 frames per second copying it even once per stage burns memory bandwidth and heats the device. Apple’s…

More in Media & graphics

• 18.1 · AVPlayer, time, and editing with compositions
• 18.2 · The audio session, interruptions, routes and AVAudioEngine
• 18.4 · HLS, buffering, Now Playing, remote commands, Picture in Picture and AirPlay
• 18.5 · ImageIO formats, Core Image, Metal, and SceneKit versus RealityKit
• 18.6 · VideoToolbox, tvOS focus and media, and building a playback platform
• 18.7 · Thumbnails and orientation, GIF timing, bitmap layout, and colour spaces
• 18.8 · Extents, working colour space, and a compute pass from start to finish
• 18.9 · Reading an asset, grabbing frames, reading samples, frame-exact time, and pixel buffers