What are CVPixelBuffer, CMSampleBuffer and IOSurface, and how do you process frames without copying them?

A video frame at 1080p BGRA is about 8 MB, and at 60 frames per second copying it even once per stage burns memory bandwidth and heats the device. Apple’s media stack therefore passes frames as reference-counted buffers that are shared, not copied. CVPixelBuffer is the pixel data: it has a width, a height, a pixel format (kCVPixelFormatType_32BGRA, or the efficient YUV 420YpCbCr8BiPlanar that cameras and video codecs use natively, which is half the size of BGRA for the same image), and bytesPerRow (the stride, which is at least width × bytesPerPixel and often larger because rows are padded for alignment: never assume width × 4). CMSampleBuffer wraps a buffer with its timing (presentation timestamp, duration) and format description; capture and decoding deliver these. A buffer created with kCVPixelBufferIOSurfacePropertiesKey is backed by an IOSurface, a kernel-managed block of memory that can be mapped by the CPU, the GPU and other processes at the same time without copying (it is how the camera, Core Image, Metal, the video encoder and the render server share a frame). To read or write pixels on the CPU you must bracket access with CVPixelBufferLockBaseAddress and UnlockBaseAddress; to use the GPU, wrap it as a Metal texture through a CVMetalTextureCache or a Core Image CIImage(cvPixelBuffer:), both of which reference the same memory.

The rules for a zero-copy pipeline. Keep frames in their native format (YUV from the camera into the encoder or Metal) and convert once, on the GPU, when you must. Use a pool (CVPixelBufferPool, via CVPixelBufferPoolCreatePixelBuffer) to recycle buffers of a fixed size and format instead of allocating one per frame: allocation is slow and fragments memory, and the pool bounds your memory. Release promptly: the capture pipeline has only a few buffers; if your code holds onto sample buffers (queueing them for later processing) the camera runs out and stops delivering frames (the “late frame” stall), so process and release, or copy a small result, within the callback’s lifetime. Lock only when needed and for the shortest time, use .readOnly when you only read, and do heavy per-pixel work on the GPU (Core Image, Metal, Accelerate vImage) rather than a Swift loop. Respect the stride when indexing, mind planar formats (separate base addresses per plane), and track colour space and orientation separately from the pixels. Watch the debugger and Instruments (the Metal System Trace, the memory graph, and the camera’s dropped frame count) for copies you did not intend, such as a UIImage conversion in the middle of a hot path, which allocates and converts the frame on the CPU.