How is a CVPixelBuffer laid out, why are rows padded, and how do pools limit memory?

18.9.5 · L5 · Interview

Predict first

What are the strides and what does the pool do at its limit?

import CoreVideo

func describe(_ name: String, _ format: OSType, _ w: Int, _ h: Int) {
    var buffer: CVPixelBuffer?
    let status = CVPixelBufferCreate(nil, w, h, format, [kCVPixelBufferIOSurfacePropertiesKey as String: [:] as [String: Any]] as CFDictionary, &buffer)
    guard status == kCVReturnSuccess, let b = buffer else { print(name, "failed"); return }
    var line = "\(name): planes \(CVPixelBufferGetPlaneCount(b)), IOSurface-backed \(CVPixelBufferGetIOSurface(b) != nil)"
    if CVPixelBufferGetPlaneCount(b) == 0 { line += ", bytesPerRow \(CVPixelBufferGetBytesPerRow(b)) (pixels need \(w * 4))" }
    for p in 0..<CVPixelBufferGetPlaneCount(b) {
        line += " | plane \(p): \(CVPixelBufferGetWidthOfPlane(b, p))x\(CVPixelBufferGetHeightOfPlane(b, p)) bytesPerRow \(CVPixelBufferGetBytesPerRowOfPlane(b, p))"
    }
    print(line)
}
describe("BGRA 100x50", kCVPixelFormatType_32BGRA, 100, 50)
describe("NV12 100x50", kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange, 100, 50)
describe("NV12 101x51", kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange, 101, 51)

var pool: CVPixelBufferPool?
CVPixelBufferPoolCreate(nil, nil, [kCVPixelBufferPixelFormatTypeKey as String: kCVPixelFormatType_32BGRA,
                                   kCVPixelBufferWidthKey as String: 64, kCVPixelBufferHeightKey as String: 64] as CFDictionary, &pool)
let limit = [kCVPixelBufferPoolAllocationThresholdKey as String: 3] as CFDictionary
var held: [CVPixelBuffer] = []
var outcomes: [String] = []
for _ in 0..<5 {
    var buffer: CVPixelBuffer?
    let status = CVPixelBufferPoolCreatePixelBufferWithAuxAttributes(nil, pool!, limit, &buffer)
    outcomes.append(status == kCVReturnSuccess ? "ok" : (status == kCVReturnWouldExceedAllocationThreshold ? "over threshold" : "error \(status)"))
    if let buffer { held.append(buffer) }
}
print("5 requests while keeping every buffer, threshold 3:", outcomes)
held.removeAll()
var again: CVPixelBuffer?
print("after releasing them, a request is", CVPixelBufferPoolCreatePixelBufferWithAuxAttributes(nil, pool!, limit, &again) == kCVReturnSuccess ? "ok" : "refused")

var lonely: CVPixelBuffer?
CVPixelBufferCreate(nil, 4, 4, kCVPixelFormatType_32BGRA, nil, &lonely)
print("base address before locking is nil:", CVPixelBufferGetBaseAddress(lonely!) == nil)
CVPixelBufferLockBaseAddress(lonely!, [])
print("base address after locking is nil:", CVPixelBufferGetBaseAddress(lonely!) == nil)
CVPixelBufferUnlockBaseAddress(lonely!, [])

BGRA row 448 for 400 bytes, NV12 planes of 128 byte rows, the 4th and 5th requests refused
BGRA row exactly 400
the pool allocates without limit

Mental model. Rows of lockers: each row is built a little longer than needed for tidy alignment, and a locker pool refuses to build more once its limit is reached.

A CVPixelBuffer is a pixel format plus rows with a bytesPerRow that may be bigger than the data. A 100×50 BGRA buffer needs 400 bytes a row but had 448 in the check. Video usually uses planar YUV such as NV12 (420YpCbCr8BiPlanarVideoRange): a full-resolution luma plane and a half-resolution interleaved chroma plane, each with its own width, height and padded row (100×50 gave planes of 100×50 and 50×25, both 128 bytes a row; odd sizes round up to 101×51 and 51×26). Ask for kCVPixelBufferIOSurfacePropertiesKey to get an IOSurface-backed buffer that Metal, the video encoder and other processes can share without copying (Q393). The base address is nil until you CVPixelBufferLockBaseAddress, and any CPU access belongs between lock and unlock.

Allocate through a CVPixelBufferPool (a writer adaptor gives you one) so buffers are recycled. A pool has a minimum and an allocation threshold: with a threshold of 3 and every buffer held, requests 4 and 5 returned kCVReturnWouldExceedAllocationThreshold instead of growing memory, and after releasing the buffers a request succeeded again. Treat that status as back-pressure: drop a frame rather than allocate. Capture pipelines run out of buffers when frames are retained too long. Not run: camera-supplied buffers and the exact number of buffers a device camera allocates.

Rows are padded per plane and the pool enforces its threshold.

Common traps

Reading pixels as width * 4 apart. Use bytesPerRow per plane.
Keeping frames in an array. The pool runs dry and the camera stalls.

They might follow up with

When is BGRA the right choice? When the consumer is Core Graphics or Core Image; use NV12 for the encoder and the GPU path.

How is a CVPixelBuffer laid out, why are rows padded, and how do pools limit memory?

Predict first

Common traps

They might follow up with

Related questions