18.9 · Reading an asset, grabbing frames, reading samples, frame-exact time, and pixel buffers

AVFoundation problems are about time and memory: frames that are not where you asked, rounding that drifts, and buffers whose rows are not what you assume.

Mental model. Media is a timeline of samples. Ask the right question about time (exact or nearby?) and about layout (what is in a row?).

Assets and frames

Asset metadata, image generation tolerance, and reading decoded samples.

  1. How do you read what is in a media file, and why are the loads async and throwing? 18.9.1 · L3
    An AVURLAsset is a lazy description: you ask for properties with try await asset.load(.duration, .isPlayable, .tracks), and for per-track facts with…
  2. Why does AVAssetImageGenerator return a frame at a different time than you asked for, and how do you get the exact one? 18.9.2 · L3
    Video is stored as keyframes plus differences, so decoding an arbitrary time is expensive. AVAssetImageGenerator therefore defaults to infinite tolerance: it…
  3. When do you use AVAssetReader instead of a player, and what do the output settings change? 18.9.3 · L5
    A player renders in real time for display; AVAssetReader pulls samples as fast as you ask for processing, analysis or transcoding. Add an…

Time and buffers

CMTime arithmetic and pixel-buffer layout and pools.

  1. How does CMTime avoid rounding errors, and how can sloppy conversion make audio and video drift? 18.9.4 · L5
    CMTime is a rational number (value ÷ timescale), so 29.97 fps video is exactly 1001/30000 s per frame and 30 frames are exactly 30030/30000 = 1.001 s. The…
  2. How is a CVPixelBuffer laid out, why are rows padded, and how do pools limit memory? 18.9.5 · L5
    A CVPixelBuffer is a pixel format plus rows with a bytesPerRow that may be bigger than the data. A 100×50 BGRA buffer needs 400 bytes a row but had 448 in the…

More in Media & graphics