Why does AVAssetImageGenerator return a frame at a different time than you asked for, and how do you get the exact one?

18.9.2 · L3 · Interview

Predict first

Which time comes back with each tolerance?

try await makeVideo(at: clip, frames: 30, fps: 30)
defer { try? FileManager.default.removeItem(at: clip) }

let generator = AVAssetImageGenerator(asset: AVURLAsset(url: clip))
let wanted = CMTime(seconds: 0.5, preferredTimescale: 600)

func grab(_ label: String) async throws {
    let result = try await generator.image(at: wanted)
    let grey = (result.image.dataProvider!.data! as Data)[0]
    print(label, "-> actualTime:", String(format: "%.3f", result.actualTime.seconds), "| frame grey level:", grey)
}
// Default tolerance is infinite: the generator may return any nearby frame it can decode cheaply.
generator.requestedTimeToleranceBefore = .positiveInfinity
generator.requestedTimeToleranceAfter = .positiveInfinity
let loose = try await generator.image(at: wanted)
print("infinite tolerance: asked 0.500, got", String(format: "%.3f", loose.actualTime.seconds))

generator.requestedTimeToleranceBefore = .zero
generator.requestedTimeToleranceAfter = .zero
try await grab("zero tolerance")

do { _ = try await generator.image(at: CMTime(seconds: 50, preferredTimescale: 600)) } catch { print("a time past the end throws") }

generator.maximumSize = CGSize(width: 16, height: 16)
let small = try await generator.image(at: wanted).image
print("maximumSize 16x16 gives", small.width, "x", small.height)
  • 0.600 with infinite tolerance, 0.500 with zero, past the end throws, maximumSize gives 16 x 16
  • both give exactly 0.500
  • zero tolerance returns the nearest keyframe

Mental model. Asking a librarian for page 137 of a scroll: with “near enough” they hand you the nearest bookmark; with “exact” they unroll to the line.

Video is stored as keyframes plus differences, so decoding an arbitrary time is expensive. AVAssetImageGenerator therefore defaults to infinite tolerance: it may return a nearby frame it can reach cheaply. In the check, asking for 0.500 s with infinite tolerance returned actualTime 0.600, and with requestedTimeToleranceBefore/After = .zero it returned exactly 0.500 (grey level 130). The loose result is fine for a scrubber thumbnail strip; use zero tolerance for a poster frame, frame stepping or analysis, and always read actualTime rather than assuming. Which nearby frame you get depends on the encoder’s keyframe placement, so the 0.600 is specific to this clip and machine.

Use the async image(at:) (or images(for:) for many times, which is cheaper than many separate calls), set maximumSize so the decoder produces a small bitmap (16×16 in the check) instead of full frames, and appliesPreferredTrackTransform = true so portrait clips are upright. A time past the end throws. Exact seeking in a player has the same trade-off: seek(to:toleranceBefore:toleranceAfter:) with zero tolerances is precise but slower. Not run: HEVC or HDR sources, remote assets.

Tolerance trades accuracy for decode cost.

Common traps

  • Assuming actualTime equals the request. Read it.
  • Generating full-size frames for thumbnails. Set maximumSize.

They might follow up with

  • How do you make many thumbnails? One generator, images(for:), loose tolerance.

Related questions

  • 18.9.1 How do you read what is in a media file, and why are the loads async and throwing?
  • 18.9.3 When do you use AVAssetReader instead of a player, and what do the output settings change?
  • 18.9.4 How does CMTime avoid rounding errors, and how can sloppy conversion make audio and video drift?
  • 18.1.1 How do AVAsset, AVPlayerItem and AVPlayer fit together, what is CMTime, and why does playback “not start”?

← How do you read what is in a media file, and why are the loads async…   When do you use AVAssetReader instead of a player, and what do the… →