What is VideoToolbox, when would you use it directly, and what is different about media on tvOS?

18.6.1 · L2 · Interview

Predict first

A team wants to play an RTSP camera stream that AVPlayer cannot open, with low latency. Which layer do they work at?

// A: AVPlayer, with a custom URL scheme
// B: VideoToolbox, decoding the compressed frames themselves into pixel buffers
// C: SceneKit

Mental model. VideoToolbox is the engine room under the player: you rarely go down there, but when the ship is nonstandard (a custom stream) you have to. tvOS is the same ship used from the sofa: the controls are a single remote and a highlighted cursor, so everything must be reachable in four directions.

VideoToolbox is the low-level framework for hardware-accelerated video encoding, decoding and pixel transfer: VTCompressionSession (encode frames to H.264, HEVC or ProRes with chosen bitrate, key-frame interval and profile), VTDecompressionSession (decode compressed samples to CVPixelBuffers) and VTPixelTransferSession (scale and convert formats). AVPlayer, AVAssetWriter and the capture pipeline use it for you, and for most apps that is where it should stay: you use AVFoundation and let it pick the hardware decoder. Reach for VideoToolbox directly when you handle raw compressed streams that AVFoundation cannot play (a custom or proprietary transport, WebRTC, an RTSP camera, a game streaming client), need frame-accurate control or low latency (a sub-100 ms live pipeline), or must encode live frames with specific settings (real-time bitrate control for video calls). Key ideas: compressed data arrives as CMSampleBuffers holding NAL units with a format description (the SPS and PPS parameter sets for H.264), decode is asynchronous and out of order (B-frames mean presentation order differs from decode order, so use the presentation timestamps), and a decoder session can be invalidated (for example when the app is backgrounded or the GPU is reset) and must be recreated. Check VTIsHardwareDecodeSupported(codec) for the codec and device, and note the simulator may not offer the hardware decoders a device has (it reports false there), so test on hardware. Hardware codecs are far more efficient than software ones for battery and heat, and HEVC and ProRes support varies by device generation.

tvOS shares AVFoundation, Core Image and Metal with iOS but changes the interaction model. There is no touch: users navigate with the Siri Remote and a focus engine that moves a highlight between focusable views, so design layouts the focus can reach by moving in the four directions (focus guides, canBecomeFocused, didUpdateFocus, SwiftUI .focusable() and @FocusState), make focused items visibly distinct (scale and parallax effects), and keep targets large and well spaced for viewing across a room (the “10-foot UI”, with safe margins for overscan). Media specifics: AVPlayerViewController provides the standard transport and info panels, top shelf extensions surface content on the home screen, MPNowPlayingInfoCenter and remote commands work as on iOS (including the remote’s play/pause button), and playback should handle HDR and Dolby formats and match the display’s frame rate and dynamic range (AVDisplayCriteria) so content plays at its native rate without a mode switch at the wrong time. Storage is limited and local persistence may be purged (use iCloud or the network as the source of truth), there is no browser or file system for the user, and apps ship as a tvOS target with its own icon and layered image assets. When sharing code with iOS keep the model and playback layers common and the UI layer separate.

For streams AVFoundation cannot open, you receive the compressed frames yourself and decode them with a VTDecompressionSession, then render the pixel buffers.

Common traps

Using VideoToolbox when AVFoundation would do. You take on parameter-set handling, ordering and session lifecycle for no benefit; use it only for streams AVFoundation cannot play.
Designing a tvOS screen for touch. Without a pointer, every control must be reachable by the focus engine.

They might follow up with

Why is decode output out of order? B-frames: presentation order differs from decode order, so use presentation timestamps.
What does the focus engine do? Moves a highlight between focusable views as the user swipes the remote in four directions.

What is VideoToolbox, when would you use it directly, and what is different about media on tvOS?

Predict first

Common traps

They might follow up with

Related questions