How do Core Image and Metal relate, why is Core Image lazy, and how do you build a smooth image-processing pipeline?

18.5.2 · L5 · Interview

Predict first

A solid colour is cropped to 100×50, blurred, cropped back and rendered. What are the extents and the output size, and is there a Metal device?

import CoreImage
import Metal

let img = CIImage(color: CIColor(red: 1, green: 0, blue: 0)).cropped(to: CGRect(x: 0, y: 0, width: 100, height: 50))
print(img.extent)
let blur = img.applyingGaussianBlur(sigma: 5)
print(blur.extent.width > 100, blur.cropped(to: img.extent).extent)
let ctx = CIContext()
let cg = ctx.createCGImage(img, from: img.extent)!
print(cg.width, cg.height, MTLCreateSystemDefaultDevice() != nil)

(0.0, 0.0, 100.0, 50.0), then true (0.0, 0.0, 100.0, 50.0), then 100 50 true
(0.0, 0.0, 50.0, 100.0), then false, then 50 100 false
(0.0, 0.0, 100.0, 50.0), then false (0.0, 0.0, 100.0, 50.0), then 100 50 true

Mental model. A CIImage is a cooking recipe, not a dish. You can add steps endlessly at no cost, and the kitchen (Core Image) only cooks, as a single combined process, when someone orders (renders). Creating a new kitchen per order (a new CIContext) wastes everything it had warmed up.

Core Image is a high-level, GPU-accelerated image-processing framework built on a simple idea: a CIImage is not pixels but a recipe for producing an image (an unevaluated graph of filters applied to a source). Applying a filter (image.applyingGaussianBlur(sigma: 5), or CIFilter(name:) with parameters) returns a new CIImage instantly without doing any work; Core Image fuses the chain into one optimised GPU program and renders it only when you ask for output through a CIContext (createCGImage(_:from:), render(_:to:) into a pixel buffer or Metal texture, or by displaying in an MTKView). Because it is a recipe, an image has an extent (its bounds in working space): a 100×50 solid colour has extent (0, 0, 100, 50), a Gaussian blur grows the extent (the blur bleeds beyond the edges, so the extent is larger), and you usually cropped(to:) the original extent after blurring to avoid a soft, expanded border. Typical filters: colour adjustments, blurs, sharpen, transforms, compositing, lookup tables, and built-in face and barcode detectors (CIDetector); custom effects use Metal Core Image kernels (CIKernel written in the Metal Shading Language).

Metal is the low-level GPU API (MTLDevice, command queues and buffers, compute and render pipelines, textures, shaders in MSL) under Core Image, Core Animation’s rendering, SwiftUI’s Canvas and many frameworks. Use Core Image for standard filters and quick results; drop to Metal compute or render pipelines for custom algorithms, game rendering, ML pre-processing or when you need full control of resources and scheduling, with MetalPerformanceShaders and MPSGraph providing tuned kernels. In both, performance rules: create the CIContext once and reuse it (creating one per frame is the most common mistake: it is expensive and discards caches); render to a CVPixelBuffer or texture backed by an IOSurface to avoid copies; avoid converting to UIImage or CGImage in a hot path (it forces a CPU round trip); do not render at a size larger than you display; keep work off the main thread; mind colour management (Core Image uses a working colour space, and wide-colour or HDR needs explicit handling); and profile with the Metal System Trace and GPU frame capture in Xcode. On the simulator Metal runs on the host GPU and differs from device behaviour, so verify performance on a real device. With video, use AVVideoComposition(asset:applyingCIFiltersWithHandler:) for a simple per-frame filter pipeline, a custom compositor for more control, and keep frames as CVPixelBuffers end to end.

The blur widens the extent past the source, cropping returns the original bounds, and rendering produces a 100×50 bitmap. A Metal device exists.

Common traps

Creating a CIContext for every frame. It is expensive and drops caches; create one and reuse it.
Converting to UIImage between filters. It forces a CPU round trip and a copy; chain CIImage operations and render once.

They might follow up with

Why does a blur change an image’s extent? It samples beyond the edges, so the result extends past the original bounds; crop back to the source extent.
When do you go from Core Image to Metal? For custom algorithms, game rendering, or full control of GPU resources.

How do Core Image and Metal relate, why is Core Image lazy, and how do you build a smooth image-processing pipeline?

Predict first

Common traps

They might follow up with

Related questions