18.5 · ImageIO formats, Core Image, Metal, and SceneKit versus RealityKit

When a frame has to be decoded, filtered or drawn, there is a right level to work at. ImageIO handles files, Core Image handles filters, Metal handles anything custom, and the 3D frameworks sit above.

Mental model. Stay lazy and stay on the GPU as long as you can. Decode once, filter as a recipe, and render once at the end.

Files and filters

Image formats with ImageIO, and Core Image and Metal.

  1. How do HEIF, JPEG, PNG and animated images differ, and what does ImageIO let you do with them? 18.5.1 · L3
    ImageIO (CGImageSource and CGImageDestination) is the low-level engine under UIImage, AsyncImage, Photos and Quick Look: it reads and writes image files…
  2. How do Core Image and Metal relate, why is Core Image lazy, and how do you build a smooth image-processing pipeline? 18.5.2 · L5
    Core Image is a high-level, GPU-accelerated image-processing framework built on a simple idea: a CIImage is not pixels but a recipe for producing an image (an…

3D and scenes

SceneKit and RealityKit: which and when.

  1. When do you use SceneKit, RealityKit, SpriteKit or Metal for 3D and games-adjacent work? 18.5.3 · L2
    Apple offers several layers, and the choice is mostly about what you are building and how long the framework will be supported. SceneKit is a high-level…

More in Media & graphics