Why do count, unicodeScalars, utf16 and utf8 give different numbers for the same string?

27.5.3 · L2 · Interview

Predict first

What does this print?

let samples = ["cafe\u{301}", "🇮🇳", "👨‍👩‍👧", "a\r\nb", "é"]
for s in samples {
    print(s.count, s.unicodeScalars.count, s.utf16.count, s.utf8.count)
}
print("café" == "cafe\u{301}", "ß".uppercased())
let s = "Hello, World"
let i = s.index(s.startIndex, offsetBy: 5)
let sub = s[..<i]
print(sub, type(of: sub), String(sub).count)
let comma = s.firstIndex(of: ",")!
print(s[s.index(after: comma)...].count, s[comma...].first!)
print(s.prefix(3), s.suffix(3), s.dropFirst(7))
  • 4 5 5 6 / 1 2 4 8 / 1 5 8 18 / 3 4 4 4 / 1 1 1 2 / true SS / Hello Substring 5 / 6 , / Hel rld World
  • 5 5 5 6 / 2 2 4 8 / 3 5 8 18 / 4 4 4 4 / 1 1 1 2 / false SS / Hello String 5 / 6 , / Hel rld World
  • 4 5 5 6 / 1 1 2 4 / 1 1 2 4 / 3 3 3 3 / 1 1 1 2 / true SS / Hello Substring 5 / 6 , / Hel rld World

Mental model. Counting letters, strokes and ink dots in the same word gives different totals, and each total is correct for its own definition.

String.count counts extended grapheme clusters, what a reader sees as one character. unicodeScalars counts code points, utf16 counts UTF-16 units (what NSString.length reports) and utf8 counts bytes. "e" plus a combining accent is 1 + 1 clusters... in the run "cafe\u{301}" is 4 characters, 5 scalars, 5 UTF-16 units and 6 bytes, while the precomposed "é" is 1, 1, 1, 2. A flag is 1 character made of 2 scalars, a family emoji is 1 character of 5 scalars (three people joined by two zero-width joiners), and "\r\n" is a single character.

Equality uses canonical equivalence, so "café" == "cafe\u{301}" is true although the bytes differ. "ß".uppercased() is "SS", so case mapping can change length. A slice like s[..<i] is a Substring that shares the original's storage; convert with String(sub) to store it. Never mix indices between different strings. Counting characters is O(n) because clusters must be found (Q51).

Grapheme clusters, scalars, UTF-16 units and UTF-8 bytes are four different measures.

Common traps

  • Using utf16.count as a character count. Emoji break it.
  • Keeping a Substring long term. It keeps the whole original string alive.

They might follow up with

  • What does NSString.length measure? UTF-16 units.

Related questions

  • 1.9.1 Why is String not a RandomAccessCollection of Character? Why is string[i + 1] not O(1)?
  • 1.1.7 What is copy-on-write? Which types use it?

← How do dictionary default subscripts and nil assignment behave when…   What do nested optionals and nil assignments do: Int??, dictionaries… →