Why can two visually identical strings differ in bytes, and how do equality, normalisation and case conversion behave for international text?

24.9.1 · L3 · Interview

Predict first

How do composed and decomposed café compare, and what do case and grapheme counts do?

import Foundation

let composed = "caf\u{E9}", decomposed = "cafe\u{301}"
print("equal:", composed == decomposed, "chars:", composed.count, decomposed.count, "scalars:", composed.unicodeScalars.count, decomposed.unicodeScalars.count, "utf8:", composed.utf8.count, decomposed.utf8.count)
print("one Set entry:", Set([composed, decomposed]).count, "utf8 equal:", composed.utf8.elementsEqual(decomposed.utf8))
print("NFC/NFD scalars:", decomposed.precomposedStringWithCanonicalMapping.unicodeScalars.count, composed.decomposedStringWithCanonicalMapping.unicodeScalars.count)
print("ligature equal:", "\u{FB01}" == "fi", "compat:", "\u{FB01}".precomposedStringWithCompatibilityMapping)
print("fold:", "résumé".folding(options: [.diacriticInsensitive, .caseInsensitive], locale: nil), "|", "Straße".uppercased(), "|", "I".lowercased(with: Locale(identifier: "tr")), "i".uppercased(with: Locale(identifier: "tr")))
print("grapheme counts:", "🇯🇵".count, "👨‍👩‍👧".count, "👨‍👩‍👧".utf16.count, "\u{1112}\u{1161}\u{11AB}".count)
print("prefix of e+accent:", "e\u{301}".hasPrefix("e"), "e\u{301}".contains("e"))

equal: true chars: 4 4 scalars: 4 5 utf8: 5 6 / one Set entry: 1 utf8 equal: false / NFC/NFD scalars: 4 5 / …
equal: false …
scalars: 4 4

Mental model. Two people can spell a word “cafe” plus an accent mark or write the accented letter in one stroke. A reader sees the same word; a machine comparing the strokes does not, unless it first converts to one handwriting.

Unicode lets the same text be written in different ways. “é” can be one scalar (U+00E9) or an “e” followed by a combining accent (U+0065 U+0301). The check shows both spellings of “café”: Swift calls them equal and counts four characters each (it compares by canonical equivalence and counts grapheme clusters), yet they have 4 versus 5 scalars and 5 versus 6 UTF-8 bytes, a Set holds one entry for both, and comparing the utf8 views says they differ. So equality inside Swift is forgiving, but anything that works on bytes is not: a database column, a filename on a file system, a hash, a signature or a server that compares raw bytes can treat the two as different. Normalise text before storing or hashing it (precomposedStringWithCanonicalMapping for NFC, which is what the check shows reducing 5 scalars to 4), and treat NFC as the wire form. Compatibility normalisation is a different, lossy operation: the “ﬁ” ligature is not equal to “fi” in Swift but becomes “fi” under the compatibility mapping, which is useful for search keys only.

Case and searching depend on language as well. "résumé" folded with diacritic- and case-insensitive options becomes “resume” (the basis of accent-insensitive search); “ß” upper-cases to “SS”; and in Turkish the dotless and dotted i convert differently (“I” lower-cases to “ı” and “i” upper-cases to “İ” with the Turkish locale), so use the locale-aware lowercased(with:) for user-visible text and plain conversion for identifiers. Remember also that count is grapheme clusters, not what users or servers mean by length: a flag and a family emoji are one character each but several scalars (the family is 8 UTF-16 units), and Korean syllables built from conjoining jamo count as one. Slicing by character index is fine; by byte offset is not. Characters made of an “e” plus an accent do not match a search for “e” with the default string search (hasPrefix and contains both return false in the check), which is correct for text but surprising for developers (Q52, Q475).

Swift compares canonically but byte-level views differ; normalise before hashing.

Common traps

Hashing or signing a string without normalising it. The same text typed on two devices can hash differently.
Using count as a byte or server length limit. Use the encoding that the limit is defined in.

They might follow up with

What do file systems do? Not run here; assume nothing and normalise names you compare.
Is NFC or NFD better? NFC is the usual interchange form; Apple file APIs have historically used decomposed forms.

Why can two visually identical strings differ in bytes, and how do equality, normalisation and case conversion behave for international text?

Predict first

Common traps

They might follow up with

Related questions