How do you split text into words, count words or search in languages that do not use spaces?

24.9.2 · L3 · Learning

Predict first

How many words does a space split find versus the tokenizer?

import Foundation
import NaturalLanguage

let t = NLTokenizer(unit: .word)
for (lang, s) in [("en", "Hello wide world"), ("zh", "我喜欢学习编程"), ("ja", "私はプログラミングが好きです"), ("th", "ฉันชอบเรียนเขียนโปรแกรม")] {
    t.string = s; t.setLanguage(NLLanguage(lang))
    var words: [String] = []
    t.enumerateTokens(in: s.startIndex..<s.endIndex) { r, _ in words.append(String(s[r])); return true }
    print(lang, "split on spaces:", s.split(separator: " ").count, "tokens:", words.count, words.joined(separator: "|"))
}
  • en split on spaces: 3 tokens: 3 / zh split on spaces: 1 tokens: 4 / ja … 1 … 6 / th … 1 … 4
  • Every language 1
  • zh tokens: 7

Mental model. English puts gaps between words like beads on a string. Chinese is a string of beads without the gaps, and you need to know the language to tell where one bead ends.

Splitting on spaces works for English and fails for Chinese, Japanese and Thai, which write words without spaces between them. Use the NaturalLanguage tokenizer, NLTokenizer(unit: .word), with the language set (or detected). The check splits one sentence in each of four languages: English has three space-separated words and three tokens, while the Chinese sentence “我喜欢学习编程” has one space-separated chunk but four tokens (我 | 喜欢 | 学习 | 编程), the Japanese one has six (the particles は and が are separate tokens) and the Thai one four. The tokenizer uses language models, so the exact boundaries can differ between OS versions: treat them as good, not exact, and never persist them as a key. For lower-level needs, enumerateSubstrings(in:options:) with .byWords and .localized uses the locale’s word rules, and the text system already applies line-breaking rules that allow a break between CJK characters.

The same lesson applies to every operation that quietly assumes English: word counts and reading-time estimates, truncation (cut at a grapheme boundary, never a byte or UTF-16 offset, or you split an emoji or an accented letter), search (index tokens rather than space-separated words, and use a diacritic-insensitive comparison, Q475), spell checking and autocorrect (UITextChecker with the language), name and initial extraction (use PersonNameComponentsFormatter) and character limits (a limit of 140 characters means very different amounts of meaning in Chinese and English). If you need text-to-token for a server, send raw text and let the server segment with the same model. The tokenizer and the line-breaking behaviour of a label on a device were not run; the check ran on the Mac frameworks.

Spaces only work for some languages; the tokenizer uses language models.

Common traps

  • Counting words with split(separator: " "). Chinese, Japanese and Thai report one word per sentence.
  • Truncating by UTF-16 offset. You can cut an emoji or combining sequence in half.

They might follow up with

  • Can I detect the language first? Yes, NLLanguageRecognizer; short text is unreliable, so prefer the user’s known language.
  • Do I need this for UI text? No, labels wrap correctly; it matters for your own logic.

Related questions

  • 1.9.2 Character vs Unicode.Scalar vs UTF-8 code unit?
  • 24.4.2 What changes by locale beyond strings: sorting and searching, the first day of the week, calendars, names and…
  • 24.9.1 Why can two visually identical strings differ in bytes, and how do equality, normalisation and case…

← Why can two visually identical strings differ in bytes, and how do…   What regional defaults should you read from the Locale instead of… →