Search for Indian languages and mixed scripts

Search for Indian languages fails in ways that English search rarely does, and three changes fix most of it. The same visible word can be stored as different sequences of Unicode code points, so every title and every query is first normalised to one form, NFCNFC (Normalization Form C): The Unicode normal form that decomposes a string and then recomposes it into one standard sequence, so canonically equivalent spellings of the same text become the same code points.. Words are then split by Unicode’s rules, which keep vowel signs, viramas and nuktas inside their word, not by a pattern written for English. And many people type Hindi, Tamil or Bengali in Latin letters, “kurta” or “saree”, or type queries in mixed scripts, so the index also keeps a Latin form of every word. Elasticsearch ships analysers for Hindi and Bengali but not for Tamil, Telugu, Marathi, Kannada or Malayalam, so those languages are assembled from Unicode components and checked against real queries.

Two spellings that look the same

Unicode encodes many letters of Indian scripts in more than one way. Devanagari क़ (qa) is one code point, U+0958, and also क followed by the nukta sign. Tamil, Malayalam and Bengali write some vowel signs in two parts, typed as one code point or as the two parts. UAX #15 calls such sequences canonically equivalentcanonical equivalence: Unicode's rule that two sequences of code points, such as a precomposed letter and the letter followed by its mark, represent the same text and should look and behave the same.: they should look and behave the same, yet a comparison of code points sees different strings (UAX #15). Which form a title arrives in depends on the keyboard, the app and any converter it passed through. The example stores four words in one form and searches in the other:

import unicodedata

# Four words, each typed in two ways that look the same on screen. Escapes keep the code points visible.
WORDS = [
    ("Hindi, 'pen'", "\u0958\u0932\u092e", "\u0915\u093c\u0932\u092e"),
    ("Tamil, 'flag'", "\u0b95\u0bca\u0b9f\u0bbf", "\u0b95\u0bc6\u0bbe\u0b9f\u0bbf"),
    ("Bengali, 'bottle'", "\u09ac\u09cb\u09a4\u09b2", "\u09ac\u09c7\u09be\u09a4\u09b2"),
    ("Malayalam, 'Kochi'", "\u0d15\u0d4a\u0d1a\u0d4d\u0d1a\u0d3f", "\u0d15\u0d46\u0d3e\u0d1a\u0d4d\u0d1a\u0d3f"),
]


def code_points(text):
    return " ".join(f"U+{ord(c):04X}" for c in text)


def nfc(text):
    return unicodedata.normalize("NFC", text)


for label, a, b in WORDS:
    print(f"{label}: {a}")
    print(f"  typed as  {code_points(a)}")
    print(f"  or as     {code_points(b)}")
    print(f"  equal? {a == b}   equal after NFC? {nfc(a) == nfc(b)}   NFC: {code_points(nfc(a))}")

# Titles stored in the first form, queries typed in the second: raw comparison, then NFC on both sides.
titles = [a for _, a, _ in WORDS]
queries = [b for _, _, b in WORDS]
raw_hits = sum(q in titles for q in queries)
index = {nfc(t) for t in titles}
nfc_hits = sum(nfc(q) in index for q in queries)
print(f"raw strings: {raw_hits} of {len(queries)} queries match; NFC on both sides: {nfc_hits} of {len(queries)}")

# Checks: both forms of every word normalise to one string, and normalising twice changes nothing.
assert all(nfc(a) == nfc(b) for _, a, b in WORDS)
assert all(nfc(nfc(a)) == nfc(a) for _, a, _ in WORDS)
print("checks passed")

Hindi, 'pen': क़लम
  typed as  U+0958 U+0932 U+092E
  or as     U+0915 U+093C U+0932 U+092E
  equal? False   equal after NFC? True   NFC: U+0915 U+093C U+0932 U+092E
Tamil, 'flag': கொடி
  typed as  U+0B95 U+0BCA U+0B9F U+0BBF
  or as     U+0B95 U+0BC6 U+0BBE U+0B9F U+0BBF
  equal? False   equal after NFC? True   NFC: U+0B95 U+0BCA U+0B9F U+0BBF
Bengali, 'bottle': বোতল
  typed as  U+09AC U+09CB U+09A4 U+09B2
  or as     U+09AC U+09C7 U+09BE U+09A4 U+09B2
  equal? False   equal after NFC? True   NFC: U+09AC U+09CB U+09A4 U+09B2
Malayalam, 'Kochi': കൊച്ചി
  typed as  U+0D15 U+0D4A U+0D1A U+0D4D U+0D1A U+0D3F
  or as     U+0D15 U+0D46 U+0D3E U+0D1A U+0D4D U+0D1A U+0D3F
  equal? False   equal after NFC? True   NFC: U+0D15 U+0D4A U+0D1A U+0D4D U+0D1A U+0D3F
raw strings: 0 of 4 queries match; NFC on both sides: 4 of 4
checks passed

Raw comparison finds none of the four. Normalisation Form C decomposes a string and recomposes it into one standard sequence, so both spellings become the same string and all four queries match. The Hindi word shows a detail worth knowing: NFC turns U+0958 into two code points, क and the nukta, because precomposed nukta letters are excluded from composition. Normalise when you index and again when you search, or the two sides drift apart. Step through two of the words:

Search for Indian languages chains NFC with further steps, each removing another source of mismatch:

• Unicode word splitting, as in the ICU tokenizer, keeps every vowel sign, virama and nukta inside its word.
• Folding turns native digits into 0 to 9, removes zero-width joiners and merges both spellings of Malayalam chillu letters.
• A Latin field stores every word transliterated, so queries in Latin letters or in mixed scripts find Devanagari titles.
• A phonetic key folds spelling variants such as kurta, kurtha and kurtaa into one term.
• Per-language analysers add stop words and stemming where Elasticsearch has them.
• Judged test queries per language and per script show whether each change helped.

Splitting words: the regex failure reproduced Pro: opens with a Pro, Premium or Ultimate pass