Unicode, UTF-16 and Indian scripts

"नमस्ते".length() is 6 in Java, although a Hindi reader sees three characters: न, म and स्ते. Java strings are sequences of chars, 16-bit units of the UTF-16 encoding, and a char is not a character as people count them. length(), charAt and substring work in chars, and even StringBuilder.reverse, which never splits a surrogate pair, knows nothing about the characters readers see, so all of them miscount or break text in Indian scripts and emoji.

You will measure strings in three units on JDK 25, step through the rules that join code points into what a reader sees, reverse and shorten text without breaking it, normalise strings that look identical but are not, and follow text to the edges of a program, where it becomes bytes. The exercise counts and reverses what a reader sees.

A string has three lengths: chars, code points and clusters

Three units are at work in every Java string. A char is one UTF-16 code unitUTF-16 code unit: One 16-bit number of the UTF-16 encoding of text, the unit a Java char holds and String.length() counts. Most characters are one code unit; characters beyond the first 65,536, such as many emoji, take two. More in “char, boolean, casting and conversions”, the unit length() counts. A code pointcode point: The number Unicode gives one character, written U+ and at least four hexadecimal digits, such as U+0928 for न. A code point above U+FFFF, such as most emoji, takes two UTF-16 code units in a Java string. More in “Strings: first look and equality” is one entry of the Unicode table, such as U+0928 DEVANAGARI LETTER NA: a char holds one code point of the first 65,536, the Basic Multilingual PlaneBasic Multilingual Plane: The first 65,536 code points, U+0000 to U+FFFF, which hold the letters of almost every modern script, Indian scripts included. Each takes one char in a Java string; a code point above it takes a surrogate pair., while a code point above U+FFFF, as most emoji are, takes two chars, a surrogate pairsurrogate pair: Two UTF-16 code units, a high one from D800 to DBFF and a low one from DC00 to DFFF, that together encode one character beyond U+FFFF, such as an emoji. length() counts both; codePointCount counts one. More in “char, boolean, casting and conversions”. A grapheme clustergrapheme cluster: What a reader counts as one character: a letter with its marks, a conjunct such as स्ते, or an emoji with its modifiers. Java finds them with the regular expression \X or with BreakIterator.getCharacterInstance(). is what a reader counts as one character: a consonant with its vowel sign, a conjunct such as स्ते, or an emoji with its skin tone. The figure counts one word and one emoji in each unit:

Counts.java measures eight strings in all three units, finds their clusters in two ways, and adds the bytes each string takes in UTF-8:

import java.text.BreakIterator;

// Four ways to measure the length of a string, on words in four Indian scripts, two emoji and two spellings of café.
void main() {
    String[][] samples = {
        {"Hindi", "नमस्ते"},
        {"Tamil", "தமிழ்"},
        {"Kannada", "ಕನ್ನಡ"},
        {"Bengali", "বাংলা"},
        {"thumbs up, medium skin tone", "👍🏽"},
        {"flag of India", "🇮🇳"},
        {"café, é as one code point", "caf\u00E9"},
        {"café, e + combining accent", "cafe\u0301"},
    };
    IO.println("chars  code points  \\X  BreakIterator  UTF-8 bytes  text");
    for (String[] sample : samples) {
        String s = sample[1];
        IO.println(String.format("%5d  %11d  %2d  %13d  %11d  %s (%s)", s.length(), s.codePointCount(0, s.length()),
                clusters(s).size(), breakIteratorCount(s), s.getBytes(StandardCharsets.UTF_8).length, s, sample[0]));
    }
    IO.println();
    for (String word : new String[] {"नमस्ते", "தமிழ்", "ಕನ್ನಡ", "বাংলা"}) {
        IO.println(word + " -> " + clusters(word));
    }
}

// The extended grapheme clusters of s, as the regular expression \X finds them: what a reader counts as characters.
List<String> clusters(String s) {
    List<String> out = new ArrayList<>();
    Matcher m = Pattern.compile("\\X").matcher(s);
    while (m.find()) {
        out.add(m.group());
    }
    return out;
}

// The same count from java.text.BreakIterator, the older API for character boundaries.
int breakIteratorCount(String s) {
    BreakIterator it = BreakIterator.getCharacterInstance(Locale.ROOT);
    it.setText(s);
    int count = 0;
    while (it.next() != BreakIterator.DONE) {
        count++;
    }
    return count;
}

chars  code points  \X  BreakIterator  UTF-8 bytes  text
    6            6   3              3           18  नमस्ते (Hindi)
    5            5   3              3           15  தமிழ் (Tamil)
    5            5   4              4           15  ಕನ್ನಡ (Kannada)
    5            5   2              2           15  বাংলা (Bengali)
    4            2   1              1            8  👍🏽 (thumbs up, medium skin tone)
    4            2   1              1            8  🇮🇳 (flag of India)
    4            4   4              4            5  café (café, é as one code point)
    5            5   4              4            6  café (café, e + combining accent)

नमस्ते -> [न, म, स्ते]
தமிழ் -> [த, மி, ழ்]
ಕನ್ನಡ -> [ಕ, ನ್, ನ, ಡ]
বাংলা -> [বাং, লা]

• For the Indian words, chars and code points agree: every Devanagari, Tamil, Kannada and Bengali code point lies in the Basic Multilingual Plane, so each takes one char. The trouble is grapheme clusters: नमस्ते is 6 code points and 3 clusters.
• Emoji lie above U+FFFF, so each code point takes two chars. 👍🏽 is the thumbs-up sign followed by a skin-tone modifier: 2 code points, 4 chars, 1 cluster. A flag is two regional indicatorOne of 26 code points, U+1F1E6 to U+1F1FF, that stand for the letters A to Z. Two of them in a row spell a country code, which phones and browsers draw as that country's flag. letters, I and N for India.
• The regular expression \X and java.text.BreakIterator counted the same number of clusters in every string. Both follow the default rules of Unicode Standard Annex #29, as of Unicode 16.0, the version Java 25’s Character class implements.
• UTF-8, the encoding of most files and web pages, takes 3 bytes for each of these Indian code points, so नमस्ते is 18 bytes on disk. Inside the JVM a string with any character beyond Latin-1 is stored as UTF-16, 2 bytes per char, while plain English text takes 1 byte per char (JEP 254).

Combining marks and conjuncts join code points into clusters

The last four lines of the run list the clusters themselves. Inspect.java shows why they fall where they do: it prints the code points of two strings with their Unicode names and general categories:

// The code points inside two strings: their numbers, their Unicode names and their general categories.
void main() {
    show("नमस्ते");
    show("👍🏽");
}

void show(String s) {
    IO.println(s + ": " + s.length() + " chars, " + s.codePointCount(0, s.length()) + " code points");
    s.codePoints().forEach(cp -> IO.println(String.format("  U+%04X  %-2s  %d char%s  %s",
            cp, category(cp), Character.charCount(cp), Character.charCount(cp) == 1 ? " " : "s", Character.getName(cp))));
}

// The two-letter Unicode general category of the code points used here.
String category(int cp) {
    return switch (Character.getType(cp)) {
        case Character.OTHER_LETTER -> "Lo";
        case Character.NON_SPACING_MARK -> "Mn";
        case Character.COMBINING_SPACING_MARK -> "Mc";
        case Character.OTHER_SYMBOL -> "So";
        case Character.MODIFIER_SYMBOL -> "Sk";
        default -> "?";
    };
}

नमस्ते: 6 chars, 6 code points
  U+0928  Lo  1 char   DEVANAGARI LETTER NA
  U+092E  Lo  1 char   DEVANAGARI LETTER MA
  U+0938  Lo  1 char   DEVANAGARI LETTER SA
  U+094D  Mn  1 char   DEVANAGARI SIGN VIRAMA
  U+0924  Lo  1 char   DEVANAGARI LETTER TA
  U+0947  Mn  1 char   DEVANAGARI VOWEL SIGN E
👍🏽: 4 chars, 2 code points
  U+1F44D  So  2 chars  THUMBS UP SIGN
  U+1F3FD  Sk  2 chars  EMOJI MODIFIER FITZPATRICK TYPE-4

Letters are Lo, “other letter”, and the virama and the vowel sign E are Mn, non-spacing marks: combining markscombining mark: A code point that attaches to the character before it instead of standing alone, such as a vowel sign, the virama or an accent. Its Unicode category is M (Mn, Mc or Me), so Character.isLetter is false for it., which attach to the character before them. The default rules of Unicode Standard Annex #29 never break before a mark, so a mark joins the cluster of the character before it, and in some scripts a virama between two consonants joins them into one conjunctconjunct: Two or more consonants joined by a virama and written as one shape, such as स्ते in नमस्ते. Unicode's default rules keep a conjunct in one grapheme cluster only in scripts whose virama is a conjunct linker.. Step through how those rules group नमस्ते, one code point at a time:

The skin-tone modifier works the same way: rule GB9 attaches it to the thumb before it, so 👍🏽 is one cluster of two code points. Marks also explain two everyday bugs. Character.isLetter is false for a vowel sign, and a word pattern made of letters alone, [\p{L}]+, cuts पुणे into pieces: include marks with [\p{L}\p{M}]+.

reverse() keeps surrogate pairs but breaks clusters

Reversing text is a common exercise and a common bug. Reverse.java reverses three strings twice, with StringBuilder.reverse() and by clusters:

// Reversing text: by chars, by code points, and by grapheme clusters.
void main() {
    for (String s : new String[] {"नमस्ते", "👍🏽 ok", "🇮🇳"}) {
        IO.println(s);
        IO.println("  StringBuilder.reverse():   " + new StringBuilder(s).reverse());
        IO.println("  reversed by \\X clusters:   " + reverseClusters(s));
    }
}

String reverseClusters(String s) {
    List<String> clusters = new ArrayList<>();
    Matcher m = Pattern.compile("\\X").matcher(s);
    while (m.find()) {
        clusters.add(m.group());
    }
    Collections.reverse(clusters);
    return String.join("", clusters);
}

नमस्ते
  StringBuilder.reverse():   ेत्समन
  reversed by \X clusters:   स्तेमन
👍🏽 ok
  StringBuilder.reverse():   ko 🏽👍
  reversed by \X clusters:   ko 👍🏽
🇮🇳
  StringBuilder.reverse():   🇳🇮
  reversed by \X clusters:   🇮🇳

StringBuilder.reverse() keeps each surrogate pair in its order, so it never produces half an emoji, but it reverses everything else code point by code point. The Hindi vowel sign े ended up at the front, attached to nothing, and the skin-tone modifier moved in front of the thumb it belonged to. The flag shows the danger best:

A flag is two regional indicator letters, the country’s code, so swapping them spells another country. Collecting the \X matches and reversing that list keeps every cluster whole, as the second line of each result shows.

Two spellings of one letter are equal only after normalising

Unicode often has two ways to write what looks like the same text: a precomposed letter, or a base letter followed by combining marks. equals compares code points, so it calls such strings different. java.text.Normalizer converts text to a standard form first, which is called normalisationnormalisation: Rewriting text in one standard form, so that two spellings that mean the same, such as é as one code point or as e and an accent, become the same code points. NFC, the usual form for storage, composes where Unicode allows it.: NFC composes where Unicode allows it, NFD takes every composed character apart. The program compares three such pairs:

import java.text.Normalizer;

// Pairs of strings that look the same but are made of different code points, before and after normalisation.
// The escapes spell out the code points, so no editor can change them by accident.
void main() {
    compare("Hindi qa as one code point", "\u0958", "qa as ka + nukta", "\u0915\u093C");
    compare("Tamil ko, one vowel sign", "\u0B95\u0BCA", "ko, the sign in two parts", "\u0B95\u0BC6\u0BBE");
    compare("café, é as one code point", "caf\u00E9", "café, e + combining accent", "cafe\u0301");
}

void compare(String nameA, String a, String nameB, String b) {
    String nfcA = Normalizer.normalize(a, Normalizer.Form.NFC);
    String nfcB = Normalizer.normalize(b, Normalizer.Form.NFC);
    IO.println(a + " (" + nameA + ") and " + b + " (" + nameB + ")");
    IO.println("  code points: " + hex(a) + " and " + hex(b) + "; equals: " + a.equals(b));
    IO.println("  after NFC:   " + hex(nfcA) + " and " + hex(nfcB) + "; equals: " + nfcA.equals(nfcB));
}

String hex(String s) {
    List<String> parts = new ArrayList<>();
    s.codePoints().forEach(cp -> parts.add(String.format("U+%04X", cp)));
    return String.join(" ", parts);
}

क़ (Hindi qa as one code point) and क़ (qa as ka + nukta)
  code points: U+0958 and U+0915 U+093C; equals: false
  after NFC:   U+0915 U+093C and U+0915 U+093C; equals: true
கொ (Tamil ko, one vowel sign) and கொ (ko, the sign in two parts)
  code points: U+0B95 U+0BCA and U+0B95 U+0BC6 U+0BBE; equals: false
  after NFC:   U+0B95 U+0BCA and U+0B95 U+0BCA; equals: true
café (café, é as one code point) and café (café, e + combining accent)
  code points: U+0063 U+0061 U+0066 U+00E9 and U+0063 U+0061 U+0066 U+0065 U+0301; equals: false
  after NFC:   U+0063 U+0061 U+0066 U+00E9 and U+0063 U+0061 U+0066 U+00E9; equals: true

Step through what NFC does to each spelling of the three pairs:

• The Hindi letter क़ (qa) exists as one code point, U+0958, and as क followed by the nukta sign, U+0915 U+093C. NFC turns both into the two-code-point form, because Unicode excludes U+0958 from composition: normalising does not always make a string shorter.
• The Tamil vowel sign ொ (o) has a single code point, U+0BCA, and a two-part spelling, U+0BC6 U+0BBE. NFC composes the two parts into one.
• é composed (U+00E9) and e with a combining acute accent (U+0065 U+0301) become equal after NFC as well.

Normalise both sides with the same form before comparing, searching or using text as a map key. Text typed on different keyboards, pasted from PDF files or produced by different operating systems mixes the forms, and NFC, the form most text already arrives in, is the usual choice for storage.

Cut text at a cluster boundary, not at a char index

A notification, an SMS preview or a database column often holds only so many characters, and the obvious substring(0, n) counts chars:

// A notification shows only the first 11 characters of a message. Three ways to cut it there.
void main() {
    String message = "Thank you 🙏🏽 for the help";

    String byChars = message.substring(0, 11);
    IO.println("substring(0, 11):  [" + byChars + "]  ends with a lone surrogate: "
            + Character.isHighSurrogate(byChars.charAt(byChars.length() - 1)));

    String byCodePoints = message.substring(0, message.offsetByCodePoints(0, 11));
    IO.println("11 code points:    [" + byCodePoints + "]  " + byCodePoints.codePoints().count() + " code points");

    IO.println("11 clusters:       [" + firstClusters(message, 11) + "]");
}

// The first n grapheme clusters of s, or all of s when it has fewer.
String firstClusters(String s, int n) {
    Matcher m = Pattern.compile("\\X").matcher(s);
    int end = 0;
    for (int i = 0; i < n && m.find(); i++) {
        end = m.end();
    }
    return s.substring(0, end);
}

substring(0, 11):  [Thank you ?]  ends with a lone surrogate: true
11 code points:    [Thank you 🙏]  11 code points
11 clusters:       [Thank you 🙏🏽]

Step through where each of the three cuts falls in the message:

• substring(0, 11) ended between the two chars of 🙏 and kept only its high surrogate, which no encoding can write: the output shows ? in its place. deleteCharAt and charAt loops cause the same damage.
• Counting code points with offsetByCodePoints kept the emoji whole but dropped its skin tone, a separate code point.
• Counting grapheme clusters with \X kept 🙏🏽 together. When a limit is measured in what people see, cut at a cluster boundary; when it is measured in bytes, as in some database columns, count UTF-8 bytes and still cut at a cluster boundary below the limit.

Encodings turn chars into bytes at the edges

Inside a program a string is a sequence of chars; files, terminals and networks carry bytes, and a character encodingcharacter encoding: The rule that turns characters into bytes and back, also called a charset: UTF-8 takes 1 to 4 bytes per code point, UTF-16 2 or 4, and ISO-8859-1 one byte for each of its 256 characters. Java's default is UTF-8 since JDK 18. turns one into the other:

import java.io.PrintStream;

// The same six code points as bytes in three encodings, read back with the wrong one, and printed through an
// encoding that has no Devanagari at all.
void main() {
    String hello = "नमस्ते";
    IO.println("file.encoding " + System.getProperty("file.encoding") + ", stdout.encoding " + System.getProperty("stdout.encoding"));

    byte[] utf8 = hello.getBytes(StandardCharsets.UTF_8);
    IO.println("UTF-8: " + utf8.length + " bytes, UTF-16BE: " + hello.getBytes(StandardCharsets.UTF_16BE).length
            + " bytes, UTF-16 (with a byte order mark): " + hello.getBytes(StandardCharsets.UTF_16).length + " bytes");

    String wrong = new String(utf8, StandardCharsets.ISO_8859_1);
    IO.println("UTF-8 bytes read as ISO-8859-1: " + wrong.length() + " chars, starting " + hexStart(wrong));

    PrintStream latin1 = new PrintStream(System.out, true, StandardCharsets.ISO_8859_1);
    latin1.println("through an ISO-8859-1 stream: " + hello);
}

String hexStart(String s) {
    List<String> parts = new ArrayList<>();
    for (int i = 0; i < 3; i++) {
        parts.add(String.format("U+%04X", (int) s.charAt(i)));
    }
    return String.join(" ", parts);
}

file.encoding UTF-8, stdout.encoding UTF-8
UTF-8: 18 bytes, UTF-16BE: 12 bytes, UTF-16 (with a byte order mark): 14 bytes
UTF-8 bytes read as ISO-8859-1: 18 chars, starting U+00E0 U+00A4 U+00A8
through an ISO-8859-1 stream: ??????

Step through the word’s way out of the JVM, and back with the wrong charset:

• Since JDK 18, file.encoding is UTF-8 on every platform unless you change it (JEP 400), so Files.readString, new String(bytes) and javac itself read UTF-8 by default. Source files with Hindi string literals compile the same everywhere.
• The same six code points take 18 bytes in UTF-8, 12 in UTF-16, and 14 when UTF-16 starts with a byte order mark.
• Decoding UTF-8 bytes with the wrong charset does not fail: ISO-8859-1 turned 18 bytes into 18 Latin characters starting with à, the familiar mojibakemojibake: Garbled text that appears when bytes written in one encoding are read with another, such as न for the UTF-8 bytes of न read as ISO-8859-1. The bytes are intact; the decoding is wrong, and no error is raised. of a text file opened with the wrong encoding.
• System.out encodes with stdout.encoding, which the JVM takes from the host environment, such as the console’s settings; it was UTF-8 here. A stream whose charset has no Devanagari prints ??????, which is what a console set to a legacy code page shows. Set the console to UTF-8, or start Java with -Dstdout.encoding=UTF-8, the one value the System documentation allows for it on the command line, when output must carry Indian scripts.

Practise: count and reverse what a reader sees

Visible.java asks for two methods built on \X. The checks use Hindi, Tamil, an emoji with a skin tone, two flags and a decomposed é. As it starts, both methods return nothing useful, so five of the six checks fail:

// Your turn: count and reverse what a reader sees. visibleLength("नमस्ते") is 3, and reverseVisible("नमस्ते") is
// "स्तेमन": the conjunct स्ते stays whole. Use the regular expression \X, which matches one extended grapheme cluster.
int visibleLength(String s) {
    return 0;    // Write your code here.
}

String reverseVisible(String s) {
    return "";   // Write your code here.
}

int failed = 0;
int total = 0;

void check(String s, int length, String reversed) {
    total++;
    int gotLength = visibleLength(s);
    String gotReversed = reverseVisible(s);
    if (gotLength != length || !gotReversed.equals(reversed)) {
        failed++;
        IO.println("FAIL \"" + s + "\": expected " + length + " and \"" + reversed + "\", got " + gotLength + " and \"" + gotReversed + "\"");
    }
}

void main() {
    check("नमस्ते", 3, "स्तेमन");
    check("தமிழ்", 3, "ழ்மித");
    check("👍🏽 ok", 4, "ko 👍🏽");
    check("🇮🇳🇯🇵", 2, "🇯🇵🇮🇳");
    check("cafe\u0301", 4, "e\u0301fac");    // é written as e and a combining accent
    check("", 0, "");
    IO.println(failed + " of " + total + " checks failed");
    if (failed > 0) {
        System.exit(1);
    }
}

FAIL "नमस्ते": expected 3 and "स्तेमन", got 0 and ""
FAIL "தமிழ்": expected 3 and "ழ்மித", got 0 and ""
FAIL "👍🏽 ok": expected 4 and "ko 👍🏽", got 0 and ""
FAIL "🇮🇳🇯🇵": expected 2 and "🇯🇵🇮🇳", got 0 and ""
FAIL "café": expected 4 and "éfac", got 0 and ""
5 of 6 checks failed

You are done when it prints 0 of 6 checks failed. Hints: Pattern.compile("\\X").matcher(s) finds one cluster per call of find(); count the matches for visibleLength, and for reverseVisible add each group() to a list, reverse it with Collections.reverse and join it with String.join("", list). Try codePointCount and StringBuilder.reverse() first, to see which checks they fail.

Six ideas are left to test, from the three units to encodings. The practice picks the one you are least sure of first and explains each choice the moment you make it:

Interview questions

Fresher · Why can "👍".length() be 2 in Java, and how do you count the characters a user sees?

A Java string is a sequence of chars, and a char is a 16-bit UTF-16 code unit. Code points above U+FFFF, which include most emoji, are stored as a surrogate pair of two chars, so "👍".length() is 2 while codePointCount(0, s.length()) is 1. Even code points are not what users count: 👍🏽 is a thumbs-up and a skin-tone modifier, 2 code points that display as one symbol, and Hindi नमस्ते has 6 code points but 3 visible characters, because vowel signs and the virama combine with the consonants around them. What a user sees as one character is an extended grapheme cluster, and Java finds them with the regular expression \X or with BreakIterator.getCharacterInstance(). A good answer adds that substring and charAt work in chars, so cutting text by index can split a surrogate pair or a cluster.

A good interviewer pushes the same idea one step further. Answer these two, then read why each option holds or fails:

For practice at length, the module bank adds two names that two systems measure differently, and two usernames that look identical:

Key takeaways

• length(), charAt, substring and StringBuilder count UTF-16 chars; codePointCount, codePoints() and offsetByCodePoints count code points; \X and BreakIterator find grapheme clusters, what readers count.
• Indian scripts fit one char per code point, but vowel signs, the virama and the nukta are combining marks, so नमस्ते is 6 chars and 3 clusters; the default rules join conjuncts in only six Indian scripts as of Unicode 16.0, so Kannada ನ್ನ splits into two clusters.
• StringBuilder.reverse() keeps surrogate pairs but breaks clusters, and turns the Indian flag into Nicaragua’s; reverse a list of \X matches instead.
• Visually identical strings can differ in code points: compare them after Normalizer.normalize(s, Form.NFC).
• Cut text at grapheme cluster boundaries; substring(0, n) can leave a lone surrogate.
• UTF-8 is Java’s default charset since JDK 18, decoding with the wrong charset gives mojibake without an error, and System.out follows stdout.encoding, which the host environment, such as the console, decides.