UTF-8, Unicode and text encodings in C++

Unicode gives every character a number, its code point, written U+0928 for the Devanagari letter न. UTF-8 stores each code point in 1 to 4 bytes, and it is the encoding of almost all text that programs exchange, from web pages and JSON to the string literals that GCC and Clang compile. A std::string holds bytes, so its size() counts bytes, not letters: the Hindi greeting नमस्ते has 18 bytes, 6 code points and 3 characters as a reader sees them, which Unicode calls grapheme clusters. C++ code that handles text therefore has to know which of the three it is counting. The standard library stores and copies UTF-8 correctly, but it has no function that counts code points or grapheme clusters, so this lesson writes a small decoder, uses the ICU library for grapheme clusters, shows what breaks when code treats bytes as characters, and explains how to print Unicode reliably on Windows, macOS and Linux.

Bytes, code points and grapheme clusters

UTF-8 is defined in RFC 3629. A code point below 128 is one byte, the same as ASCII; larger code points use a lead byte that says how many bytes follow, and continuation bytes that each carry 6 bits:

In hex, c is the single byte 63, é is C3 A9, € is E2 82 AC, the Devanagari न is E0 A4 A8, and 😀 is F0 9F 98 80.

The decoder below follows that table. It reads the lead byte, keeps its payload bits, and adds 6 bits from each continuation byte:

// Bytes and code points of UTF-8 text, with a small decoder written out in full.
#include <cstddef>
#include <print>
#include <string_view>
#include <vector>

// Decodes UTF-8 into code points. It assumes valid input (the exercise below adds the checks).
std::vector<char32_t> decode(std::string_view text) {
    std::vector<char32_t> out;
    for (std::size_t i = 0; i < text.size();) {
        const auto lead = static_cast<unsigned char>(text[i]);
        // The lead byte says how many bytes the character has:
        // 0xxxxxxx, 110xxxxx, 1110xxxx or 11110xxx.
        const int length = lead < 0x80 ? 1 : lead < 0xE0 ? 2 : lead < 0xF0 ? 3 : 4;
        char32_t cp = length == 1 ? lead : lead & (0x7F >> length);
        for (int k = 1; k < length; ++k) cp = (cp << 6) | (static_cast<unsigned char>(text[i + k]) & 0x3F);
        out.push_back(cp);
        i += length;
    }
    return out;
}

int main() {
    // \u00E9 is é as one code point;
    // e followed by \u0301, a combining acute accent, looks the same.
    for (std::string_view word : {"chai", "caf\u00E9", "cafe\u0301", "€", "😀", "नमस्ते"}) {
        std::println("{}: {} bytes, code points: {}", word, word.size(), decode(word).size());
    }
    std::print("नमस्ते is");
    for (char32_t cp : decode("नमस्ते")) std::print(" U+{:04X}", static_cast<unsigned>(cp));
    std::println("");
    std::println("the two spellings of café are equal: {}",
                 std::string_view("caf\u00E9") == std::string_view("cafe\u0301"));
}

from session import run

run("""
clang++ --version | head -n 1
clang++ -std=c++23 -Wall -Wextra code_points.cpp -o code_points && ./code_points
g++-16 --version | head -n 1
g++-16 -std=c++23 -Wall -Wextra code_points.cpp -o code_points && ./code_points
""")

$ clang++ --version | head -n 1
Apple clang version 21.0.0 (clang-2100.3.34.2)
$ clang++ -std=c++23 -Wall -Wextra code_points.cpp -o code_points && ./code_points
chai: 4 bytes, code points: 4
café: 5 bytes, code points: 4
café: 6 bytes, code points: 5
€: 3 bytes, code points: 1
😀: 4 bytes, code points: 1
नमस्ते: 18 bytes, code points: 6
नमस्ते is U+0928 U+092E U+0938 U+094D U+0924 U+0947
the two spellings of café are equal: false
$ g++-16 --version | head -n 1
g++-16 (Homebrew GCC 16.2.0) 16.2.0
$ g++-16 -std=c++23 -Wall -Wextra code_points.cpp -o code_points && ./code_points
chai: 4 bytes, code points: 4
café: 5 bytes, code points: 4
café: 6 bytes, code points: 5
€: 3 bytes, code points: 1
😀: 4 bytes, code points: 1
नमस्ते: 18 bytes, code points: 6
नमस्ते is U+0928 U+092E U+0938 U+094D U+0924 U+0947
the two spellings of café are equal: false

Every Devanagari letter and sign takes 3 bytes, so नमस्ते has 18 bytes and 6 code points:

The two cafés show why code points are still not characters. "caf\u00E9" ends with é as one code point, U+00E9. "cafe\u0301" ends with e and U+0301, a combining acute accent that the font draws on top of the e. They look the same and print the same, but they have different bytes, so == says they are different. Unicode Standard Annex #15 defines normalisation forms that make such strings comparable: in the composed form, NFC, both become the first spelling.

Grapheme clusters need a library

What a reader counts as one character is a grapheme cluster, defined by the rules of Unicode Standard Annex #29. Rule GB9c keeps Indic conjuncts together, so स्ते, a consonant, the virama sign, another consonant and a vowel sign, is one cluster of 4 code points. The C++ standard library has no function for this, so programs use a library such as ICU, which Unicode itself maintains:

// User-perceived characters (grapheme clusters, Unicode Standard Annex #29) counted with ICU,
// because the C++ standard library has no function for them.
#include <memory>
#include <print>
#include <string_view>
#include <unicode/brkiter.h>
#include <unicode/unistr.h>
#include <unicode/uversion.h>

int count_graphemes(std::string_view utf8) {
    UErrorCode status = U_ZERO_ERROR;
    std::unique_ptr<icu::BreakIterator> it(
        icu::BreakIterator::createCharacterInstance(icu::Locale::getRoot(), status));
    if (U_FAILURE(status)) return -1;
    const auto length = static_cast<int32_t>(utf8.size());
    it->setText(icu::UnicodeString::fromUTF8(icu::StringPiece(utf8.data(), length)));
    int count = 0;
    for (auto boundary = it->next(); boundary != icu::BreakIterator::DONE; boundary = it->next()) ++count;
    return count;
}

int main() {
    std::println("ICU {} (Unicode {})", U_ICU_VERSION, U_UNICODE_VERSION);
    for (std::string_view word : {"chai", "caf\u00E9", "cafe\u0301", "😀", "👍🏽", "नमस्ते", "क्षत्रिय"}) {
        std::println("{}: {} bytes, grapheme clusters: {}", word, word.size(), count_graphemes(word));
    }
}

from session import run

run("""
# ICU from Homebrew (brew install icu4c): its headers and library are in /opt/homebrew/opt/icu4c.
clang++ -std=c++23 -Wall -Wextra graphemes.cpp -I/opt/homebrew/opt/icu4c/include -L/opt/homebrew/opt/icu4c/lib -licuuc -o graphemes && ./graphemes
""")

# ICU from Homebrew (brew install icu4c): its headers and library are in /opt/homebrew/opt/icu4c.
$ clang++ -std=c++23 -Wall -Wextra graphemes.cpp -I/opt/homebrew/opt/icu4c/include -L/opt/homebrew/opt/icu4c/lib -licuuc -o graphemes && ./graphemes
ICU 78.3 (Unicode 17.0)
chai: 4 bytes, grapheme clusters: 4
café: 5 bytes, grapheme clusters: 4
café: 6 bytes, grapheme clusters: 4
😀: 4 bytes, grapheme clusters: 1
👍🏽: 8 bytes, grapheme clusters: 1
नमस्ते: 18 bytes, grapheme clusters: 3
क्षत्रिय: 24 bytes, grapheme clusters: 3

Both cafés are 4 clusters, the thumbs-up with a skin-tone modifier (two code points, 8 bytes) is 1, and नमस्ते and क्षत्रिय are 3 each. Count grapheme clusters when the number must match what a person sees, for example in a length limit for a name field; count bytes when the limit is about storage.

What breaks when bytes are treated as characters

Code written for ASCII, where one byte is one character, quietly breaks other text. Reversing a string byte by byte, reversing it code point by code point, and cutting it after 4 bytes:

// Three byte-level mistakes with UTF-8 text,
// and a code-point-level fix that is still not enough.
#include <algorithm>
#include <print>
#include <string>
#include <string_view>

// The length of the UTF-8 sequence that starts with this byte (valid input assumed).
int sequence_length(unsigned char lead) { return lead < 0x80 ? 1 : lead < 0xE0 ? 2 : lead < 0xF0 ? 3 : 4; }

// Reverses the order of the code points, keeping the bytes of each one together.
std::string reverse_code_points(std::string_view text) {
    std::string out;
    for (std::size_t i = 0; i < text.size();) {
        const auto n = static_cast<std::size_t>(sequence_length(static_cast<unsigned char>(text[i])));
        out.insert(0, text.substr(i, n));
        i += n;
    }
    return out;
}

int main() {
    for (std::string_view word : {"chai", "naïve", "नमस्ते"}) {
        std::string bytes(word);
        std::ranges::reverse(bytes);  // byte by byte: breaks every character longer than one byte
        std::println("{}: bytes reversed [{}], code points reversed [{}], first 4 bytes [{}]", word, bytes,
                     reverse_code_points(word), word.substr(0, 4));
    }
}

from session import run

run("""
clang++ -std=c++23 -Wall -Wextra reverse.cpp -o reverse && ./reverse
g++-16 -std=c++23 -Wall -Wextra reverse.cpp -o reverse && ./reverse
""")

$ clang++ -std=c++23 -Wall -Wextra reverse.cpp -o reverse && ./reverse
chai: bytes reversed [iahc], code points reversed [iahc], first 4 bytes [chai]
naïve: bytes reversed [ev��an], code points reversed [evïan], first 4 bytes [naï]
नमस्ते: bytes reversed [��त���ฤதਤ�], code points reversed [ेत्समन], first 4 bytes [न�]
$ g++-16 -std=c++23 -Wall -Wextra reverse.cpp -o reverse && ./reverse
chai: bytes reversed [iahc], code points reversed [iahc], first 4 bytes [chai]
naïve: bytes reversed [ev��an], code points reversed [evïan], first 4 bytes [naï]
नमस्ते: bytes reversed [��त���ฤதਤ�], code points reversed [ेत्समन], first 4 bytes [न�]

• Reversing the bytes swaps the bytes inside every multi-byte character, so the result is not valid UTF-8; the session shows the invalid bytes as �, U+FFFD, as most terminals do. For नमस्ते some reversed bytes even happen to form letters of other scripts.
• Reversing the code points keeps each character intact, but the vowel sign े now comes first, with nothing to attach to, and the virama joins the wrong letters. Reversing what a reader sees needs grapheme clusters.
• substr(0, 4) cuts after a byte count, which ended “naïve” cleanly but split म in the middle. Cut at a code-point boundary, or better at a grapheme boundary, whenever a limit applies to text.

The same applies to indexing: s[3] is a byte. Keep text as UTF-8 in std::string, search it with find (a UTF-8 search for a valid UTF-8 needle never matches in the middle of a character), and only decode when you need code points.

char8_t and u8 literals

C++20 added char8_t, a character type that holds UTF-8 code units, and changed the type of u8"…" literals to arrays of const char8_t (P0482R6). The aim is to say in the type that the bytes are UTF-8:

// u8"…" literals (C++20) have the type const char8_t[N]:
// UTF-8 by type, but no longer a char string.
#include <print>
#include <string>
#include <string_view>
#include <type_traits>

int main() {
    // 18 bytes and the final zero
    static_assert(std::is_same_v<decltype(u8"नमस्ते"), const char8_t(&)[19]>);

    const std::u8string hello = u8"नमस्ते";
    // std::print takes char text, so convert explicitly, one code unit at a time.
    const std::string bytes(hello.begin(), hello.end());
    std::println("{} has {} UTF-8 code units", bytes, hello.size());
}

// Since C++20 a u8 literal does not convert to const char*.
#include <cstdio>

int main() {
    const char* greeting = u8"नमस्ते";
    std::puts(greeting);
}

from session import run

run("""
clang++ -std=c++23 -Wall -Wextra char8.cpp -o char8 && ./char8
g++-16 -std=c++23 -Wall -Wextra char8.cpp -o char8 && ./char8
clang++ -std=c++23 -Wall -Wextra char8_error.cpp -o char8_error
g++-16 -std=c++23 -Wall -Wextra char8_error.cpp -o char8_error
# Built as C++17, where u8 literals were still arrays of char:
clang++ -std=c++17 -Wall -Wextra char8_error.cpp -o char8_error && ./char8_error
""")

$ clang++ -std=c++23 -Wall -Wextra char8.cpp -o char8 && ./char8
नमस्ते has 18 UTF-8 code units
$ g++-16 -std=c++23 -Wall -Wextra char8.cpp -o char8 && ./char8
नमस्ते has 18 UTF-8 code units
$ clang++ -std=c++23 -Wall -Wextra char8_error.cpp -o char8_error
char8_error.cpp:5:17: error: cannot initialize a variable of type 'const char *' with an lvalue of type 'const char8_t[19]'
    5 |     const char* greeting = u8"नमस्ते";
      |                 ^          ~~~~~~~~
1 error generated.
[exit status 1]
$ g++-16 -std=c++23 -Wall -Wextra char8_error.cpp -o char8_error
char8_error.cpp: In function 'int main()':
char8_error.cpp:5:28: error: invalid conversion from 'const char8_t*' to 'const char*' [-fpermissive]
    5 |     const char* greeting = u8"नमस्ते";
      |                            ^~~~~~~~
      |                            |
      |                            const char8_t*
[exit status 1]
# Built as C++17, where u8 literals were still arrays of char:
$ clang++ -std=c++17 -Wall -Wextra char8_error.cpp -o char8_error && ./char8_error
नमस्ते

The static_assert proves the type: 18 bytes and the terminating zero. The cost is compatibility: since C++20 a u8 literal no longer converts to const char*, so code like char8_error.cpp that compiled as C++17 is an error in C++20 and later, and std::print and most libraries take char text. In practice, most code keeps UTF-8 in ordinary std::string and char literals, which GCC and Clang encode as UTF-8 by default (GCC’s -finput-charset and -fexec-charset both default to UTF-8), and converts at the few places where an API asks for char8_t.

Printing Unicode on Windows, macOS and Linux

On macOS and Linux the terminal expects UTF-8, so writing the bytes of a UTF-8 string is enough: every session on this page printed Hindi correctly that way. Windows differs in two places, which these recorded runs on a Mac cannot show, so here is what the C++ standard and Microsoft’s documentation specify:

• The compiler. MSVC reads a source file without a byte order mark in the user’s code page unless you pass /utf-8, which sets both the source and the execution character set to UTF-8. Without it, a literal such as "नमस्ते" may be stored as different bytes. GCC and Clang need no option.
• The console. A Windows console shows bytes in its own code page. std::print and std::println (C++23, P2093R14) handle this: when string literals are UTF-8 and the output is a console, they write through the console’s Unicode API, so the text appears correctly whatever the code page. std::cout and printf send bytes, which the console decodes with its code page, often producing garbage unless the program first switches the console to UTF-8 (code page 65001) with SetConsoleOutputCP.

So the portable recipe is: keep sources in UTF-8, build with /utf-8 on MSVC, and print with std::println.

Asking for the encoding: std::text_encoding (C++26)

C++26 adds std::text_encoding (P1885R12), which names the encoding of string literals and of the environment the program runs in:

// std::text_encoding (C++26) names the encoding of string literals
// and of the program's environment.
#include <print>
#include <text_encoding>

int main() {
    std::println("string literals: {}", std::text_encoding::literal().name());
    std::println("environment:     {}", std::text_encoding::environment().name());
}

from session import run

run("""
# GCC 16 keeps std::text_encoding::environment() in libstdc++exp, its library of experimental features.
g++-16 -std=c++26 -Wall -Wextra text_encoding.cpp -o show_encoding -lstdc++exp && ./show_encoding
# The same program in the plain C locale:
LC_ALL=C ./show_encoding
clang++ -std=c++2c -Wall -Wextra text_encoding.cpp -o show_encoding
""")

# GCC 16 keeps std::text_encoding::environment() in libstdc++exp, its library of experimental features.
$ g++-16 -std=c++26 -Wall -Wextra text_encoding.cpp -o show_encoding -lstdc++exp && ./show_encoding
string literals: UTF-8
environment:     UTF-8
# The same program in the plain C locale:
$ LC_ALL=C ./show_encoding
string literals: UTF-8
environment:     US-ASCII
$ clang++ -std=c++2c -Wall -Wextra text_encoding.cpp -o show_encoding
text_encoding.cpp:4:10: fatal error: 'text_encoding' file not found
    4 | #include <text_encoding>
      |          ^~~~~~~~~~~~~~~
1 error generated.
[exit status 1]

With GCC 16 the literals are UTF-8 and the environment is UTF-8 in this session’s UTF-8 locale, but US-ASCII in the plain C locale: the encoding of input is a property of where the program runs, not of the program. GCC 16 keeps environment() in libstdc++exp, its library of experimental features, so the command adds -lstdc++exp. Apple clang 21’s library has no <text_encoding> header yet; upstream libc++ lists the paper as complete in release 23.

Exercise: count and validate UTF-8

Write utf8_length, which counts the code points of valid UTF-8 text, and utf8_is_valid, which accepts only well-formed UTF-8 as RFC 3629 defines it: every sequence complete, in its shortest form, without surrogates (U+D800 to U+DFFF) and nothing above U+10FFFF. The checks must all print ok:

// Exercise: write utf8_length and utf8_is_valid.
// Build and run this file: every check must say "ok".
#include <cstddef>
#include <print>
#include <string_view>

// The number of code points in valid UTF-8 text.
std::size_t utf8_length(std::string_view text) {
    return text.size();  // counts bytes: replace this
}

// Whether text is well-formed UTF-8 as RFC 3629 defines it: every sequence complete,
// in its shortest form, no surrogates (U+D800 to U+DFFF) and nothing above U+10FFFF.
bool utf8_is_valid(std::string_view text) {
    (void)text;
    return true;  // replace this
}

int failures = 0;

void check(bool passed, std::string_view what) {
    std::println("{}  {}", passed ? "ok    " : "FAILED", what);
    if (!passed) ++failures;
}

int main() {
    check(utf8_length("chai") == 4, "ASCII: one code point per byte");
    check(utf8_length("नमस्ते") == 6, "Devanagari: 6 code points in 18 bytes");
    check(utf8_length("😀!") == 2, "an emoji (4 bytes) and an exclamation mark");
    check(utf8_is_valid("नमस्ते"), "valid Devanagari text");
    check(!utf8_is_valid("\xE0\xA4"), "a sequence cut short");
    check(!utf8_is_valid("\x80"), "a continuation byte with no lead byte");
    check(!utf8_is_valid("\xC0\xAF"), "an overlong form of '/'");
    check(!utf8_is_valid("\xED\xA0\x80"), "an encoded surrogate, U+D800");
    check(!utf8_is_valid("\xF4\x90\x80\x80"), "a code point above U+10FFFF");
    std::println("{} of 9 checks failed", failures);
    return failures == 0 ? 0 : 1;
}

from session import run

run("""
clang++ -std=c++23 -Wall -Wextra -fsanitize=address,undefined utf8_exercise.cpp -o utf8_exercise && ./utf8_exercise
""")

$ clang++ -std=c++23 -Wall -Wextra -fsanitize=address,undefined utf8_exercise.cpp -o utf8_exercise && ./utf8_exercise
ok      ASCII: one code point per byte
FAILED  Devanagari: 6 code points in 18 bytes
FAILED  an emoji (4 bytes) and an exclamation mark
ok      valid Devanagari text
FAILED  a sequence cut short
FAILED  a continuation byte with no lead byte
FAILED  an overlong form of '/'
FAILED  an encoded surrogate, U+D800
FAILED  a code point above U+10FFFF
7 of 9 checks failed
[exit status 1]

The starter counts bytes and accepts everything, so two checks pass by luck and seven fail. First hint: for utf8_length, count the bytes that are not continuation bytes, the ones where (b & 0xC0) != 0x80. For utf8_is_valid, decode each sequence as the decoder above does, and then check the length against the input and the code point against the smallest value that needs that many bytes. RFC 3629 explains why overlong forms such as \xC0\xAF (a slash in two bytes) must be rejected: they have been used to slip characters past security checks.

Self-check

Interview questions

Why does std::string::size() not return the number of characters in a Hindi word? std::string stores bytes, and size() counts them. In UTF-8 every Devanagari letter and sign takes 3 bytes, so नमस्ते has a size() of 18. It has 6 code points, because the conjunct स्ते is written with 4 of them, and a reader sees 3 characters, the grapheme clusters of Unicode Standard Annex #29. Counting code points needs a UTF-8 decoder and counting grapheme clusters needs a library such as ICU; the standard library offers neither.

Key takeaways

• A code point is a number, U+0000 to U+10FFFF; UTF-8 stores it in 1 to 4 bytes; a grapheme cluster is what a reader sees as one character and may be several code points.
• std::string::size() counts bytes: 18 for नमस्ते, which has 6 code points and 3 grapheme clusters.
• Visually identical strings can differ in bytes (é as one code point or as e plus a combining accent); normalise before comparing.
• Reversing, cutting or indexing by bytes breaks UTF-8 text; work with code points or, for what readers see, grapheme clusters from a library such as ICU.
• u8 literals are char8_t since C++20 and no longer convert to const char*.
• On Windows, build with /utf-8 and print with std::println, which writes to the console through its Unicode API.
• C++26 std::text_encoding names the literal and environment encodings: GCC 16 has it (with -lstdc++exp for environment()), Apple clang 21 does not yet.