Strings, bytes and runes: UTF-8 done right

len("नमस्ते") is 18 in Go, although the word is written with 6 code pointscode point: The number Unicode gives a character, written U+00E9 for é. A Go rune holds one code point, and UTF-8 encodes it in one to four bytes.: Go strings hold bytes, and the UTF-8UTF-8: The encoding of Go source code and string literals: each Unicode character takes one to four bytes, so 'A' takes one byte and '₹' three. len of a string counts these bytes. More in “Basic types: numbers, strings, bools, conversion” encoding takes three bytes for each of these Devanagari code points. Code that counts, indexes or cuts text by bytes is right for English and wrong for most of the world’s writing.

You will see the bytes of a string, loop over it by byte and by runerune: Go's type for one Unicode character, an alias of int32 that holds its code point. A rune such as ₹ takes several bytes in UTF-8, and ranging over a string gives one rune per pass. More in “Control flow: if, for, range and switch”, and convert it to []byte and []rune knowing what each conversion copies. The programs here all ran with Go 1.27.1, and each shows the output that run produced.

A string is a read-only sequence of UTF-8 bytes

The specification defines a string as a sequence of bytes, and Go source files are UTF-8, so a string literal holds the UTF-8 bytes of its text. This program prints four strings with their lengths and their bytes in hex:

// bytes.go: a string is a sequence of bytes, and UTF-8 uses one to four of them per code point.
package main

import "fmt"

func main() {
	for _, s := range []string{"go", "héllo", "€5", "नमस्ते"} {
		fmt.Printf("%-8q len %2d  bytes % x\n", s, len(s), s)
	}
	s := "héllo"
	fmt.Println(s[0], s[1], s[2]) // indexing gives bytes, as numbers
}

"go"     len  2  bytes 67 6f
"héllo"  len  6  bytes 68 c3 a9 6c 6c 6f
"€5"     len  4  bytes e2 82 ac 35
"नमस्ते" len 18  bytes e0 a4 a8 e0 a4 ae e0 a4 b8 e0 a5 8d e0 a4 a4 e0 a5 87
104 195 169

ASCII letters took one byte each, so “go” has length 2. “é” took two bytes, c3 a9, and “€” three, so “héllo” has length 6 and “€5” length 4. Each Devanagari code point of “नमस्ते” took three bytes, 18 in all. len counts bytes, always, and indexing a string gives a byte, not a character: s[1] of “héllo” is 195, the first half of é.

Strings are also immutable. The specification says that once a string is created, its contents cannot change, and the compiler refuses to assign to a byte of one:

// immutable.go: a byte of a string cannot be assigned.
package main

import "fmt"

func main() {
	s := "hello"
	s[0] = 'H'
	fmt.Println(s)
}

# command-line-arguments
./immutable.go:8:2: cannot assign to s[0] (neither addressable nor a map index expression)

cannot assign to s[0] (neither addressable nor a map index expression): to change text, build a new string. That rule is what lets Go strings be shared freely, sliced without copying and used as map keys.

Indexing reads bytes; range reads runes

A loop over the indexes visits bytes. A range loop over a string decodes it instead: each pass gives the byte offset where a code point starts and that code point as a rune:

// runes.go: a byte loop and a range loop over the same string, and the count of runes.
package main

import (
	"fmt"
	"unicode/utf8"
)

func main() {
	s := "héllo"
	for i := 0; i < len(s); i++ {
		fmt.Printf("%d:%x ", i, s[i]) // one byte per pass
	}
	fmt.Println()
	for i, r := range s {
		fmt.Printf("%d:%c ", i, r) // one rune per pass, with the byte offset where it starts
	}
	fmt.Println()
	fmt.Println(len(s), utf8.RuneCountInString(s))
	fmt.Println(len("नमस्ते"), utf8.RuneCountInString("नमस्ते"))
}

0:68 1:c3 2:a9 3:6c 4:6c 5:6f 
0:h 1:é 3:l 4:l 5:o 
6 5
18 6

The byte loop printed six bytes, c3 and a9 among them. The range loop printed five runes, and its offsets jumped from 1 to 3, because é covers offsets 1 and 2. Step through the range loop pass by pass:

So utf8.RuneCountInString is the count of runes, 5 for “héllo” and 6 for “नमस्ते”, while len stays the count of bytes. A rune is an int32 that holds one code point; the specification calls the values a range loop produces over a string exactly that, Unicode code points. A byte that is not valid UTF-8 comes out of the loop as the replacement characterreplacement character: U+FFFD, the code point that stands in for bytes that are not valid UTF-8. A range loop over a string gives it for each invalid byte., U+FFFD.

Converting to []byte and []rune copies

A string’s bytes cannot change, so converting it to a slice that can change must copy them. This program changes the copies and prints the string afterwards:

// convert.go: []byte and []rune conversions copy, so changing them leaves the string alone.
package main

import "fmt"

func main() {
	s := "héllo"

	b := []byte(s) // a copy of the 6 bytes
	b[0] = 'H'
	fmt.Println(string(b), s)

	r := []rune(s) // a copy, decoded: one int32 per code point
	r[1] = 'e'
	fmt.Println(string(r), s, len(r))

	fmt.Println(string(rune(233)), string([]byte{0xc3, 0xa9})) // both are "é"
}

Héllo héllo
hello héllo 5
é é

Changing b[0] gave “Héllo” and left s as it was, and changing r[1] gave “hello”. A []rune is the way to read code points by position, r[1] being é, at four bytes a rune: 20 bytes for these 5 runes against 6 for the string.

The strings package does the rest of the work on text, and four ideas come with it. strings.Builder builds a string piece by piece without copying everything each time, which a loop of += does on every pass. Functions such as strings.Cut and strings.Fields take text apart without counting indexes, and strings.EqualFold compares two strings without regard to case. Cutting text to a length is where bytes and runes matter most: s[:n] counts bytes, so it can end in the middle of a rune. And a character on screen, a grapheme clustergrapheme cluster: What a reader sees as one character, which can be several code points: a letter and an accent, an emoji with a modifier, or a Devanagari conjunct. Unicode Standard Annex 29 defines the boundaries., can take several runes, as a letter with an accent or an emoji with a skin tone does, so even a rune count can split one.

Building and taking apart text with the strings package, measured Pro: opens with a Pro, Premium or Ultimate pass

Invalid UTF-8, characters made of several runes, and safe truncation Pro: opens with a Pro, Premium or Ultimate pass

Practise

Practice covers the bytes of a string, byte and rune loops, and conversions, opening with the idea your earlier answers showed least sure. Every option comes with the reason it is right or the step it skips:

Interview questions

Warm-up (fresher): Why is len(“héllo”) 6 in Go?

Because len counts bytes, and Go strings hold UTF-8: h, l, l and o take one byte each and é takes two, c3 a9. The word has 5 runes, which utf8.RuneCountInString reports, and a range loop over it makes 5 passes.

Three longer questions of the module bank build on this lesson:

Key takeaways

• Go strings are read-only sequences of bytes, UTF-8 for text, so len counts bytes: 6 for “héllo” and 18 for “नमस्ते”.
• Indexing a string gives a byte; a range loop gives runes and the byte offset where each starts, and utf8.RuneCountInString counts runes.
• []byte(s) and []rune(s) copy, so changing the slice never changes the string; []rune reads code points by position.