Endianness, strings and text in memory

A 64-bit integer takes eight bytes of memory, and a string takes one byte per ASCII character, three per Devanagari character and four per most emoji. The order in which those bytes land, and how a byte count differs from a character count, decides whether a program reads back what it wrote. On every mainstream x86-64, AArch64 and RISC-V system the value 0x1122334455667788 is stored with 0x88 at the lowest address and 0x11 at the highest; the Devanagari word for namaste takes 18 bytes to hold six code points.

Below, you write those bytes with Python’s struct module, swap them with two short idioms, name the byte-swap instructions each ISA offers, and see why len(s.encode('utf-8')) is not always len(s). It builds on hexadecimalhexadecimal: Base 16, written with the digits 0 to 9 and A to F. Each digit stands for four bits, so 0x2C is the byte 0010 1100, which is 44; the prefix 0x marks a hexadecimal number in C, Python and most assemblers. More in “Bits, bytes, words and hexadecimal” and byte sizes from the earlier bits-bytes-hex lesson.

Little-endian stores the lowest byte first

A multi-byte integer in a register is one number; in memory it is a run of bytes at consecutive addresses, and some rule has to decide which byte goes where. The rule every mainstream x86-64, AArch64 and RISC-V system uses is called the little-endianlittle-endian: The byte-order rule in which the lowest-value byte of a multi-byte value sits at the lowest address. Every mainstream x86-64, AArch64 and RISC-V system is little-endian in practice. rule: the lowest-value byte of the number sits at the lowest address. The reversed rule, with the highest-value byte at the lowest address, is called big-endianbig-endian: The byte-order rule in which the highest-value byte of a multi-byte value sits at the lowest address. Internet protocols call this 'network byte order' and use it on the wire. and is mostly seen in network protocols and in a few older systems. The choice is called the value’s byte orderbyte order: The order the bytes of a multi-byte value take in memory, from the base address upwards. The two orders in use are little-endian (low byte first) and big-endian (high byte first)., or its endiannessendianness: The rule a computer follows to place the bytes of a multi-byte value in memory. Little-endian puts the lowest-value byte at the lowest address; big-endian puts the highest-value byte there..

A Python one-liner shows both orders for the same number. The struct module’s pack turns an integer into a bytes object under a format letter: '<' means little-endian and '>' means big-endian, and 'Q' is an unsigned 64-bit integer:

"""Show the byte order of 0x1122334455667788 under little-endian and big-endian packing.

Every mainstream computer that runs this track's three ISAs stores integers little-endian: the
least significant byte sits at the lowest address. The struct module packs a value under a chosen
byte order ('<' little, '>' big), and printing the bytes side by side shows the reversal.
"""

import struct

value = 0x1122334455667788

little = struct.pack("<Q", value)
big = struct.pack(">Q", value)


def hexpairs(data: bytes) -> str:
    return " ".join(f"{b:02x}" for b in data)


print(f"value             = 0x{value:016x}")
print(f"little-endian ('<Q')  = [{hexpairs(little)}]")
print(f"big-endian    ('>Q')  = [{hexpairs(big)}]")
print()
print("Memory layout, little-endian (byte at address + 0 on the left):")
print("  offset: 0  1  2  3  4  5  6  7")
print("  byte:   " + "  ".join(f"{b:02x}" for b in little))

value             = 0x1122334455667788
little-endian ('<Q')  = [88 77 66 55 44 33 22 11]
big-endian    ('>Q')  = [11 22 33 44 55 66 77 88]

Memory layout, little-endian (byte at address + 0 on the left):
  offset: 0  1  2  3  4  5  6  7
  byte:   88  77  66  55  44  33  22  11

The two lines of hex are mirror images of each other. The little-endian output starts with the low byte 0x88 at offset 0 and ends with the high byte 0x11 at offset 7; the big-endian output starts with 0x11 and ends with 0x88. Walk the eight bytes of the little-endian store one at a time to see where each one lands:

The flow starts with the whole 64-bit value, states the little-endian rule, and then places the first byte, the middle six and the last byte in turn. The ending matches the Python output byte for byte: 88 77 66 55 44 33 22 11.

MySmartCoPilot’s hex viewer shows any file byte by byte, and its data inspector reads the bytes at the cursor as a 64-bit integer in either byte order, so the same eight bytes can be read both ways.

Byte swap turns one order into the other

A program that reads a value stored on a big-endian machine, or over a network that uses network byte orderThe order of bytes used by Internet protocols for multi-byte integers, which is big-endian., and runs on a little-endian processor has to reverse the bytes before it uses the value. The reversal is called a byte swapbyte swap: The operation that reverses the bytes of a multi-byte value, turning its little-endian form into the big-endian form or back. x86-64 does it with bswap, AArch64 with rev and RISC-V with rev8 under the Zbb extension., and every ISA this track teaches has a one-instruction form of it:

• x86-64 has bswap for a 32-bit or 64-bit register and movbe to load or store a 2-, 4- or 8-byte value with its bytes reversed in one step. Intel documents both in Volume 2 of the SDM.
• AArch64 has rev, which reverses every byte of its register in either width (rev x0, x0 for eight bytes, rev w0, w0 for four), rev32 to reverse the bytes inside each 32-bit half of a 64-bit register, and rev16 to reverse the bytes inside each 16-bit half. The Arm A64 ISA guide names each one.
• RISC-V has rev8 in the Zbb bit-manipulation extension, which reverses every byte of an XLEN-wide register. Zbb is not part of the RV64GC base, so a general RV64 build needs -march=rv64gc_zbb and a matching run-time to use rev8; without Zbb the swap is a sequence of shifts and masks. rev8 works on the full register width only, so the specification’s own hint for a 4-byte swap on RV64 is rev8 followed by a shift right by 32.

Each ISA gets a listing of its own below, and all three load the same eight bytes from memory before reversing them. The files are listings rather than programs, so this lesson does not assemble them; the Python example after them reproduces the same swap end to end.

# x86-64 byte reversal, in GNU as Intel syntax. bswap reverses a whole register;
# movbe reverses the bytes of a memory operand while it moves it.
        .intel_syntax noprefix
        mov     rax, qword ptr [rdi]    # rax = the eight bytes at rdi, in memory order
        bswap   rax                     # reverse all eight: 0x1122334455667788 -> 0x8877665544332211
        bswap   eax                     # the 32-bit form reverses the low four bytes only
        movbe   rax, qword ptr [rdi]    # load eight big-endian bytes, reversed on the way in
        movbe   qword ptr [rdi], rax    # store rax with its bytes reversed

// AArch64 byte reversal. rev takes every byte of its register; rev32 and rev16
// reverse the bytes inside each 32-bit or 16-bit part of it.
        ldr     x0, [x1]        // x0 = the eight bytes at x1, in memory order
        rev     x0, x0          // reverse all eight: 0x1122334455667788 -> 0x8877665544332211
        rev     w2, w0          // the 32-bit form reverses four bytes
        rev32   x0, x0          // reverse the bytes inside each 32-bit half of x0
        rev16   w0, w0          // reverse the bytes inside each 16-bit half of w0

# RV64 byte reversal. rev8 belongs to the Zbb extension, so this file assembles
# only with -march=rv64gc_zbb; an RV64GC build has no byte-reverse instruction.
        ld      a0, 0(a1)       # a0 = the eight bytes at a1, in memory order
        rev8    a0, a0          # reverse all eight: 0x1122334455667788 -> 0x8877665544332211
        srai    a0, a0, 32      # rev8 is full-width only: shift back by 32 for a 4-byte swap

In Python, two short idioms do the same swap at the byte level. The first writes the integer out in big-endian order with int.to_bytes and reads the same bytes back as little-endian with int.from_bytes; the second does the equivalent round trip through struct:

"""Reverse the byte order of a 64-bit value two ways: int.to_bytes/from_bytes and struct.

A byte swap turns a value stored one way into the same bits read the other way: it is the
software form of x86's bswap, Arm's rev and RISC-V's rev8 (from the Zbb extension). Both
versions here produce the same number from the same input, which makes the trade-off between
the two Python idioms visible: to_bytes/from_bytes names the two orders in words; struct uses
format letters.
"""

import struct

value = 0x1122334455667788


def swap_with_int(x: int) -> int:
    """Write x out in big-endian order, then read the same bytes as little-endian."""
    big_bytes = x.to_bytes(8, byteorder="big")
    return int.from_bytes(big_bytes, byteorder="little")


def swap_with_struct(x: int) -> int:
    """Pack x as big-endian, unpack the same bytes as little-endian."""
    big_bytes = struct.pack(">Q", x)
    (reversed_value,) = struct.unpack("<Q", big_bytes)
    return reversed_value


a = swap_with_int(value)
b = swap_with_struct(value)

print(f"input                = 0x{value:016x}")
print(f"swap via to_bytes    = 0x{a:016x}")
print(f"swap via struct      = 0x{b:016x}")
print(f"two idioms agree     = {a == b}")

round_trip = swap_with_int(a)
print(f"swap of the swap     = 0x{round_trip:016x}  (back to the input)")

input                = 0x1122334455667788
swap via to_bytes    = 0x8877665544332211
swap via struct      = 0x8877665544332211
two idioms agree     = True
swap of the swap     = 0x1122334455667788  (back to the input)

Both idioms print 0x8877665544332211, the exact mirror of the input, and the two agree. The second swap takes the output back to 0x1122334455667788, because reversing the order twice restores it. On the processor side, the same idea runs in one instruction on x86-64 (bswap) and AArch64 (rev), and in one instruction on RISC-V where the CPU has Zbb.

Strings are bytes in memory, with ASCII at the base

A string in memory is a run of bytes, one byte per character for English text and more bytes for most other scripts. The ASCIIASCII: A 7-bit character code defined in RFC 20 that assigns 128 codes to the English letters, digits and punctuation; each character fits in one byte. UTF-8 encodes the same code points as ASCII in one byte each. standard, documented in RFC 20, assigns a 7-bit code to each of 128 characters: the uppercase letters A through Z take values 0x41 to 0x5A, the lowercase letters 0x61 to 0x7A, the digits 0x30 to 0x39 and the space 0x20. Those codes fit in one byte each, so the 5-character string “hello” takes five bytes: 0x68 0x65 0x6C 0x6C 0x6F.

Two ways to mark the end of a string live together on real systems. A C-style string ends with a null terminatornull terminator: A single byte of value 0 appended to a string to mark its end. The C library's strlen, puts and printf all walk forward until they meet one; a length-prefixed call like write(2) uses a separate length instead., a single byte of value 0 that is not part of the text; strlen, puts and printf all walk forward until they meet one. A length-prefixed string, like what write(2) takes, carries the length as a separate number, so the content may hold a 0 byte.

Each assembler has its own string directives that lay these bytes out at a known label. The two listings below are not assembled by this lesson; they name what each line produces:

; NASM lays out exactly the bytes the line lists, so the terminator is yours to add.
        section .rodata
msg:    db "hello", 0           ; 6 bytes: 68 65 6c 6c 6f 00
text:   db "hello"              ; 5 bytes, with nothing marking the end
banner: db "ping", 10           ; 5 bytes: 70 69 6e 67 0a, a newline and no NUL

# GNU as string directives. .ascii writes the characters and nothing else;
# .asciz and .string append one zero byte.
        .section .rodata
plain:  .ascii  "hello"         # 5 bytes: 68 65 6c 6c 6f
zeroed: .asciz  "hello"         # 6 bytes: the same five, then 00
same:   .string "hello"         # the same six bytes as .asciz
line:   .ascii  "hello\n"       # 6 bytes: the newline is the single byte 0a

The directives differ only in what they put after the characters. For the same five letters, this is what each line leaves at its label:

The last row is the one that catches people out: an escape inside the string counts once, so \n is the single byte 0x0A rather than two characters, and .ascii still appends nothing after it. That is why strlen on a label written with .ascii walks past its end, while write(2) with a length of 5 is content with either form. MySmartCoPilot’s text to binary converter lists the bytes of each character of a string, so a label’s bytes can be predicted before assembling.

UTF-8 encodes code points in one to four bytes

ASCII covers 128 characters, which is enough for an English keyboard. For every other script, and for the emoji, the modern answer is Unicode: each character has a number called its code pointcode point: The number Unicode assigns to a character, written as U+XXXX. UTF-8 and other encodings turn code points into bytes; U+0928 is the Devanagari letter NA, for example., written as U+XXXX, and a program chooses an encoding to turn code points into bytes. The encoding this section follows is UTF-8UTF-8: The byte encoding of Unicode defined in the Unicode Standard section 3.9 and RFC 3629. It stores each code point in one to four bytes under a leading-bits rule, with ASCII code points taking one byte each., which the Unicode Standard defines in section 3.9 and RFC 3629 specifies for Internet use. UTF-8 encodes each code point in one to four bytes under a simple range rule: U+0000 to U+007F take one byte each and match ASCII, U+0080 to U+07FF take two bytes, U+0800 to U+FFFF take three, and U+10000 to U+10FFFF take four. The first byte of a multi-byte sequence names the length by its leading bits (110…, 1110…, 11110…) and the bytes after it start with 10, which marks a continuation.

A Python one-liner compares len(s), which counts code points, with len(s.encode('utf-8')), which counts storage bytes. Three samples make the gap visible: a 5-letter ASCII word, the Devanagari word for namaste, and the rocket emoji.

"""Count UTF-8 bytes beside code points on three samples: ASCII, Devanagari, an emoji.

Byte length and character count are the same thing only for ASCII text; UTF-8 encodes any
Unicode code point in 1 to 4 bytes, so a buffer sized by len(s) is not big enough for a non-
ASCII string, and a byte index into the encoded form is not a character index.
"""

samples = [
    ("hello", "hello"),
    ("namaste (devanagari)", "\u0928\u092e\u0938\u094d\u0924\u0947"),
    ("rocket emoji", "\U0001f680"),
]


def hexpairs(data: bytes) -> str:
    return " ".join(f"{b:02x}" for b in data)


print(f"{'sample':24}  {'code points':>12}  {'bytes':>5}  utf-8 bytes")
print("-" * 70)
for label, text in samples:
    encoded = text.encode("utf-8")
    print(f"{label:24}  {len(text):>12}  {len(encoded):>5}  [{hexpairs(encoded)}]")

print()
print("Each code point of 'namaste' in U+XXXX form:")
for ch in "\u0928\u092e\u0938\u094d\u0924\u0947":
    print(f"  U+{ord(ch):04X}  {len(ch.encode('utf-8'))} bytes")

print()
print("Rocket emoji code point and its UTF-8 bytes:")
rocket = "\U0001f680"
print(f"  U+{ord(rocket):04X}  {len(rocket.encode('utf-8'))} bytes")

sample                     code points  bytes  utf-8 bytes
----------------------------------------------------------------------
hello                                5      5  [68 65 6c 6c 6f]
namaste (devanagari)                 6     18  [e0 a4 a8 e0 a4 ae e0 a4 b8 e0 a5 8d e0 a4 a4 e0 a5 87]
rocket emoji                         1      4  [f0 9f 9a 80]

Each code point of 'namaste' in U+XXXX form:
  U+0928  3 bytes
  U+092E  3 bytes
  U+0938  3 bytes
  U+094D  3 bytes
  U+0924  3 bytes
  U+0947  3 bytes

Rocket emoji code point and its UTF-8 bytes:
  U+1F680  4 bytes

The ASCII word reports 5 code points and 5 bytes; namaste reports 6 code points and 18 bytes; the rocket emoji reports 1 code point and 4 bytes. Walk through the Devanagari word to see how its six code points become eighteen bytes:

The flow starts with the word, lists its six code points from U+0928 to U+0947, states the three-bytes-per-code-point rule for the U+0800 to U+FFFF range, encodes U+0928 into 0xE0 0xA4 0xA8, multiplies six by three to reach 18 bytes, and reads the full encoded form back as e0 a4 a8 e0 a4 ae e0 a4 b8 e0 a5 8d e0 a4 a4 e0 a5 87. The character count and the byte count agree only when the text is pure ASCII; above U+007F they always disagree. MySmartCoPilot’s Unicode character inspector shows the code point and the UTF-8 bytes of every character pasted into it.

Practise

The self-check covers the four ideas above, from the byte at offset 0 to the bytes of an emoji; it asks first about the one you have answered least well:

Interview questions

Why is a Devanagari string wider in bytes than in characters, and where does that gap show up in code?

Unicode gives every character a code point, a number such as U+0928, and UTF-8 is the rule that turns code points into bytes. The rule packs code points U+0000 to U+007F into one byte each (which is why English ASCII text takes the same room in UTF-8 as it does in ASCII) and spreads code points above that over two, three or four bytes under a leading-bits scheme that the Unicode Standard sets out in section 3.9. Devanagari characters sit in U+0900 to U+097F, which falls in the three-byte range, so a Devanagari word of six code points takes 18 bytes; an emoji above U+FFFF, such as the rocket at U+1F680, takes four bytes. The gap shows up as soon as a program sizes a buffer by len(s) and then copies in s.encode('utf-8'): the destination is too small and the write stops in the middle of a character, which the next reader cannot decode. It also shows up when a program steps through a string one byte at a time and treats each step as a character, which overcounts and splits characters across its own boundaries. The fix on both sides is the same: size the buffer by len(s.encode('utf-8')), which is the exact byte count, and walk the bytes with a UTF-8 reader (or skip the continuation bytes that start with 10 when counting code points by hand).

The module bank holds two harder cases for this lesson: a length prefix read from a socket in the wrong byte order, and a database column sized for characters that UTF-8 stores in up to four bytes. Their rows give the first step and the follow-ups:

Key takeaways

• Every mainstream x86-64, AArch64 and RISC-V system stores a multi-byte integer with its lowest byte at the lowest address: struct.pack('<Q', 0x1122334455667788) produces 88 77 66 55 44 33 22 11, which is the memory the processor sees.
• A byte swap reverses that order in one instruction: x86-64 has bswap for a register and movbe for load or store, AArch64 has rev, rev16 and rev32, and RISC-V has rev8 under the Zbb extension, which the RV64GC base does not include.
• ASCII assigns 7-bit codes to 128 characters, so "hello" is five bytes; a null terminator is a single byte of value 0 that marks the end for C-style strings, while write(2) and other length-prefixed interfaces carry the length as a separate number.
• UTF-8 encodes each Unicode code point in one to four bytes, so len(s) (code points) and len(s.encode('utf-8')) (bytes) are the same only for pure ASCII; namaste has 6 code points and 18 bytes, the rocket emoji has 1 code point and 4 bytes.
• A buffer sized by character count is too small for most non-English text, and a byte-by-byte scan is not the same as a code-point scan: size by the encoded length, and decode or skip continuation bytes when counting characters.