Process startup: argc, argv, envp and auxv

When ./echo_args one two three starts on Linux, its first instruction finds 4 at the address in rsp: the argument count, followed by four pointers to the argument strings, the environment and a vector of facts from the kernel. No main has run and no C library is loaded; a program that links neither reads everything from that block.

This lesson reads argc, argv and envp from the stack at _start on three ISAs, walks the auxiliary vectorauxiliary vector: The list of type and value pairs the Linux kernel places after the environment pointers at start-up, such as AT_PAGESZ with the page size. A pair whose type is AT_NULL, 0, ends it. to the page size, and builds the same block in a model you can run. It then shows the block macOS builds and hands to main in a recorded run, and what the C start-up code does before main on Linux.

At the first instruction, the stack pointer points at argc

The x86-64 psABI fixes the initial process stackinitial process stack: The block the kernel writes at the stack pointer before a program's first instruction: argc, the argv pointers and a NULL, the envp pointers and a NULL, the auxiliary vector, then the strings. word by word: argc at rsp, the argument pointers from rsp + 8, a zero word, the environment pointers, another zero, then the auxiliary vector’s pairs of eight-byte words, a null entry, and above all of that the strings themselves. The kernel’s ELF loader writes the block in that order on every architecture, in create_elf_tables in fs/binfmt_elf.c, so AArch64 and RISC-V programs find the same layout at sp. The argument vectorargument vector (argv): The array of pointers to a program's command-line arguments, argv[0] being the program's name as it was run. It is ended by a NULL pointer, at argv[argc]. needs no count to walk, because argv[argc] is NULL; the environmentenvironment variable: A NAME=value string a program inherits from the process that started it. The kernel passes the list as envp, an array of pointers ended by a NULL. list has no count at all, only its terminating NULL.

The model below builds that block for ./echo_args one two three with the environment A=1 B=2 and a short auxiliary vector in the kernel’s order, then walks it the way the assembly in this lesson does. Its offsets are the ABI’s; its pointer values are labels, not addresses:

"""A model of the Linux initial process stack: built in the order the x86-64 psABI and the kernel's ELF loader lay
it out, then walked the way pagesz-x86-64.asm walks it.

The model starts ./echo_args one two three with the environment A=1 B=2 and a short auxiliary vector in the kernel's
order (a real one has about twenty entries). Every word is 8 bytes. The addresses are relative to the stack pointer,
and the AT_HWCAP value is an example; the order, the sizes and the 16-byte alignment of the stack pointer are the
ABI's.
"""

ARGV = ["./echo_args", "one", "two", "three"]
ENVP = ["A=1", "B=2"]
AT_NULL, AT_PHDR, AT_PAGESZ, AT_ENTRY, AT_HWCAP, AT_RANDOM, AT_EXECFN = 0, 3, 6, 9, 16, 25, 31
NAMES = {0: "AT_NULL", 3: "AT_PHDR", 6: "AT_PAGESZ", 9: "AT_ENTRY", 16: "AT_HWCAP", 25: "AT_RANDOM", 31: "AT_EXECFN"}


def build() -> tuple[list[tuple[str, object]], int]:
    """Returns the stack as (label, value) words from the stack pointer upwards, and the size of the string area."""
    strings = [s + "\0" for s in ARGV + ENVP]
    string_bytes = sum(len(s) for s in strings) + 16  # the strings plus AT_RANDOM's 16 random bytes
    words: list[tuple[str, object]] = [("argc", len(ARGV))]
    words += [(f"argv[{i}]", f'-> "{s}"') for i, s in enumerate(ARGV)]
    words.append((f"argv[{len(ARGV)}]", "NULL"))
    words += [(f"envp[{i}]", f'-> "{s}"') for i, s in enumerate(ENVP)]
    words.append((f"envp[{len(ENVP)}]", "NULL"))
    aux = [(AT_HWCAP, "0x178bfbff (an example)"), (AT_PAGESZ, 4096), (AT_PHDR, "-> program headers"),
           (AT_ENTRY, "-> _start"), (AT_RANDOM, "-> 16 random bytes"), (AT_EXECFN, '-> "./echo_args"'),
           (AT_NULL, 0)]
    for i, (key, value) in enumerate(aux):
        words.append((f"auxv[{i}].a_type", f"{key} ({NAMES[key]})"))
        words.append((f"auxv[{i}].a_val", value))
    return words, string_bytes


def walk(words: list[tuple[str, object]]) -> None:
    """Finds AT_PAGESZ the way the assembly does: skip argc + 1 words, then envp up to its NULL, then pairs."""
    argc = words[0][1]
    i = 1 + argc + 1  # past argc, argc pointers and their NULL
    print(f"walk: argc = {argc}, so envp[0] is at rsp+{8 * i}")
    while words[i][1] != "NULL":
        i += 1
    i += 1
    print(f"walk: envp ends at rsp+{8 * (i - 1)}, so the first auxiliary pair is at rsp+{8 * i}")
    while True:
        key = int(str(words[i][1]).split()[0])
        if key == AT_NULL:
            print("walk: AT_NULL reached, no AT_PAGESZ")
            return
        if key == AT_PAGESZ:
            value = words[i + 1][1]
            print(f"walk: AT_PAGESZ at rsp+{8 * i}, a_val = {value} = 2**{int(value).bit_length() - 1}")
            return
        i += 2


words, string_bytes = build()
for n, (label, value) in enumerate(words):
    print(f"rsp+{8 * n:<4} {label:<16} {value}")
block = 8 * len(words)
padding = (-(block + string_bytes)) % 16
print(f"{len(words)} words = {block} bytes of pointers and pairs, {padding} bytes of padding, then {string_bytes} bytes of strings")
walk(words)

rsp+0    argc             4
rsp+8    argv[0]          -> "./echo_args"
rsp+16   argv[1]          -> "one"
rsp+24   argv[2]          -> "two"
rsp+32   argv[3]          -> "three"
rsp+40   argv[4]          NULL
rsp+48   envp[0]          -> "A=1"
rsp+56   envp[1]          -> "B=2"
rsp+64   envp[2]          NULL
rsp+72   auxv[0].a_type   16 (AT_HWCAP)
rsp+80   auxv[0].a_val    0x178bfbff (an example)
rsp+88   auxv[1].a_type   6 (AT_PAGESZ)
rsp+96   auxv[1].a_val    4096
rsp+104  auxv[2].a_type   3 (AT_PHDR)
rsp+112  auxv[2].a_val    -> program headers
rsp+120  auxv[3].a_type   9 (AT_ENTRY)
rsp+128  auxv[3].a_val    -> _start
rsp+136  auxv[4].a_type   25 (AT_RANDOM)
rsp+144  auxv[4].a_val    -> 16 random bytes
rsp+152  auxv[5].a_type   31 (AT_EXECFN)
rsp+160  auxv[5].a_val    -> "./echo_args"
rsp+168  auxv[6].a_type   0 (AT_NULL)
rsp+176  auxv[6].a_val    0
23 words = 184 bytes of pointers and pairs, 6 bytes of padding, then 50 bytes of strings
walk: argc = 4, so envp[0] is at rsp+48
walk: envp ends at rsp+64, so the first auxiliary pair is at rsp+72
walk: AT_PAGESZ at rsp+88, a_val = 4096 = 2**12

argc is 4 at rsp+0, argv runs from rsp+8 to its NULL at rsp+40, envp from rsp+48 to its NULL at rsp+64, and the pairs start at rsp+72. Step through the same block:

echo_args uses only the first part of it. On each ISA, _start sets a pointer to rsp + 8 (sp + 8), loads argv[i], stops at NULL, counts the string’s bytes up to its NULnull terminator: A single byte of value 0 appended to a string to mark its end. The C library's strlen, puts and printf all walk forward until they meet one; a length-prefixed call like write(2) uses a separate length instead. More in “Endianness, strings and text in memory” and writes it with a newline:

; echo_args-x86-64.asm: print each argv entry on its own line, reading argv from the stack at _start.
; Linux x86-64, NASM syntax, no C library.
        global  _start
        section .rodata
nl:     db      10
        section .text
_start:
        lea     r13, [rsp + 8]      ; [rsp] holds argc, so argv[0] is the next word
.next:  mov     rsi, [r13]          ; argv[i]
        test    rsi, rsi
        jz      .done               ; argv[argc] is NULL
        xor     edx, edx            ; the string's length, counted up to its NUL
.len:   cmp     byte [rsi + rdx], 0
        je      .write
        inc     rdx
        jmp     .len
.write: mov     eax, 1              ; write(1, argv[i], length)
        mov     edi, 1
        syscall
        mov     eax, 1              ; write(1, "\n", 1)
        mov     edi, 1
        lea     rsi, [rel nl]
        mov     edx, 1
        syscall
        add     r13, 8              ; the kernel keeps r13; it changes only rax, rcx and r11
        jmp     .next
.done:  mov     eax, 231            ; exit_group(0)
        xor     edi, edi
        syscall

// echo_args-aarch64.s: print each argv entry on its own line, reading argv from the stack at _start.
// Linux AArch64, GNU as syntax, no C library.
        .section .rodata
nl:     .ascii  "\n"
        .text
        .globl  _start
_start:
        add     x19, sp, #8         // [sp] holds argc, so argv[0] is the next word
next:   ldr     x1, [x19], #8       // argv[i], then step to argv[i + 1]
        cbz     x1, done            // argv[argc] is NULL
        mov     x2, #0              // the string's length, counted up to its NUL
len:    ldrb    w3, [x1, x2]
        cbz     w3, write
        add     x2, x2, #1
        b       len
write:  mov     x0, #1              // write(1, argv[i], length)
        mov     x8, #64
        svc     #0
        mov     x0, #1              // write(1, "\n", 1)
        adr     x1, nl
        mov     x2, #1
        mov     x8, #64
        svc     #0
        b       next
done:   mov     x0, #0              // exit_group(0)
        mov     x8, #94
        svc     #0

# echo_args-riscv64.s: print each argv entry on its own line, reading argv from the stack at _start.
# Linux RISC-V 64, GNU as syntax, no C library.
        .section .rodata
nl:     .ascii  "\n"
        .text
        .globl  _start
_start:
        addi    s1, sp, 8           # 0(sp) holds argc, so argv[0] is the next word
next:   ld      a1, 0(s1)           # argv[i]
        addi    s1, s1, 8
        beqz    a1, done            # argv[argc] is NULL
        li      a2, 0               # the string's length, counted up to its NUL
len:    add     t0, a1, a2
        lbu     t1, 0(t0)
        beqz    t1, write
        addi    a2, a2, 1
        j       len
write:  li      a0, 1               # write(1, argv[i], length)
        li      a7, 64
        ecall
        li      a0, 1               # write(1, "\n", 1)
        la      a1, nl
        li      a2, 1
        li      a7, 64
        ecall
        j       next
done:   li      a0, 0               # exit_group(0)
        li      a7, 94
        ecall

The session assembles and links all six Linux programs of this lesson with LLD and reads one back. The entry pointentry point: The address where the kernel's loader starts a program: the value the executable's header names (e_entry in an ELF header, LC_MAIN in a Mach-O file). The Linux convention is the symbol _start; a C program's main runs later, called by the startup code. More in “First program: return an exit status” in the ELF header is the address of _start, and no program header asks for an interpreter, so nothing at all runs before the program’s own first instruction:

"""Assemble and link this lesson's six Linux programs without a C library, then read one of the ELF files back.

ld.lld makes static executables whose entry point is _start; llvm-readelf and llvm-nm show the entry address, the
program headers and the symbol. Nothing here runs the programs: the lab script does that in a Linux lab.
"""

import pathlib
import sys

sys.path.insert(0, str(pathlib.Path(__file__).parent / "sessions"))
import session

session.run(
    """
nasm -f elf64 -o linux/echo_args-x86-64.o linux/echo_args-x86-64.asm
nasm -f elf64 -o linux/pagesz-x86-64.o linux/pagesz-x86-64.asm
clang --target=aarch64-linux-gnu -c -o linux/echo_args-aarch64.o linux/echo_args-aarch64.s
clang --target=aarch64-linux-gnu -c -o linux/pagesz-aarch64.o linux/pagesz-aarch64.s
llvm-mc --triple=riscv64-linux-gnu -filetype=obj -o linux/echo_args-riscv64.o linux/echo_args-riscv64.s
llvm-mc --triple=riscv64-linux-gnu -filetype=obj -o linux/pagesz-riscv64.o linux/pagesz-riscv64.s
ld.lld -o linux/echo_args-x86-64 linux/echo_args-x86-64.o
ld.lld -o linux/pagesz-x86-64 linux/pagesz-x86-64.o
ld.lld -o linux/echo_args-aarch64 linux/echo_args-aarch64.o
ld.lld -o linux/pagesz-aarch64 linux/pagesz-aarch64.o
ld.lld -o linux/echo_args-riscv64 linux/echo_args-riscv64.o
ld.lld -o linux/pagesz-riscv64 linux/pagesz-riscv64.o
# The entry point is the address of _start, and no program header asks for an interpreter.
llvm-readelf -h linux/echo_args-x86-64 | grep -E 'Type:|Machine:|Entry'
llvm-nm linux/echo_args-x86-64 | grep _start
llvm-readelf -l linux/echo_args-x86-64 | grep -E 'INTERP|LOAD'
# The AArch64 walk: ldp reads one a_type and a_val pair; rbit and clz count the zero bits.
llvm-objdump -d --no-show-raw-insn linux/pagesz-aarch64 | grep -E 'ldp|rbit|clz'
"""
)

$ nasm -f elf64 -o linux/echo_args-x86-64.o linux/echo_args-x86-64.asm
$ nasm -f elf64 -o linux/pagesz-x86-64.o linux/pagesz-x86-64.asm
$ clang --target=aarch64-linux-gnu -c -o linux/echo_args-aarch64.o linux/echo_args-aarch64.s
$ clang --target=aarch64-linux-gnu -c -o linux/pagesz-aarch64.o linux/pagesz-aarch64.s
$ llvm-mc --triple=riscv64-linux-gnu -filetype=obj -o linux/echo_args-riscv64.o linux/echo_args-riscv64.s
$ llvm-mc --triple=riscv64-linux-gnu -filetype=obj -o linux/pagesz-riscv64.o linux/pagesz-riscv64.s
$ ld.lld -o linux/echo_args-x86-64 linux/echo_args-x86-64.o
$ ld.lld -o linux/pagesz-x86-64 linux/pagesz-x86-64.o
$ ld.lld -o linux/echo_args-aarch64 linux/echo_args-aarch64.o
$ ld.lld -o linux/pagesz-aarch64 linux/pagesz-aarch64.o
$ ld.lld -o linux/echo_args-riscv64 linux/echo_args-riscv64.o
$ ld.lld -o linux/pagesz-riscv64 linux/pagesz-riscv64.o
# The entry point is the address of _start, and no program header asks for an interpreter.
$ llvm-readelf -h linux/echo_args-x86-64 | grep -E 'Type:|Machine:|Entry'
  Type:                              EXEC (Executable file)
  Machine:                           Advanced Micro Devices X86-64
  Entry point address:               0x201130
$ llvm-nm linux/echo_args-x86-64 | grep _start
0000000000201130 T _start
0000000000201175 t _start.done
0000000000201140 t _start.len
0000000000201135 t _start.next
000000000020114b t _start.write
$ llvm-readelf -l linux/echo_args-x86-64 | grep -E 'INTERP|LOAD'
  LOAD           0x000000 0x0000000000200000 0x0000000000200000 0x000121 0x000121 R   0x1000
  LOAD           0x000130 0x0000000000201130 0x0000000000201130 0x00004e 0x00004e R E 0x1000
# The AArch64 walk: ldp reads one a_type and a_val pair; rbit and clz count the zero bits.
$ llvm-objdump -d --no-show-raw-insn linux/pagesz-aarch64 | grep -E 'ldp|rbit|clz'
  210134:      	ldp	x11, x12, [x10], #0x10
  210144:      	rbit	x12, x12
  210148:      	clz	x0, x12

The entry is 0x201130 and llvm-nm puts _start at 0x201130. The gABI’s e_entry is where the system first transfers control, and the two LOAD headers, with no INTERP among them, mean the kernel maps the file and jumps. In your Linux lab, run-in-linux-lab.sh builds all six programs with NASM and GNU binutils and runs each one under env -i A=1 B=2 with the arguments one two three, the block of the model above:

#!/bin/sh
# Builds and runs this lesson's six Linux programs in the Linux lab of the toolchain-setup lesson
# (NASM and GNU binutils for x86-64, AArch64 and RISC-V, with qemu-user registered through binfmt).
# env -i gives every program the same two environment strings, so their stacks hold the same block each run.
# Run it from this folder: sh run-in-linux-lab.sh
set -u
nasm -f elf64 -o echo_args-x86-64.o echo_args-x86-64.asm && x86_64-linux-gnu-ld -o echo_args-x86-64 echo_args-x86-64.o
nasm -f elf64 -o pagesz-x86-64.o pagesz-x86-64.asm && x86_64-linux-gnu-ld -o pagesz-x86-64 pagesz-x86-64.o
for name in echo_args pagesz; do
  aarch64-linux-gnu-as -o "$name-aarch64.o" "$name-aarch64.s" && aarch64-linux-gnu-ld -o "$name-aarch64" "$name-aarch64.o"
  riscv64-linux-gnu-as -o "$name-riscv64.o" "$name-riscv64.s" && riscv64-linux-gnu-ld -o "$name-riscv64" "$name-riscv64.o"
done
for isa in x86-64 aarch64 riscv64; do
  env -i A=1 B=2 "./echo_args-$isa" one two three
  echo "echo_args-$isa: exit=$?"
  env -i A=1 B=2 "./pagesz-$isa"
  echo "pagesz-$isa: exit=$? (the page size is 2 to that power)"
done

import pathlib
import sys

sys.path.insert(0, str(pathlib.Path(__file__).parent / "sessions"))
import linux_lab

# The Linux lab is an AArch64 machine: the AArch64 programs run natively, and the kernel hands the x86-64 and RISC-V
# programs to qemu-user through their binfmt entries.
linux_lab.run("""
uname -m
cd linux
sh run-in-linux-lab.sh
""")

$ uname -m
aarch64
$ cd linux
$ sh run-in-linux-lab.sh
./echo_args-x86-64
one
two
three
echo_args-x86-64: exit=0
pagesz-x86-64: exit=12 (the page size is 2 to that power)
./echo_args-aarch64
one
two
three
echo_args-aarch64: exit=0
pagesz-aarch64: exit=12 (the page size is 2 to that power)
./echo_args-riscv64
one
two
three
echo_args-riscv64: exit=0
pagesz-riscv64: exit=12 (the page size is 2 to that power)

Each echo_args printed four lines, its own name first: argv[0] holds the name the program was started with, and argc counts it too. The AArch64 programs read the block the lab’s kernel built; for the x86-64 and RISC-V ones, qemu-user built the same block on the emulated stack. The auxiliary-vector section below reads the pagesz lines.

macOS builds the same block and hands main pointers into it

On macOS the kernel builds the stack too. XNU’s exec_copyout_strings in bsd/kern/kern_exec.c documents it from the stack pointer up: argc, the argument pointers, a zero, the environment pointers, a zero, then a third list of “apple” strings ending in a zero, with the strings above. The difference is who reads it. A macOS program has no _start of its own: dyld runs first, reads LC_MAIN (as the first-program lesson showed) and calls main with argc in x0 and three pointers into that block in x1, x2 and x3. The AArch64 echo_args.s walks argv from x1 exactly as the Linux versions walk it from sp + 8, and block.s reads the word just below argv[0]:

// echo_args.s: print each argv entry on its own line (macOS on Apple silicon, AArch64).
// dyld calls main(argc, argv, envp, apple); argv arrives in x1, a pointer into the block the kernel built.
        .text
        .globl  _main
        .p2align 2
_main:
        stp     x29, x30, [sp, #-32]!
        stp     x19, x20, [sp, #16]
        mov     x29, sp
        mov     x19, x1                 // x19 walks argv
next:
        ldr     x20, [x19], #8          // x20 = argv[i]; step to argv[i+1]
        cbz     x20, done               // argv[argc] is NULL
        mov     x0, x20
        bl      _strlen
        mov     x2, x0                  // length
        mov     x1, x20
        mov     x0, #1
        bl      _write
        mov     x0, #1
        adrp    x1, nl@PAGE
        add     x1, x1, nl@PAGEOFF
        mov     x2, #1
        bl      _write
        b       next
done:
        mov     w0, #0
        ldp     x19, x20, [sp, #16]
        ldp     x29, x30, [sp], #32
        ret
        .section __TEXT,__cstring
nl:     .ascii  "\n"

// block.s: two helpers that look at the block of pointers main receives (macOS on Apple silicon, AArch64).
//   long word_below(char **argv)    the 8-byte word just below argv[0]
//   long count_words(char **list)   how many pointers come before the NULL that ends the list
        .text
        .globl  _word_below, _count_words
        .p2align 2
_word_below:
        ldur    x0, [x0, #-8]           // the word at argv - 8: an unscaled negative offset
        ret
_count_words:
        mov     x9, x0
        mov     x0, #0
1:      ldr     x10, [x9], #8           // the next pointer, then step
        cbz     x10, 2f                 // the NULL that ends the list
        add     x0, x0, #1
        b       1b
2:      ret

// block_main.c: checks where argc, argv, envp and the apple strings sit relative to each other.
// macOS passes main a fourth argument, apple; only the name before each '=' is printed, never a value.
#include <stdio.h>
#include <string.h>

long word_below(char **argv);
long count_words(char **list);

int main(int argc, char **argv, char **envp, char **apple) {
    long nenv = count_words(envp);
    printf("argc = %d; the word just below argv[0] = %ld\n", argc, word_below(argv));
    printf("envp starts %ld words after argv[0] (argc + 1 = %d)\n", (long)(envp - argv), argc + 1);
    printf("%ld environment strings; apple starts %ld words after envp[0]\n", nenv, (long)(apple - envp));
    const char *eq = strchr(apple[0], '=');
    printf("apple[0] begins with \"%.*s=\"\n", eq ? (int)(eq - apple[0]) : 0, apple[0]);
    return 0;
}

"""Run two AArch64 programs on macOS that look at what main receives: echo_args.s prints argv, and block.s with
block_main.c checks where argc, envp and the apple strings sit. env -i fixes the environment at two strings.
"""

import pathlib
import sys

sys.path.insert(0, str(pathlib.Path(__file__).parent / "sessions"))
import session

session.run(
    """
clang -o macos/echo_args macos/echo_args.s
./macos/echo_args one "two words" three; echo "exit=$?"
clang -o macos/block macos/block_main.c macos/block.s
env -i A=1 B=2 ./macos/block one two three
"""
)

$ clang -o macos/echo_args macos/echo_args.s
$ ./macos/echo_args one "two words" three; echo "exit=$?"
./macos/echo_args
one
two words
three
exit=0
$ clang -o macos/block macos/block_main.c macos/block.s
$ env -i A=1 B=2 ./macos/block one two three
argc = 4; the word just below argv[0] = 4
envp starts 5 words after argv[0] (argc + 1 = 5)
2 environment strings; apple starts 3 words after envp[0]
apple[0] begins with "executable_path="

echo_args printed its four arguments, "two words" as one. The second run shows the layout from the inside: the word just below argv[0] is 4, the argc the kernel wrote; envp starts 5 words after argv[0], past the four pointers and their NULL; and with two environment strings, the apple list starts 3 words after envp[0]. Its first entry begins executable_path=, which XNU’s comment calls the exec path. The values are not printed: they hold paths and random numbers that change from run to run.

The auxiliary vector carries the kernel’s facts, ending in a null entry

After the environment’s NULL the kernel leaves pairs of words: a type, a_type, and a value, a_val. getauxval(3) describes the entries a program reads most: AT_PAGESZ, the page size; AT_HWCAP, a bit mask of processor capabilities; AT_RANDOM, the address of 16 random bytes; AT_ENTRY, the program’s entry point; and AT_PHDR, where its program headers are in memory. The numbers are in the kernel’s include/uapi/linux/auxvec.h: AT_NULL 0, AT_PHDR 3, AT_PAGESZ 6, AT_ENTRY 9, AT_HWCAP 16, AT_RANDOM 25. The vector has no length; AT_NULL ends it.

pagesz finds AT_PAGESZ without libc. It skips argc + 1 words, walks envp to its NULL, then reads pairs until the type is 6 or 0. It cannot print a number without writing digits itself, so it exits with the page size’s power of two instead: bsf on x86-64, rbit and clz on AArch64 and a shift loop on base RISC-V each count the zero bits below the one set bit. The model’s walk above reached AT_PAGESZ at rsp+88 and read 4096, two to the 12th, so with the 4,096-byte pages of x86-64 Linux the program is built to exit with 12:

; pagesz-x86-64.asm: walk past argv and envp to the auxiliary vector, find AT_PAGESZ (6) and exit with the page
; size's power of two: 12 for 4096 bytes. Exit status 255 if the vector has no AT_PAGESZ.
; Linux x86-64, NASM syntax, no C library.
        global  _start
        section .text
_start:
        mov     rcx, [rsp]              ; argc
        lea     rsi, [rsp + 8*rcx + 16] ; past argc, argc pointers and their NULL: envp[0]
.env:   cmp     qword [rsi], 0          ; envp ends with a NULL too
        lea     rsi, [rsi + 8]          ; lea changes no flags, so jne still tests the cmp
        jne     .env                    ; rsi now points at the first auxiliary pair
.aux:   mov     rax, [rsi]              ; a_type
        test    rax, rax
        jz      .missing                ; AT_NULL ends the vector
        cmp     rax, 6                  ; AT_PAGESZ
        je      .found
        add     rsi, 16                 ; each pair is two 8-byte words
        jmp     .aux
.found: bsf     rdi, [rsi + 8]          ; a_val is a power of two: the number of its one bit
        mov     eax, 231                ; exit_group(that number)
        syscall
.missing:
        mov     edi, 255
        mov     eax, 231
        syscall

// pagesz-aarch64.s: walk past argv and envp to the auxiliary vector, find AT_PAGESZ (6) and exit with the page
// size's power of two: 12 for 4096 bytes. Exit status 255 if the vector has no AT_PAGESZ.
// Linux AArch64, GNU as syntax, no C library.
        .text
        .globl  _start
_start:
        ldr     x9, [sp]                // argc
        add     x10, sp, #16
        add     x10, x10, x9, lsl #3    // envp[0] = sp + 16 + 8 * argc
env:    ldr     x11, [x10], #8          // envp[i], then step
        cbnz    x11, env                // x10 now points past the NULL: the first auxiliary pair
aux:    ldp     x11, x12, [x10], #16    // a_type, a_val, then step to the next pair
        cbz     x11, missing            // AT_NULL ends the vector
        cmp     x11, #6                 // AT_PAGESZ
        b.ne    aux
        rbit    x12, x12                // trailing zeros are the leading zeros of the reversed value
        clz     x0, x12                 // 12 for 4096, 14 for 16384, 16 for 65536
        mov     x8, #94                 // exit_group(that number)
        svc     #0
missing:
        mov     x0, #255
        mov     x8, #94
        svc     #0

# pagesz-riscv64.s: walk past argv and envp to the auxiliary vector, find AT_PAGESZ (6) and exit with the page
# size's power of two: 12 for 4096 bytes. Exit status 255 if the vector has no AT_PAGESZ.
# Linux RISC-V 64, GNU as syntax, base instructions only, no C library.
        .text
        .globl  _start
_start:
        ld      t0, 0(sp)               # argc
        slli    t0, t0, 3
        add     t1, sp, t0
        addi    t1, t1, 16              # envp[0] = sp + 16 + 8 * argc
env:    ld      t2, 0(t1)
        addi    t1, t1, 8
        bnez    t2, env                 # t1 now points at the first auxiliary pair
aux:    ld      t2, 0(t1)               # a_type
        ld      t3, 8(t1)               # a_val
        addi    t1, t1, 16
        beqz    t2, missing             # AT_NULL ends the vector
        li      t4, 6                   # AT_PAGESZ
        bne     t2, t4, aux
        li      a0, 0                   # count the zero bits below the page size's one bit
bits:   andi    t4, t3, 1
        bnez    t4, out
        srli    t3, t3, 1
        addi    a0, a0, 1
        j       bits
out:    li      a7, 94                  # exit_group(that number)
        ecall
missing:
        li      a0, 255
        li      a7, 94
        ecall

The listing in the build session above shows the heart of the AArch64 walk: ldp x11, x12, [x10], #0x10 loads a type and its value and moves to the next pair in one instruction, and rbit with clz turns 4096 into 12. Linux on arm64 can also be built for 16 KiB or 64 KiB pages, as the address-space lesson says, and then the same program exits with 14 or 16: reading AT_PAGESZ is how a program without libc learns which. In the lab run above all three pagesz programs ended with 12: the AArch64 one read the vector of the lab’s kernel, which uses 4 KiB pages, and qemu-user gave the other two a vector with the same 4096.

Before main, the C runtime’s start-up code does the walking

A C program does not start at main either. When a compiler driver links it, it adds start-up codestart-up code: The code a compiler driver links around a C program, such as crt1.o's _start, which reads the initial stack, prepares the C library, calls main and passes its return value to exit. around your objects, and its _start is the one in the ELF header. clang -### prints the link it would perform without performing it, so the files it adds can be read without a Linux system:

"""Ask clang which start-up files a Linux link of an ordinary C program would add, without linking anything.

clang -### prints the commands it would run; grep keeps the object files whose names start with crt or Scrt.
"""

import pathlib
import sys

sys.path.insert(0, str(pathlib.Path(__file__).parent / "sessions"))
import session

session.run(
    """
clang -### --target=x86_64-linux-gnu linux/main.c -o main 2>&1 | grep -o '[A-Za-z0-9_./-]*crt[A-Za-z0-9_]*[.]o' | sed 's#.*/##' | tr '\\n' ' '; echo
clang -### --target=x86_64-linux-gnu -static linux/main.c -o main 2>&1 | grep -o '[A-Za-z0-9_./-]*crt[A-Za-z0-9_]*[.]o' | sed 's#.*/##' | tr '\\n' ' '; echo
"""
)

$ clang -### --target=x86_64-linux-gnu linux/main.c -o main 2>&1 | grep -o '[A-Za-z0-9_./-]*crt[A-Za-z0-9_]*[.]o' | sed 's#.*/##' | tr '\n' ' '; echo
Scrt1.o crti.o crtbeginS.o crtendS.o crtn.o
$ clang -### --target=x86_64-linux-gnu -static linux/main.c -o main 2>&1 | grep -o '[A-Za-z0-9_./-]*crt[A-Za-z0-9_]*[.]o' | sed 's#.*/##' | tr '\n' ' '; echo
crt1.o crti.o crtbeginT.o crtend.o crtn.o

The usual position-independent link adds Scrt1.o, crti.o, crtbeginS.o, crtendS.o and crtn.o; a static link swaps in crt1.o, crtbeginT.o and crtend.o. The _start in crt1.o, or in Scrt1.o, its position-independent build, reads argc and argv from the stack as echo_args does and calls __libc_start_main, which the Linux Standard Base specifies: it initialises the C library’s execution environment, calls main(argc, argv, envp) and passes the value main returns to exit. exit flushes the library’s buffers and ends the process with exit_group. Step through it:

That is why main may return and _start may not: main has a caller, the start-up code, while _start was jumped to by the kernel with no return address on the stack. A program that defines its own _start takes over all of this work, as the programs of this lesson do.

Practise

Two tasks: one that runs on macOS now, one for your Linux lab.

1. On macOS, write count_env in AArch64 assembly: it takes envp, main’s third argument, and returns the number of environment strings before the NULL. Call it from a C main and print the result. Then write getenv_asm(envp, name), which compares the bytes of name with each string up to its = and returns a pointer to the value. Test both under env -i A=1 AB=2 ABC=3, where every name is a prefix of the next, so getenv_asm(envp, "AB") must return 2 and not 3.
2. In your Linux lab, change pagesz-x86-64.asm to look for AT_ENTRY (9) instead and exit with the low byte of its value. Compare it with the low byte of the entry address llvm-readelf -h prints for the same file.

Interview questions

What is on the stack when a Linux program reaches _start?

The block the kernel built for it, starting at the stack pointer. At rsp (or sp on AArch64 and RISC-V) is argc, the number of arguments. Then come argc pointers to the argument strings, argv, ended by a NULL; then the pointers to the environment strings, envp, ended by another NULL; then the auxiliary vector, pairs of a type and a value such as AT_PAGESZ with the page size or AT_RANDOM with the address of 16 random bytes, ended by AT_NULL. The strings themselves sit above the block. No return address is there, because the kernel jumped to the entry point instead of calling it, so _start must end with an exit system call. A C program’s _start in crt1.o reads this block and calls __libc_start_main, which calls main(argc, argv, envp).

Two problems from the module’s bank go further, one about offsets that shift and one about learning the machine without a C library:

Key takeaways

• At _start on Linux the stack pointer points at argc, followed by argc argument pointers and a NULL, the environment pointers and a NULL, and the auxiliary vector’s type and value pairs, with the strings above.
• A program without libc walks that block itself: argv is rsp + 8, envp starts at rsp + 8 * argc + 16, and the auxiliary vector starts after the environment’s NULL.
• The auxiliary vector carries the kernel’s facts, such as AT_PAGESZ, AT_HWCAP, AT_RANDOM and AT_ENTRY, and ends with AT_NULL; walking it finds the page size without a system call.
• macOS builds the same block, with apple strings in place of the auxiliary vector, but dyld runs first and hands main pointers into it: the word below argv[0] was argc, 4, in the recorded run.
• A linked C program starts in crt1.o’s _start, which calls __libc_start_main; that calls main(argc, argv, envp) and passes its return value to exit.