The stack: push, pop and red zones

save_two needs two registers that its caller is still using, so it copies their values to memory first and puts them back before it returns. On Apple silicon the stack pointer sits 16 bytes lower while it works and is back where it started once it returns, and a version that saves one register in an 8-byte step, the way x86-64’s push does, ends with a bus error at its very next load.

That memory is the stack on every ISA, but each ISA moves its stack pointer its own way: x86-64 with push and pop, AArch64 with pair stores that update sp, RISC-V with an addi and plain stores. You will trace the stack pointer through all three, put saved registers back in the right order, and see which ABIs let a function keep data below the stack pointer without moving it at all, and why kernel code is built never to do that.

The stack grows down, and push lowers rsp before it stores

A stackstack: Memory a program uses last in, first out: the value stored most recently is the first taken back. The stack pointer holds the address of that value, and on x86-64, AArch64 and RISC-V the stack grows towards lower addresses. is memory used last in, first out: the value stored most recently is the first one taken back. The stack pointerstack pointer: The register that addresses the top of the stack: rsp on x86-64, sp on AArch64 and sp (x2) on RISC-V. Push, pop and function prologues move it; losing it loses the stack. More in “Registers on x86-64, AArch64 and RISC-V” holds the address of that most recent value, the top of the stack, and on x86-64, AArch64 and RISC-V the stack grows towards lower addresses. To pushpush: Putting a value on the stack: lower the stack pointer, then store the value at the new address. x86-64's push instruction does both and moves rsp by 8 for a 64-bit register; AArch64 and RISC-V use a store and a stack-pointer update. a value therefore means to lower the stack pointer and write at the new address, and to poppop: Taking the most recent value off the stack: load it from the address in the stack pointer, then raise the stack pointer past it. The bytes stay in memory, below the stack pointer, until something overwrites them. one means to read it and raise the stack pointer again. Intel’s manual defines the x86-64 instructions in exactly those steps, the RISC-V psABI says its stack grows downwards, and AAPCS64 calls Arm’s stack full-descending: the part in use runs from sp up to the stack’s base, and everything below sp is inactive.

save_two(long *seen) borrows two registers whose values its caller still needs, writes the stack pointer it has after saving them through seen, returns 40 + 2 and hands both registers back. Which registers a function must hand back is a rule of its own, covered in callee-saved registers and stack frames; here the three versions borrow rbx and r12, x19 and x20, and s1 and s2:

// save_two(long *seen) for AArch64 on Apple platforms: borrows x19 and x20, then hands them back.
// *seen receives sp as it is after the save, so the C driver can print how far sp moved.
    .text
    .globl _save_two
    .p2align 2
_save_two:
    stp x19, x20, [sp, #-16]!     // sp = sp - 16, then x19 to [sp] and x20 to [sp, #8]
    mov x1, sp
    str x1, [x0]                  // *seen = sp after the save
    mov x19, #40                  // the old values are saved: reuse both
    mov x20, #2
    add x0, x19, x20              // return value: 42
    ldp x19, x20, [sp], #16       // x19 from [sp], x20 from [sp, #8], then sp = sp + 16
    ret

// get_sp(): returns sp as its caller has it; this function moves sp nowhere
    .globl _get_sp
    .p2align 2
_get_sp:
    mov x0, sp
    ret

# save_two(long *seen) for x86-64 Linux (System V): borrows rbx and r12, then hands them back.
.intel_syntax noprefix
.text
.globl save_two
save_two:
    push rbx                 # rsp = rsp - 8, then rbx to [rsp]
    push r12                 # rsp = rsp - 8, then r12 to [rsp]
    mov [rdi], rsp           # *seen = rsp after the saves
    mov rbx, 40              # the old values are saved: reuse both
    mov r12, 2
    lea rax, [rbx + r12]     # return value: 42
    pop r12                  # r12 from [rsp], then rsp = rsp + 8: the last one saved comes back first
    pop rbx
    ret

# save_two(long *seen) for RISC-V Linux (LP64D): borrows s1 and s2, then hands them back.
.text
.globl save_two
save_two:
    addi sp, sp, -16         # make room: two 8-byte slots
    sd s1, 0(sp)             # s1 to the slot at sp
    sd s2, 8(sp)             # s2 to the slot above it
    sd sp, 0(a0)             # *seen = sp after the saves
    li s1, 40                # the old values are saved: reuse both
    li s2, 2
    add a0, s1, s2           # return value: 42
    ld s1, 0(sp)             # each register comes back from its own slot
    ld s2, 8(sp)
    addi sp, sp, 16          # sp is back where it started
    ret

The C driver reads the stack pointer just before the call and just after it with get_sp, a two-instruction helper in the AArch64 file, and prints differences only, because the addresses themselves change from one run to the next:

/* Calls save_two once and prints how far sp moved: inside save_two after its save, and after it returned.
   Only differences are printed, because the addresses themselves change from one run to the next. */
#include <stdio.h>

long save_two(long *seen);
long get_sp(void);

int main(void) {
    long seen = 0;
    long before = get_sp();
    long result = save_two(&seen);
    long after = get_sp();
    printf("save_two returned %ld\n", result);
    printf("sp inside, after the save: %+ld bytes\n", seen - before);
    printf("sp after the return:       %+ld bytes\n", after - before);
    return 0;
}

The session builds and runs the AArch64 version on Apple silicon, then assembles the x86-64 and RISC-V versions into Linux object files and lists them with the bytes of each instruction:

"""Builds save_two for AArch64 and runs it, then assembles the x86-64 and RISC-V versions and lists their bytes."""

import pathlib
import sys

sys.path.insert(0, str(pathlib.Path(__file__).parent / "sessions"))
import session  # noqa: E402

session.run("""
# AArch64: compile the C driver, assemble save_two_aarch64.s, link them into one program and run it
clang -O1 -o save_two save_two_main.c save_two_aarch64.s
./save_two
# x86-64: assemble the System V version into a Linux object file, then list it with the bytes of each instruction
clang --target=x86_64-linux-gnu -c -o save_two_x86_64.o save_two_x86_64.s
llvm-objdump -d -M intel save_two_x86_64.o
# RISC-V: assemble the LP64D version into a Linux object file and list it the same way
llvm-mc --triple=riscv64-linux-gnu -filetype=obj -o save_two_riscv64.o save_two_riscv64.s
llvm-objdump -d save_two_riscv64.o
""")

# AArch64: compile the C driver, assemble save_two_aarch64.s, link them into one program and run it
$ clang -O1 -o save_two save_two_main.c save_two_aarch64.s
$ ./save_two
save_two returned 42
sp inside, after the save: -16 bytes
sp after the return:       +0 bytes
# x86-64: assemble the System V version into a Linux object file, then list it with the bytes of each instruction
$ clang --target=x86_64-linux-gnu -c -o save_two_x86_64.o save_two_x86_64.s
$ llvm-objdump -d -M intel save_two_x86_64.o

save_two_x86_64.o:	file format elf64-x86-64

Disassembly of section .text:

0000000000000000 <save_two>:
       0: 53                           	push	rbx
       1: 41 54                        	push	r12
       3: 48 89 27                     	mov	qword ptr [rdi], rsp
       6: 48 c7 c3 28 00 00 00         	mov	rbx, 0x28
       d: 49 c7 c4 02 00 00 00         	mov	r12, 0x2
      14: 4a 8d 04 23                  	lea	rax, [rbx + r12]
      18: 41 5c                        	pop	r12
      1a: 5b                           	pop	rbx
      1b: c3                           	ret
# RISC-V: assemble the LP64D version into a Linux object file and list it the same way
$ llvm-mc --triple=riscv64-linux-gnu -filetype=obj -o save_two_riscv64.o save_two_riscv64.s
$ llvm-objdump -d save_two_riscv64.o

save_two_riscv64.o:	file format elf64-littleriscv

Disassembly of section .text:

0000000000000000 <save_two>:
       0: ff010113     	addi	sp, sp, -0x10
       4: 00913023     	sd	s1, 0x0(sp)
       8: 01213423     	sd	s2, 0x8(sp)
       c: 00253023     	sd	sp, 0x0(a0)
      10: 02800493     	li	s1, 0x28
      14: 00200913     	li	s2, 0x2
      18: 01248533     	add	a0, s1, s2
      1c: 00013483     	ld	s1, 0x0(sp)
      20: 00813903     	ld	s2, 0x8(sp)
      24: 01010113     	addi	sp, sp, 0x10
      28: 00008067     	ret

x86-64 has one instruction for each half of the job. Intel’s manual defines push as lowering the stack pointer and then storing the operand on the top of the stack, and pop as loading the top of the stack into the operand and then raising the stack pointer; for a 64-bit register both move rsp by 8. They are short, too. In the listing push rbx is the single byte 53 and push r12 the two bytes 41 54, where 41 is the prefix that selects one of r8 to r15, and the pops at the end are 41 5c and 5b. save_two pushes rbx first, so r12 ends up lowest, at the top of the stack, and the first pop has to take r12 back. Step through it with rsp written relative to S, the address where the call left its return addressreturn address: The address a call records so that the callee can come back: the address of the instruction right after the call. x86-64's call pushes it onto the stack; AArch64's bl and blr and RISC-V's jal and call write it to x30 or ra. More in “Calls, returns and the link register” (calls and returns follows that address from the call to the ret):

A pop leaves the old value where it was. After pop rbx both saved values are still in memory, but below rsp, in the unused part of the stack, where the next push or the next call may overwrite them at any moment. Popping gives a slot up; it does not clear it.

x86-64 is the only one of the three ISAs with dedicated push and pop instructions. AArch64 builds the same two steps from a store that updates sp, RISC-V from two separate instructions, and both move the stack pointer in bigger steps.

AArch64 and RISC-V have no push, and move sp in 16-byte steps

AArch64 has no push or pop instruction, and neither has RV64GC, the RISC-V instruction set this lesson’s code is built for. Saving a register there is an ordinary store through sp, plus arithmetic on sp itself, and the two ISAs package that differently.

AArch64 folds both jobs into one instruction. stp x19, x20, [sp, #-16]! is a store pair in its pre-index form: it works out sp - 16, stores x19 there and x20 8 bytes higher, and writes the new address back into sp. That update of the base register is called writebackwriteback: The update of the base register by a pre-indexed or post-indexed load or store. AArch64's stp x19, x20, [sp, #-16]! writes sp - 16 back into sp, and ldp x19, x20, [sp], #16 adds 16 to sp after the load., and the ! asks for it. ldp x19, x20, [sp], #16 is the post-index load pair: it loads both registers from sp and then adds 16 to it. The forms come from addressing modes and the pair instructions from loads and stores. The run above shows their effect: inside save_two, after its stp, sp was 16 bytes lower than before the call, and after the return it was back, at +0 bytes.

RISC-V has neither pairs nor writeback. Its save_two moves the stack pointer with addi sp, sp, -16 and stores each register in its own stack slotstack slot: One place on the stack that holds one value while a function runs, found at a fixed offset from the stack pointer, such as [rsp + 16] or 8(sp)., sd s1, 0(sp) and sd s2, 8(sp); the restore loads both from the same offsets and ends with addi sp, sp, 16. The three ways side by side:

On x86-64 the order of the restores is fixed, because each pop takes whatever is on top. On AArch64 and RISC-V each register comes back from the offset it was stored at, so the two loads could swap places; what has to come last is the instruction that raises sp, since anything may overwrite what lies below sp once it has run. Every version ends with the stack pointer at its entry value, which the +0 bytes line confirms for AArch64. Saves like these at the start of a function and restores at its end belong to its prologueprologue: The instructions at the start of a function that set up its stack frame before the body runs: they save the callee-saved registers the body writes, save the link register when the function makes calls (on x86-64, call has already put the return address on the stack), and reserve its stack space. More in “Callee-saved registers and stack frames” and epilogueepilogue: The instructions at the end of a function that undo its prologue: they reload the saved registers (pops come in the reverse order of the pushes), give the stack space back so the stack pointer is where it was on entry, and return. More in “Callee-saved registers and stack frames”, which the lesson on callee-saved registers takes apart.

Why 16 and not 8, as a push would use? AAPCS64 states that whenever memory is accessed through sp, the hardware requires sp to be a multiple of 16, and Arm’s pseudocode for stp and ldp checks the alignment of sp as soon as it reads sp as the base register, before any offset is added. one_push breaks the rule on purpose. It saves one register with str x19, [sp, #-8]!, an 8-byte step: the store itself passes, since sp was aligned when it started, but it leaves sp 8 bytes off a multiple of 16, and the ldr that follows uses that sp as its base:

// one_push() for AArch64 on Apple platforms: saves ONE register the way x86-64's push would, in an 8-byte step.
// After the store, sp is a multiple of 8 but not of 16, and the load that takes x19 back uses sp as its base.
    .text
    .globl _one_push
    .p2align 2
_one_push:
    str x19, [sp, #-8]!     // sp = sp - 8, then x19 to [sp]: sp is now 8 bytes off a multiple of 16
    mov x0, #1
    ldr x19, [sp], #8       // a load whose base is that sp
    ret

/* Calls one_push once. The line below is printed only if one_push returns. */
#include <stdio.h>

long one_push(void);

int main(void) {
    printf("one_push returned %ld\n", one_push());
    return 0;
}

"""Builds one_push for AArch64 and runs it. The program is stopped by a signal, so the shell's own message about it
is dropped and the exit status is printed instead; an exit status above 128 is 128 plus the signal's number."""

import pathlib
import sys

sys.path.insert(0, str(pathlib.Path(__file__).parent / "sessions"))
import session  # noqa: E402

session.run("""
# AArch64: one register saved in an 8-byte step, as a push would do it
clang -O1 -o one_push one_push_main.c one_push_aarch64.s
{ ./one_push; } 2>/dev/null; echo "exit=$?"
# the name of signal 138 - 128 = 10
kill -l 10
""")

# AArch64: one register saved in an 8-byte step, as a push would do it
$ clang -O1 -o one_push one_push_main.c one_push_aarch64.s
$ { ./one_push; } 2>/dev/null; echo "exit=$?"
exit=138
# the name of signal 138 - 128 = 10
$ kill -l 10
BUS

The driver’s line never appeared. The program ended with exit status 138, which is how the shell reports a process stopped by a signalsignal: A notification the operating system delivers to a process, such as SIGFPE after an integer divide fault; unhandled, it ends the process, and the shell reports status 128 plus the signal number. More in “Division: traps, defined results, remainders”: 138 - 128 = 10, and kill -l 10 names signal 10 BUS, the bus error. On AArch64 a function with one register to save still moves sp by 16 and leaves 8 bytes of the space unused. To RISC-V hardware sp is the ordinary register x2, but its psABI requires sp to be 16-byte aligned at every procedure entry and to stay aligned while the procedure runs, so RISC-V code moves sp in 16-byte steps too. x86-64 hardware has no such rule for a push; the alignment rules of its ABIs concern the stack at and around calls, and they have details of their own.

A red zone lets a leaf keep data below the stack pointer, on two of these ABIs

So far every save lowered the stack pointer before it stored, because anything that writes to the stack may overwrite the memory below it. Two ABIs carve out an exception. The System V psABI sets aside the 128 bytes just below rsp, and signal and interrupt handlers must leave them as they are. Any function may park short-lived data there, data it will not need once it calls something, and a leaf functionleaf function: A function that makes no calls of its own. Nothing overwrites its link register while it runs, so it needs no save of x30 or ra, and at -O2 compilers often give it no stack space at all. More in “Calls, returns and the link register” can fit its entire frame in it and skip moving rsp altogether. That area is the red zonered zone: The 128 bytes just below the stack pointer that the System V x86-64 ABI, and Apple on arm64, keep safe from signal and interrupt handlers, so a function may keep temporary data there without moving the stack pointer.. Apple documents the same 128 bytes below sp for arm64: its platforms leave them unchanged during exceptions, and user-mode programs may use them for local variables.

leaf_sum makes the difference visible. Its four temporaries are volatile, so the compiler must store each one to memory and load it back, and it calls nothing:

/* A leaf function whose four locals must live in memory: volatile makes the compiler store and load each one.
   long long is 64 bits on every target of the session, so each one keeps the same 32 bytes of data. */
long long leaf_sum(long long a, long long b) {
    volatile long long tmp[4];
    tmp[0] = a;
    tmp[1] = b;
    tmp[2] = a + b;
    tmp[3] = a - b;
    return tmp[0] + tmp[1] + tmp[2] + tmp[3];
}

The session prints what clang -O2 makes of it for five ABIs, and for x86-64 Linux a second time with the red zone switched off:

"""Prints what clang -O2 makes of the same leaf function for five ABIs, and for x86-64 Linux once more with the red
zone switched off. The grep after each command keeps labels and instructions and drops directives and comments."""

import pathlib
import sys

sys.path.insert(0, str(pathlib.Path(__file__).parent / "sessions"))
import session  # noqa: E402

session.run("""
# x86-64 Linux, System V ABI
clang --target=x86_64-linux-gnu -O2 -S -masm=intel -fno-asynchronous-unwind-tables -o - leaf.c | grep -E '^[_[:alpha:]]|^[[:space:]]+[[:lower:]]'
# the same target with the red zone switched off
clang --target=x86_64-linux-gnu -O2 -S -masm=intel -fno-asynchronous-unwind-tables -mno-red-zone -o - leaf.c | grep -E '^[_[:alpha:]]|^[[:space:]]+[[:lower:]]'
# x86-64 Windows, Microsoft x64 ABI
clang --target=x86_64-pc-windows-msvc -O2 -S -masm=intel -fno-asynchronous-unwind-tables -o - leaf.c | grep -E '^[_[:alpha:]]|^[[:space:]]+[[:lower:]]'
# arm64 on Apple platforms
clang --target=arm64-apple-macos -O2 -S -fno-asynchronous-unwind-tables -o - leaf.c | grep -E '^[_[:alpha:]]|^[[:space:]]+[[:lower:]]'
# AArch64 Linux, AAPCS64
clang --target=aarch64-linux-gnu -O2 -S -fno-asynchronous-unwind-tables -o - leaf.c | grep -E '^[_[:alpha:]]|^[[:space:]]+[[:lower:]]'
# RISC-V Linux, LP64D
clang-23 --target=riscv64-linux-gnu -march=rv64gc -mabi=lp64d -O2 -S -fno-asynchronous-unwind-tables -o - leaf.c | grep -E '^[_[:alpha:]]|^[[:space:]]+[[:lower:]]'
""")

# x86-64 Linux, System V ABI
$ clang --target=x86_64-linux-gnu -O2 -S -masm=intel -fno-asynchronous-unwind-tables -o - leaf.c | grep -E '^[_[:alpha:]]|^[[:space:]]+[[:lower:]]'
leaf_sum:                               # @leaf_sum
	mov	qword ptr [rsp - 24], rdi
	mov	qword ptr [rsp - 8], rsi
	lea	rax, [rsi + rdi]
	mov	qword ptr [rsp - 40], rax
	sub	rdi, rsi
	mov	qword ptr [rsp - 16], rdi
	mov	rax, qword ptr [rsp - 24]
	add	rax, qword ptr [rsp - 8]
	add	rax, qword ptr [rsp - 40]
	add	rax, qword ptr [rsp - 16]
	ret
# the same target with the red zone switched off
$ clang --target=x86_64-linux-gnu -O2 -S -masm=intel -fno-asynchronous-unwind-tables -mno-red-zone -o - leaf.c | grep -E '^[_[:alpha:]]|^[[:space:]]+[[:lower:]]'
leaf_sum:                               # @leaf_sum
	sub	rsp, 40
	mov	qword ptr [rsp + 16], rdi
	mov	qword ptr [rsp + 32], rsi
	lea	rax, [rsi + rdi]
	mov	qword ptr [rsp], rax
	sub	rdi, rsi
	mov	qword ptr [rsp + 24], rdi
	mov	rax, qword ptr [rsp + 16]
	add	rax, qword ptr [rsp + 32]
	add	rax, qword ptr [rsp]
	add	rax, qword ptr [rsp + 24]
	add	rsp, 40
	ret
# x86-64 Windows, Microsoft x64 ABI
$ clang --target=x86_64-pc-windows-msvc -O2 -S -masm=intel -fno-asynchronous-unwind-tables -o - leaf.c | grep -E '^[_[:alpha:]]|^[[:space:]]+[[:lower:]]'
leaf_sum:                               # @leaf_sum
	sub	rsp, 40
	mov	qword ptr [rsp + 16], rcx
	mov	qword ptr [rsp + 32], rdx
	lea	rax, [rdx + rcx]
	mov	qword ptr [rsp], rax
	sub	rcx, rdx
	mov	qword ptr [rsp + 24], rcx
	mov	rax, qword ptr [rsp + 16]
	mov	rcx, qword ptr [rsp + 32]
	add	rcx, rax
	mov	rdx, qword ptr [rsp]
	mov	rax, qword ptr [rsp + 24]
	add	rax, rdx
	add	rax, rcx
	add	rsp, 40
	ret
# arm64 on Apple platforms
$ clang --target=arm64-apple-macos -O2 -S -fno-asynchronous-unwind-tables -o - leaf.c | grep -E '^[_[:alpha:]]|^[[:space:]]+[[:lower:]]'
_leaf_sum:                              ; @leaf_sum
	sub	sp, sp, #32
	str	x0, [sp, #24]
	str	x1, [sp, #16]
	add	x8, x1, x0
	str	x8, [sp, #8]
	sub	x8, x0, x1
	str	x8, [sp]
	ldr	x8, [sp, #24]
	ldr	x9, [sp, #16]
	add	x8, x9, x8
	ldr	x9, [sp, #8]
	ldr	x10, [sp], #32
	add	x9, x9, x10
	add	x0, x8, x9
	ret
# AArch64 Linux, AAPCS64
$ clang --target=aarch64-linux-gnu -O2 -S -fno-asynchronous-unwind-tables -o - leaf.c | grep -E '^[_[:alpha:]]|^[[:space:]]+[[:lower:]]'
leaf_sum:                               // @leaf_sum
	sub	sp, sp, #32
	add	x8, x1, x0
	sub	x9, x0, x1
	str	x0, [sp, #24]
	str	x1, [sp, #16]
	str	x8, [sp, #8]
	str	x9, [sp]
	ldr	x8, [sp, #24]
	ldr	x9, [sp, #16]
	ldr	x10, [sp, #8]
	ldr	x11, [sp], #32
	add	x8, x9, x8
	add	x9, x10, x11
	add	x0, x8, x9
	ret
# RISC-V Linux, LP64D
$ clang-23 --target=riscv64-linux-gnu -march=rv64gc -mabi=lp64d -O2 -S -fno-asynchronous-unwind-tables -o - leaf.c | grep -E '^[_[:alpha:]]|^[[:space:]]+[[:lower:]]'
leaf_sum:                               # @leaf_sum
	addi	sp, sp, -32
	sd	a0, 24(sp)
	sd	a1, 16(sp)
	add	a2, a1, a0
	sub	a0, a0, a1
	sd	a2, 8(sp)
	sd	a0, 0(sp)
	ld	a0, 24(sp)
	ld	a1, 16(sp)
	ld	a2, 8(sp)
	ld	a3, 0(sp)
	add	a0, a0, a1
	add	a2, a2, a3
	add	a0, a0, a2
	addi	sp, sp, 32
	ret

The x86-64 Linux build never touches rsp. It stores its four values in four 8-byte slots at rsp - 24, rsp - 8, rsp - 40 and rsp - 16, all inside the red zone, adds them up and returns. With -mno-red-zone the same function begins with sub rsp, 40 and ends with add rsp, 40 before ret, and its slots move to rsp + 16, rsp + 32, rsp and rsp + 24: the very same addresses as before, now above the stack pointer instead of below it. The Windows build does the same, because Microsoft x64 has no red zone. Microsoft counts all memory below rsp as volatile, open to being overwritten by the operating system, a debugger or an interrupt handler, and requires rsp to be set before a function reads or writes its stack frame. All six builds:

Two rows need a word. AAPCS64 itself has no red zone: no thread may read or write the inactive part of its stack, which is everything below sp. Apple’s arm64 grants the 128 bytes on top of that, yet clang moved sp for this function all the same; a red zone lets code keep data below the stack pointer, and nothing obliges a compiler to. RISC-V’s psABI tells procedures not to rely on data below sp persisting, and its build moved sp by 32.

The zone only holds data that is not needed after a call, because a call writes below the stack pointer too: on x86-64 it pushes its return address at rsp - 8, the top of the zone, and the callee may use the same 128 bytes as its own red zone. That is why the psABI offers the space for temporary data and names leaf functions as the ones that can live in it entirely.

Kernel code turns the red zone off, because interrupts push onto its stack

Linux builds its x86-64 kernel with -mno-red-zone: the flag is set in the kernel’s arch/x86/Makefile, beside its other code-generation flags, and the kernel’s Rust code gets the matching -Cno-redzone=y. GCC’s manual describes the flag as turning off the red zone that the x86-64 ABI mandates, the 128 bytes that signal and interrupt handlers do not modify. Why would a kernel give that up? Because of where the processor saves its state when an interruptA request from a device or a timer that makes the processor set aside the code it is running, run a handler and then resume that code where it stopped. arrives, as Intel’s manual describes it.

When an interrupt arrives while user code runs, its handler runs at a more privileged level, and the processor first switches to a stack that the kernel set up for that level; only then does it save the interrupted state. The user’s stack, red zone and all, is never written. Kernel code already runs at the handler’s level, so for an interrupt that arrives while kernel code runs there is no switch: the processor saves the state on the current stack, right where rsp points. In 64-bit mode it first aligns rsp down to a multiple of 16 and then pushes five 8-byte values, the old SS, RSP, RFLAGS, CS and RIP: 40 bytes just below rsp, or 48 when an exception adds an error code. Only interrupts whose descriptor selects a separate stack from the interrupt stack table avoid this. That is delivery through the interrupt descriptor table (IDT), the scheme x86-64 processors have long used. FRED, a newer scheme that Intel’s manual also describes, first lowers rsp at the same privilege level by an amount the kernel configures, a multiple of 64 bytes, and the kernel’s build sets -mno-red-zone all the same. Step through one interrupt, delivered through the IDT, that lands on a kernel leaf’s red-zone data:

The failure leaves no trace. Nothing faults, no signal is sent, and the value is wrong only when an interrupt lands in a window a few instructions long, between a store below rsp and the load that reads it back, so it is rare in a test and more frequent under heavy interrupt load. With -mno-red-zone the compiler does what the second leaf_sum build showed: it lowers rsp first, so the function’s data sits at and above rsp and an interrupt frame lands below it. User code keeps its red zone, because an interrupt taken in user code writes its frame on another stack, and the psABI binds signal handlers to leave the 128 bytes alone.

Practise

Four tasks on copies of this lesson’s files. Build the AArch64 programs and assemble or compile the others with the commands of the sessions above, and for every task write your prediction down before you run anything.

1. Copy save_two_aarch64.s and let save_two borrow x21 and x22 as well: a second stp straight after the first, the matching ldp just before the first ldp, and values for x21 and x22 in the body, with the result still 42. Before you link it with save_two_main.c, write down both sp lines the driver will print.
2. Trace the stack pointer through the eleven instructions of save_two_riscv64.s and the nine of save_two_x86_64.s, writing it after each one as an offset from S. Check both traces against the offsets that the stores and loads use in the listings.
3. In a copy of save_two_x86_64.s, swap the two pops, then assemble and list it. Say which value rbx and r12 each hold when ret runs, and what the caller would see afterwards.
4. Copy leaf.c, give tmp 20 elements and add all 20 into the result, then compile it for x86_64-linux-gnu with and without -mno-red-zone. The values take 160 bytes and the red zone 128: decide first how far each build must move rsp, then read what clang did.

Interview questions

What does x86-64’s push do to rsp and to memory, and how do AArch64 and RISC-V save a register without one?

push lowers rsp by 8 and then stores its operand at the new address, so rsp always points at the most recent value, and pop loads the value at rsp into its operand and then raises rsp by 8. Registers therefore come back in the reverse of the order they were pushed. AArch64 and RV64GC RISC-V have no push or pop. AArch64 stores a pair with writeback: stp x19, x20, [sp, #-16]! lowers sp by 16 and stores both registers in one instruction, and ldp x19, x20, [sp], #16 undoes it. RISC-V lowers sp with addi sp, sp, -16, stores each register at its own offset with sd, and reverses that with ld and addi sp, sp, 16. Both keep sp a multiple of 16, and on AArch64 an access through a misaligned sp faults.

The two questions below put the same rules under pressure: a kernel helper whose temporaries change only under load, and a save sequence that has to move from x86-64 to the other two ISAs.

Key takeaways

• All three stacks grow towards lower addresses and the stack pointer addresses the most recent value: x86-64’s push lowers rsp by 8 and then stores, and pop loads and then raises rsp by 8, so registers come back in the reverse of the order they were pushed.
• AArch64 saves a pair and lowers sp in one stp with [sp, #-16]! and undoes it with ldp and [sp], #16, while RISC-V moves sp with addi and stores each register in its own slot with sd; every version ends with the stack pointer back at its entry value.
• AArch64 and RISC-V move sp in 16-byte steps, and on Apple silicon one_push, which saved one register in an 8-byte step, died with a bus error at its next load through sp.
• System V on x86-64 and Apple on arm64 promise a 128-byte red zone below the stack pointer; clang’s x86-64 Linux build of leaf_sum stored there without moving rsp, while every other build, Apple’s included, lowered the stack pointer first.
• In kernel code an interrupt delivered through the IDT pushes its frame on the current stack, just below rsp, so Linux compiles its x86-64 kernel with -mno-red-zone.