Debugging crashes: backtraces and std::stacktrace

total_marks reads a mark through a null pointer, and all the shell says is Segmentation fault: 11, exit status 139. A program crashes when the operating system stops it for doing something it may not do, such as reading memory it does not own, which raises the signalsignal: A notice the operating system delivers to a process, such as SIGSEGV for an invalid memory access or SIGABRT from std::abort. Unless the program handles it, the process ends, and a shell reports the exit status as 128 plus the signal's number. SIGSEGV, or when it calls std::abort, which raises SIGABRT. The quickest way to see why is a backtracebacktrace: The list of function calls active at one moment, innermost first: frame 0 is the function that was running, frame 1 its caller, and so on down to main and the system code that started it. With -g, each frame shows its file and line.: the list of function calls that were active at that moment, the innermost first, each with its file and line when the program was built with -g.

There are three ways to get one. A sanitizersanitizer: A compiler option, such as -fsanitize=address or -fsanitize=undefined, that builds extra checks into a program, so that memory errors and undefined behaviour are reported, with their source line, while it runs. More in “Finding memory bugs with AddressSanitizer and UBSan” build, such as AddressSanitizer, prints the stack in its report. A debugger stops at the crash and prints it when you type bt. And the program can print its own stack, which C++23 standardises as std::stacktrace and older code does with backtrace(). Start with the sanitizer: it needs no setup beyond a compiler option.

The shell says only that it crashed; AddressSanitizer says where

total_marks adds up the marks of a roll of students, and one entry of the roll is a null pointer:

#include <print>
#include <string>
#include <vector>

struct Student {
    std::string name;
    int marks;
};

int total_marks(const std::vector<Student*>& roll) {
    int total = 0;
    for (const Student* s : roll) total += s->marks;  // a null pointer in roll crashes here
    return total;
}

int main() {
    Student asha{"Asha", 82};
    Student ravi{"Ravi", 74};
    std::vector<Student*> roll{&asha, nullptr, &ravi};
    std::println("total: {}", total_marks(roll));
}

from session import run

# The same crash three times: built normally with each compiler, then with AddressSanitizer.
run("""
clang++ -std=c++23 -g crash.cpp -o crash && ./crash
g++-16 -std=c++23 -g crash.cpp -o crash-gcc && ./crash-gcc
clang++ -std=c++23 -g -fsanitize=address -fno-omit-frame-pointer crash.cpp -o crash-asan && ./crash-asan
""")

$ clang++ -std=c++23 -g crash.cpp -o crash && ./crash
Segmentation fault: 11
[exit status 139]
$ g++-16 -std=c++23 -g crash.cpp -o crash-gcc && ./crash-gcc
Segmentation fault: 11
[exit status 139]
$ clang++ -std=c++23 -g -fsanitize=address -fno-omit-frame-pointer crash.cpp -o crash-asan && ./crash-asan
AddressSanitizer:DEADLYSIGNAL
==…==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000018 (pc 0x… bp 0x… sp 0x… T0)
==…==The signal is caused by a READ memory access.
==…==Hint: address points to the zero page.
    #0 in total_marks(std::__1::vector<Student*, std::__1::allocator<Student*>> const&) crash.cpp:12
    #1 in main crash.cpp:20
SUMMARY: AddressSanitizer: SEGV
[13 more lines of the report left out]
==…==ABORTING
Abort trap: 6
[exit status 134]

Step through the loop over the roll, then compare what each build says about the crash:

The plain builds say only Segmentation fault: 11 and exit with status 139, which is 128 plus 11, the number of SIGSEGV. Nothing says where. The AddressSanitizer build says it all:

• What: a SEGV caused by a READ, at address 0x000000000018, with the hint that the address is in the zero page. Address 24 is a null pointer plus the offset of marks, which follows a 24-byte std::string in libc++.
• Where: frame #0 is total_marks at crash.cpp:12, the line that read s->marks; frame #1 is main at line 20, the line that called it.

The report is shortened here to its key lines, as in the sanitizers lesson. Because a sanitizer report already contains the stack, a sanitized test build is the first place to reproduce any crash.

Read a backtrace from the top, starting at your own code

Read a backtrace from the top, as the diagram reads the stack of this crash:

Frame #0 is the function that was running, each later frame is the caller of the one before, and the last frames belong to the system code that started main. The first frame in your own code is usually the line to look at, and the frames below it show how the program got there, which is where the bad value, here the null pointer in roll, came from. Three build options make the frames worth reading:

• -g writes the debug informationdebug information: The names, types and line numbers that -g (with MSVC, /Zi) stores with a program, so that a debugger can show the running program in terms of its source. It can be combined with any -O level. More in “Compiler flags every C++ learner should use” that turns addresses into functions, files and lines.
• -fno-omit-frame-pointer keeps a pointer to each frame, so that tools can walk from one frame to the next.
• -O0 or -Og keeps each function as a frame of its own; higher optimisation inlines functions into their callers.

A debugger stops at the crash, and std::stacktrace needs a library that has it

A debugger runs the program and stops it at the crash. With lldb, the debugger of Apple’s tools, lldb ./crash loads the program and run starts it; when it stops, bt prints the backtrace, frame select 1 moves to main’s frame and frame variable shows that frame’s local variables. gdb, the GNU debugger on Linux, offers the same with run, bt, frame 1 and info locals. On macOS, the first time in a login session that a debugger examines one of your processes, the system asks for an administrator’s password; the DevToolsSecurity manual page (man DevToolsSecurity) explains how members of the developer group can skip that prompt.

std::stacktrace, from the C++23 header <stacktrace>, captures the current stack as a value that a program can print or write to a log, for example from an error path. GCC’s libstdc++ implements it and needs the extra library -lstdc++exp when linking, while libc++ has not implemented it. The feature-test macrofeature-test macro: A macro such as __cpp_modules or __cpp_lib_modules whose value, a year and month such as 202207, says that a compiler or its library claims a feature. A toolchain may build a feature without defining its macro, or define the macro without the feature. More in “C++20 modules: export, import and partitions” __cpp_lib_stacktrace tells code which case it is in:

#include <print>
#include <version>  // defines the feature-test macros of the standard library

int main() {
#if defined(_LIBCPP_VERSION)
    std::println("standard library: libc++");
#elif defined(__GLIBCXX__)
    std::println("standard library: libstdc++");
#endif
#if defined(__cpp_lib_stacktrace)
    std::println("std::stacktrace: available (__cpp_lib_stacktrace = {})", __cpp_lib_stacktrace);
#else
    std::println("std::stacktrace: not available (__cpp_lib_stacktrace is not defined)");
#endif
}

from session import run

# Asks each standard library on this Mac whether it provides std::stacktrace.
run("""
clang++ -std=c++23 stacktrace_check.cpp -o stacktrace_check && ./stacktrace_check
g++-16 -std=c++23 stacktrace_check.cpp -o stacktrace_check-gcc && ./stacktrace_check-gcc
""")

$ clang++ -std=c++23 stacktrace_check.cpp -o stacktrace_check && ./stacktrace_check
standard library: libc++
std::stacktrace: not available (__cpp_lib_stacktrace is not defined)
$ g++-16 -std=c++23 stacktrace_check.cpp -o stacktrace_check-gcc && ./stacktrace_check-gcc
standard library: libstdc++
std::stacktrace: not available (__cpp_lib_stacktrace is not defined)

On this Mac the answer is no for both. libc++ has no std::stacktrace at all, and Homebrew’s GCC 16 for macOS was configured without it, so its <stacktrace> header declares nothing. On Linux the same GCC provides it, so portable code tests the macro and falls back to another way of printing the stack. Either way, a program most often prints its own stack from a terminate handler, installed with std::set_terminate, or from a handler for a signal such as SIGSEGV. Side by side:

Practise: read a crash and its stack

Eight questions on the three ideas above, starting with the one you have answered least well.

Printing a stack trace from code, and turning it into files and lines Pro: opens with a Pro, Premium or Ultimate pass

Logging before the program dies: signal handlers and a terminate handler of your own Pro: opens with a Pro, Premium or Ultimate pass

Interview questions

Key takeaways

• A crash is a signal: exit status 139 is SIGSEGV, 134 is SIGABRT, and the shell alone never says where.
• A backtrace lists the active calls, innermost first; start at the first frame in your own code.
• Build with -g -fno-omit-frame-pointer and low optimisation for readable stacks, and reproduce crashes in a sanitizer build first, because its report contains the stack.
• std::stacktrace is C++23, but neither standard library on this Mac provides it: test __cpp_lib_stacktrace before using it.