C++ Module 15 – Build systems, testing and tooling

Profiling C++: finding where the time goes

Profile C++ before optimising it: build with -O2 -g and frame pointers, read a perf profile and a flame graph, and pick a profiler for your system.

Advanced
15 minutes
Examples run with Clang 21.0.0, GCC 16.2.0 and Python 3.14.8
By MySmartCoPilot

Free: Measure the build you ship, with symbols and frame pointers · Self time and total time point at the linear search · A flame graph draws the samples as nested bars · 2 more sections. With a pass: 2 in-depth parts and 3 interview answers, about 9 minutes.

What you will learn

Build a program for profiling with -O2 -g -fno-omit-frame-pointer, and say what each flag keeps
Read the self and total time of a sampling profile, and the limit a function's share puts on a speed-up
Read a flame graph by the width and height of its bars, ignoring their order
Choose a profiler for Linux, macOS or Windows, and exact counters when shares are not enough
Fix a hot spot version by version, measuring every version and stopping when the profile is flat

Before you start

What you’ll learn · Before you start

What you will learn

Build a program for profiling with -O2 -g -fno-omit-frame-pointer, and say what each flag keeps
Read the self and total time of a sampling profile, and the limit a function's share puts on a speed-up
Read a flame graph by the width and height of its bars, ignoring their order
Choose a profiler for Linux, macOS or Windows, and exact counters when shares are not enough
Fix a hot spot version by version, measuring every version and stopping when the profile is flat

Before you start

On this page

The word count below spends about 2.7 seconds of processor time on two million words, and nothing in its code says which step is slow. A profiler: A tool that measures where a running program spends its time, function by function and line by line, so that you speed up the part that matters instead of the part you suspect. shows where a running program spends its time, so you can speed up the part that matters instead of the part you suspect. The common kind, a sampling profiler: A profiler that interrupts the running program at regular intervals and records its call stack each time; the functions in the most samples are where the time goes. perf, Instruments' Time Profiler and Visual Studio's CPU Usage work this way., interrupts the program thousands of times a second and records the call stack each time; the functions that appear in the most samples are where the time goes.

To profile C++, build the version you would ship with debug information and frame pointers, -O2 -g -fno-omit-frame-pointer, run it under perf on Linux, Instruments or sample on macOS, or Visual Studio’s CPU Usage tool on Windows, and read two numbers per function: its self time: The share of a profile's samples taken while the processor was running a function's own instructions, not the functions it called. time, spent in its own code, and its total time: The share of a profile's samples whose call stack contains a function: its self time plus the time of everything it calls. perf calls it children. time, which includes everything it calls.

Measure the build you ship, with symbols and frame pointers

This program counts the words of a two-million-word text and prints the five most common. It works, but slowly:

The program

wordfreq_v1.cpp

// Counts how often each word occurs and prints the five most common: the first, slow version.
#include "text.hpp"
#include <algorithm>
#include <cctype>
#include <ctime>
#include <print>
#include <string>
#include <utility>
#include <vector>

using Counts = std::vector<std::pair<std::string, int>>;

std::string to_lower(std::string word) {
    for (char& c : word) c = static_cast<char>(std::tolower(static_cast<unsigned char>(c)));
    return word;
}

// The entry of `word`, added the first time it is seen: a linear search through every word so far.
std::pair<std::string, int>& entry_for(Counts& counts, const std::string& word) {
    for (auto& entry : counts)
        if (entry.first == word) return entry;
    counts.emplace_back(word, 0);
    return counts.back();
}

Counts count_words(const std::string& text) {
    Counts counts;
    std::size_t i = 0;
    while (i < text.size()) {
        while (i < text.size() && !std::isalpha(static_cast<unsigned char>(text[i]))) ++i;
        const std::size_t start = i;
        while (i < text.size() && std::isalpha(static_cast<unsigned char>(text[i]))) ++i;
        if (i > start) ++entry_for(counts, to_lower(text.substr(start, i - start))).second;
    }
    return counts;
}

int main() {
    const std::string text = make_text(2'000'000);
    const std::clock_t start = std::clock();  // processor time, which other programs on the computer barely affect
    Counts counts = count_words(text);
    std::ranges::sort(counts, [](const auto& a, const auto& b) { return a.second > b.second || (a.second == b.second && a.first < b.first); });
    const long ms = (std::clock() - start) * 1000L / CLOCKS_PER_SEC;
    std::print("{} different words; most common:", counts.size());
    for (int k = 0; k < 5; ++k) std::print(" {} {}", counts[k].first, counts[k].second);
    std::println("\ncounted in {} ms of processor time", ms);
}

text.hpp

// A made-up text of `words` words, the same on every run: 3,000 different words, a few of them very common,
// in mixed case and with punctuation, like real prose.
#pragma once
#include <cstdint>
#include <string>
#include <vector>

inline std::string make_text(std::size_t words) {
    const char* syllables[] = {"ka", "ri", "to", "ma", "ne", "su", "lo", "vi", "da", "pe", "go", "zu"};
    std::vector<std::string> vocabulary;
    for (int n = 0; vocabulary.size() < 3000; ++n) {
        std::string w;
        for (int k = n; ; k /= 12) {
            w += syllables[k % 12];
            if (k < 12) break;
        }
        vocabulary.push_back(w);
    }
    std::uint64_t state = 2026;  // a fixed seed: the same text every time
    auto next = [&state] { state = state * 6364136223846793005ULL + 1442695040888963407ULL; return state >> 33; };
    std::string text;
    for (std::size_t i = 0; i < words; ++i) {
        std::size_t index = next() % 3000;
        if (next() % 4 != 0) index = index * index / 3000 % 300;  // three in four words come from the 300 most common
        std::string w = vocabulary[index];
        if (next() % 10 == 0) w[0] = static_cast<char>(w[0] - 'a' + 'A');
        text += w;
        text += (next() % 12 == 0) ? ", " : " ";
    }
    return text;
}

Built for profiling, and timed Python · timing_v1_session.py

from session import run

# The first version, built the way a profiler needs it, and timed by its own clock.
run("""
clang++ -std=c++23 -O2 -g -fno-omit-frame-pointer wordfreq_v1.cpp -o wordfreq_v1 && ./wordfreq_v1
""")

Output

$ clang++ -std=c++23 -O2 -g -fno-omit-frame-pointer wordfreq_v1.cpp -o wordfreq_v1 && ./wordfreq_v1
3000 different words; most common: ka 31757 ri 14731 ma 12637 to 11947 su 10328
counted in 2675 ms of processor time

This output changes from run to run: Processor time changes a little from run to run; the counts are fixed, the milliseconds are the measurement.

Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 timing_v1_session.py

The program measures its own processor time with std::clock, which other programs on the computer disturb far less than a stopwatch. More than two seconds for two million words is slow, but the code does not say why. Lowering every word, copying it into a new std::string, the final sort and the search for each word’s entry are all suspects, and guessing among them is how hours get spent on the wrong one.

The build flags matter. -O2 profiles the code you ship, because an unoptimised build spends its time in different places. -g lets the profiler name functions and lines. -fno-omit-frame-pointer keeps the frame pointer: A register that points into the current function's stack frame, so that the chain of calls can be followed from each function to its caller. Compilers may drop it to free the register; -fno-omit-frame-pointer keeps it, which keeps sanitizer stack traces complete. More in “Finding memory bugs with AddressSanitizer and UBSan”, a register that links each stack frame to its caller; GCC drops it from -O1 upwards, and without it a profiler that walks frame pointers loses callers.

Checkpoint

Check: Building a program for profiling

Question 1 of 2 Why does the lesson profile an -O2 build rather than an -O0 build?
Choose one answer.

-O2 is the code you ship, and an unoptimised build spends its time in different places -O2 adds the names the profiler shows -O2 keeps the frame pointer perf cannot run -O0 programs

Show the answer: Question 1 of 2

Answer: -O2 is the code you ship, and an unoptimised build spends its time in different places

Profile the program you ship: -O2 for the code, -g for the names, -fno-omit-frame-pointer for the callers. Each flag does one job, and leaving one out costs that job.
Question 2 of 2 You build with -O2 -g but leave out -fno-omit-frame-pointer, then record with perf's frame-pointer walk. What goes wrong?
Choose one answer.

Functions have no names in the report The program becomes much slower perf refuses to record The profiler can lose callers, because GCC drops the frame pointer from -O1 upwards

Show the answer: Question 2 of 2

Answer: The profiler can lose callers, because GCC drops the frame pointer from -O1 upwards

A sampling profiler that walks frame pointers needs them in the program. -fno-omit-frame-pointer keeps that register, so every sample's stack reaches back to main.

Self time and total time point at the linear search

perf, the Linux kernel’s profiler, records samples with perf record and summarises them with perf report:

perf record and perf report on the slow version (recorded) Text · perf_v1.txt

$ g++ -std=c++23 -O2 -g -fno-omit-frame-pointer wordfreq_v1.cpp -o wordfreq_v1
$ perf record --call-graph fp -o v1.data ./wordfreq_v1
[a warning that kernel symbols are hidden from ordinary users: 12 lines]
3000 different words; most common: ka 31757 ri 14731 ma 12637 to 11947 su 10328
counted in 4909 ms of processor time
[ perf record: Woken up 6 times to write data ]
[ perf record: Captured and wrote 1.377 MB v1.data (11848 samples) ]
$ perf report -i v1.data --stdio --no-children --sort symbol -g none --percent-limit 1 -q -w 8,60
    41.27%  [.] memcmp                                                    -      -
    39.26%  [.] entry_for(std::vector<std::pair<std::__cxx11::basic_stri  -      -
    11.75%  [.] memcmp@plt                                                -      -
     2.30%  [.] count_words(std::__cxx11::basic_string<char, std::char_t  -      -
$ perf report -i v1.data --stdio --children --sort symbol -g none --percent-limit 5 -q -w 8,8,60
    99.97%     0.00%  [.] __libc_start_call_main                                    -      -
    99.97%     0.00%  [.] __libc_start_main_impl (inlined)                          -      -
    99.97%     0.00%  [.] _start                                                    -      -
    99.97%     0.00%  [.] call_init (inlined)                                       -      -
    99.84%     0.00%  [.] main                                                      -      -
    96.18%     2.30%  [.] count_words(std::__cxx11::basic_string<char, std::char_t  -      -
    92.42%    39.26%  [.] entry_for(std::vector<std::pair<std::__cxx11::basic_stri  -      -
    63.99%     0.00%  [.] bool std::operator==<char, std::char_traits<char>, std:: (inlined)  -      -
    63.18%     0.00%  [.] std::char_traits<char>::compare(char const*, char const* (inlined)  -      -
    41.37%    41.27%  [.] memcmp                                                    -      -
    11.76%    11.75%  [.] memcmp@plt                                                -      -

About this recording

perf 7.0.14 recorded it in a virtual machine with Ubuntu 26.04.1 LTS, Linux 7.0 and GCC 15.2.0 on arm64. Ubuntu lets ordinary users record with perf only once kernel.perf_event_paranoid is lowered; this run followed sudo sysctl -w kernel.perf_event_paranoid=1. The last two columns, which perf report adds when it sorts by symbol, give each function’s average instructions per cycle (IPC) and the share of its instructions with a sampled IPC; no sample of this run carried an IPC, so both show dashes.

Step through one of those samples, and how perf report adds all of them up:

Interactive · message by message How 11,848 samples become self and total time Step through one sample of the slow word count, then the two reports that perf report made from all of them.

The drawing shows the whole scene; its 7 steps are listed below.

Every step as text

Sampling starts. perf record --call-graph fp -o v1.data ./wordfreq_v1 starts the program and has it interrupted at regular intervals, thousands of times a second, with its call stack recorded each time.
One sample. At one interrupt the processor is inside memcmp. Following the frame pointers, perf records the chain below it: the string comparison inside entry_for, then count_words, then main.
Self and total. That sample counts as self time for memcmp alone, the function the processor was in, and as total time for every function on the chain: entry_for, count_words and main each get one.
All the samples. The run left 11,848 samples in v1.data. perf report adds them up, function by function, and prints each count as a share of all the samples.
Sorted by self time. The first report sorts by self time: memcmp 41 %, entry_for 39 %, memcmp@plt 12 % and count_words 2 %. Most samples landed in the comparison and in the loop around it.
With children. The second report adds total time, which perf calls children: count_words is in 96 % of the stacks and entry_for, the linear search, in 92 %.
What is left. Everything outside entry_for is under 8 % of the samples. Making all of it faster could save at most that much time, so the search is the thing to change.

The first report sorts functions by self time: memcmp holds 41 % of the samples, entry_for 39 % and memcmp@plt, the stub that calls memcmp, 12 %. The second adds total time, which perf calls children: count_words is in 96 % of the stacks and entry_for, the linear search, in 92 %. So almost all of the time goes into comparing each new word with every word seen so far. Lowering, copying and sorting together take less than 8 %, so tuning any of them could never save more than about 8 % of the time.

Key idea

Self time shows where the processor was, and total time shows who asked for that work: here, the linear search.

Checkpoint

Check: Self time and total time

Question 1 of 2 One sample finds the processor inside memcmp, with entry_for, count_words and main on the stack below it. Which function gets self time from that sample?
Choose one answer.

main, because every stack starts there All four functions entry_for, because it called memcmp memcmp only

Show the answer: Question 1 of 2

Answer: memcmp only

Each sample adds one to the self count of the function the processor was in, and one to the total count of every function on its stack. That is how memcmp reached 41 % self and entry_for 92 % total.
Question 2 of 2 Everything outside entry_for held about 8 % of the samples. If all of that code took no time at all, at most what share of the run time would you save?
Type a number.
Your answer %

Show the answer: Question 2 of 2

Answer: 8 % (anything from 7.5 to 8.5 counts)

The samples outside entry_for stand for about 8 % of the run time, and removing all of it removes at most those 8 %. The 92 % in the search is untouched, which is why the search is the thing to change.

A flame graph draws the samples as nested bars

A flame graph: A drawing of a profile as nested bars: each bar is a function, as wide as its share of the samples and standing on the function that called it. Height is call depth; left to right is not time. turns the same samples into one picture:

The slow word count as a flame graph, from the recorded perf samples

Text description of the diagram

Each bar is a function. Its width is the share of the 11,848 samples whose call stack contained it, and each bar sits on the function that called it, so the stack grows upwards. The order from left to right means nothing; it is not time.

The bottom bar, main, spans the whole width: 99.8 % of the samples.
On it, count_words takes 96.2 %. Beside it, a narrow bar for make_text, which builds the text before the count, takes 2.3 %.
On count_words, entry_for, the linear search, takes 92.4 %.
On entry_for, std::operator==, the string comparison of every entry with the new word, takes 64.0 %, and std::char_traits::compare, inlined into it, takes 63.2 %.
At the top, memcmp takes 41.4 % and memcmp@plt, the stub that calls memcmp in the C library, takes 11.8 %.

The widest bars near the top are where the time goes: comparing strings inside the linear search, about 92 % of the samples in all.

Each bar is a function, as wide as its share of the samples, standing on the function that called it. The order from left to right means nothing: Brendan Gregg’s description of the format sorts frames alphabetically so that equal stacks merge, and this one puts the widest first. Read it from the top: the widest bars at the top edge are where the processor was, and the bars beneath them say who asked for that work. Here one tower dominates, from main through entry_for to memcmp: the 92 % of total time that the report gave entry_for, visible at a glance.

Checkpoint

Check: Reading a flame graph

Question 1 of 2 Where in a flame graph do you look first to find where the processor was?
Choose one answer.

The bottom bar The bar furthest to the left The tallest tower, wherever it is narrow The widest bars at the top edge

Show the answer: Question 1 of 2

Answer: The widest bars at the top edge

Read a flame graph from the top: the widest bars at the top edge are where the processor was, and the bars beneath them say who asked for that work.
Question 2 of 2 Two bars sit side by side at the same height, make_text to the right of count_words. What does that tell you about when they ran?
Choose one answer.

They ran at the same moment, on two threads make_text ran after count_words Nothing, because left to right is not time make_text called count_words

Show the answer: Question 2 of 2

Answer: Nothing, because left to right is not time

Width is a share of samples and height is the call depth. Brendan Gregg's description of the format sorts frames alphabetically so that equal stacks merge, which is why left to right can never mean time.

Every system has a sampling profiler, and counters give exact numbers

System	Tool	How it measures	Start it with
Linux	perf	Samples call stacks	`perf record -g ./app`, then `perf report`
macOS	Instruments, Time Profiler	Samples call stacks	Xcode’s Instruments app
macOS	`sample`	Samples call stacks	`sample <pid> 5`
Windows	Visual Studio CPU Usage	Samples call stacks	Debug, Performance Profiler (Alt+F2)
Any	Clang coverage counters	Counts every line	`-fprofile-instr-generate -fcoverage-mapping`
Any	Tracy	Times marked zones	Markers in the code, and Tracy’s viewer

Instruments comes with Xcode, not with the Command Line Tools, and its Time Profiler template shows a call tree.
sample ships with macOS and prints a call tree of the samples it takes from a running process.
Clang’s coverage counters count how often every line runs, exactly, at the cost of a slower instrumented build.
Tracy is an open-source frame and sampling profiler for games and other applications: it times zones you mark in the code.

Checkpoint

Check: Choosing a profiler

Question 1 of 2 You profile on a Mac that has the Command Line Tools but not Xcode. Which tool of the table can you use?
Choose one answer.

Visual Studio's CPU Usage sample perf Instruments with its Time Profiler

Show the answer: Question 1 of 2

Answer: sample

Every system has a sampling profiler: perf on Linux, Instruments or sample on macOS, Visual Studio's CPU Usage on Windows. On a Mac without Xcode, sample is the one that is already there.
Question 2 of 2 You need to know exactly how many times a line ran, not its share of the samples. Which tool of the table gives that?
Choose one answer.

sample perf Clang's coverage counters Tracy

Show the answer: Question 2 of 2

Answer: Clang's coverage counters

Sampling tells you where the time goes; counters tell you how often code runs. Clang's -fprofile-instr-generate -fcoverage-mapping build gives the second.

Check yourself

Nine questions on the profiling build, self and total time, flame graphs and each system’s profiler, in the order your answers call for; every choice comes with its reason.

Fixing the word count, step by step, with numbers Pro: opens with a Pro, Premium or Ultimate pass

A profile tells you where the time goes; counting and re-measuring tell you whether a fix worked. This part fixes the word count in two steps, measuring after each, and shows when to stop.

In this part:

Exact counts: how many comparisons the search made
Version 2: a hash table instead of a list
Version 3: removing the small costs, and when to stop

2 diagrams · 5 runnable examples (C++, Python, Text) · about 7 minutes

Opening this part with your pass…

This part opens with any MySmartCoPilot pass.

Pro pass · 1 week₹79$2.99
30 days₹179$5.99
1 year₹1,699$45

One payment; a pass never renews. The price shown is the total you pay: no fees or charges are added at checkout.

Choose a pass I have a licence key

It also opens every other in-depth part in Learn and every Pro tool. Sales are final, except where the law gives you a right to withdraw, for example in the EU and the UK (refund policy).

perf in practice: permissions, call graphs and inlining Pro: opens with a Pro, Premium or Ultimate pass

A profile is only as good as its stacks. This part sets up perf on Ubuntu, reproduces what missing frame pointers do to a profile, and shows how inlining and the other systems’ profilers change what you read.

In this part:

Letting perf run on Ubuntu
Call graphs without frame pointers, reproduced
Inlined functions in a profile
Profiling on macOS and Windows

2 runnable examples (Text) · about 6 minutes

Opening this part with your pass…Pro part. It opens with the same pass as the part above.

Interview questions

Fresher to mid · Warm-up

What is the difference between a sampling profiler and an instrumenting profiler?

A sampling profiler, such as perf, Instruments' Time Profiler or Visual Studio's CPU Usage, interrupts the running program at regular intervals and records its call stack. It changes the program very little and runs on the optimised build you ship, but its numbers are shares of samples, so rare or very short functions may not appear. An instrumenting profiler adds code to the program itself: compiler counters such as Clang's coverage, or timers around zones you mark, as Tracy does. It gives exact counts or exact times for what it instruments, but the added code costs time and can change what it measures. A good habit is to find the hot spot with sampling, then count or time it exactly with instrumentation.
Fresher to mid

You made a parsing function three times faster in a micro-benchmark, but the whole program became only 4 % faster. Explain the numbers, and describe how you should have chosen what to optimise.

Model answer, Pro. Model answers open with any pass; prices are in the notice on this page. First step and follow-ups
Mid to senior

After a release, a C++ service's p99 latency doubled, yet a CPU profile under the same load looks the same as before and CPU use did not rise. Why can a CPU profile miss the cause, and what do you measure instead?

Model answer, Pro. First step and follow-ups
Mid to senior

The top line of a perf report is memcpy, with 35 % self time, in a C++ program that never calls memcpy directly. What does that tell you, and how do you find the code to change?

Model answer, Pro. First step and follow-ups

Key takeaways

Measure before optimising: profile the optimised build, with -g and -fno-omit-frame-pointer.
Self time is a function’s own code; total time includes its callees. Fix what is wide at the top of the stack.
A flame graph’s width is a share of samples, its height the call depth, and left to right means nothing.
Use perf on Linux, Instruments or sample on macOS and Visual Studio’s CPU Usage on Windows; count lines exactly with Clang’s coverage counters when you need numbers rather than shares.

Terms in this lesson

flame graph: A drawing of a profile as nested bars: each bar is a function, as wide as its share of the samples and standing on the function that called it. Height is call depth; left to right is not time.
profiler: A tool that measures where a running program spends its time, function by function and line by line, so that you speed up the part that matters instead of the part you suspect.
sampling profiler: A profiler that interrupts the running program at regular intervals and records its call stack each time; the functions in the most samples are where the time goes. perf, Instruments' Time Profiler and Visual Studio's CPU Usage work this way.
self time: The share of a profile's samples taken while the processor was running a function's own instructions, not the functions it called.
total time: The share of a profile's samples whose call stack contains a function: its self time plus the time of everything it calls. perf calls it children.

References

perf wiki (Linux kernel community)
perf-record(1) (Debian manual pages) (Debian)
perf-report(1) (Debian manual pages) (Debian)
Perf events and tool security (The Linux Kernel documentation)
Flame Graphs (Brendan Gregg)
Source-based Code Coverage (Clang documentation) (LLVM Project)
Options That Control Optimization (GCC) (GNU Project)
Analyzing main thread activity (Profiling apps using Instruments) (Apple)
CPU profiling in the Performance Profiler (Visual Studio) (Microsoft)
Tracy Profiler (Bartosz Taudul (Tracy project))

Report a problem with this lesson

In this module: Build systems, testing and tooling

CMake for C++: targets, properties and presets Free: 20 minutes
Static and shared libraries in C++ Free core, Pro depth: 7 of 20 minutes free
Unit tests with GoogleTest or Catch2 and CTest Free core, Pro depth: 7 of 20 minutes free
vcpkg and Conan: C++ dependency management Free core, Pro depth: 7 of 20 minutes free
clang-tidy, clang-format and static analysis Free core, Pro depth: 6 of 15 minutes free
Debugging C++ with LLDB and GDB Free core, Pro depth: 6 of 15 minutes free
Sanitizer and hardened C++ build presets Free core, Pro depth: 4 of 10 minutes free
Profiling C++: finding where the time goes Free core, Pro depth: 6 of 15 minutes free
Portability across compilers, platforms and judges Free core, Pro depth: 4 of 10 minutes free

Your country

Profiling C++: finding where the time goes

What you will learn

Before you start

What you will learn

Before you start

Measure the build you ship, with symbols and frame pointers

Self time and total time point at the linear search

A flame graph draws the samples as nested bars

Every system has a sampling profiler, and counters give exact numbers

Check yourself

Fixing the word count, step by step, with numbers Pro: opens with a Pro, Premium or Ultimate pass

perf in practice: permissions, call graphs and inlining Pro: opens with a Pro, Premium or Ultimate pass

Interview questions

What is the difference between a sampling profiler and an instrumenting profiler?

You made a parsing function three times faster in a micro-benchmark, but the whole program became only 4 % faster. Explain the numbers, and describe how you should have chosen what to optimise.

After a release, a C++ service's p99 latency doubled, yet a CPU profile under the same load looks the same as before and CPU use did not rise. Why can a CPU profile miss the cause, and what do you measure instead?

The top line of a perf report is memcpy, with 35 % self time, in a C++ program that never calls memcpy directly. What does that tell you, and how do you find the code to change?

Key takeaways

References

Related tools