C++ Module 23 – Design, idioms and maintainable C++
Testing C++: fakes, property tests, golden files
Choose the right C++ test for each layer, replace dependencies with fakes through interfaces, templates or the linker, and write property and golden tests.
What you will learn
- Choose the kind of test that fits each layer of a C++ program
- Replace dependencies with fakes through interfaces, templates or the linker
- Write property-based tests that shrink and golden-file tests that stay reviewable
- Distinguish fakes, stubs and mocks by what each one checks
- Reproduce a seeded failure on every standard library, and catch a test whose own code has undefined behaviour
Before you start
What you’ll learn · Before you start
What you will learn
- Choose the kind of test that fits each layer of a C++ program
- Replace dependencies with fakes through interfaces, templates or the linker
- Write property-based tests that shrink and golden-file tests that stay reviewable
- Distinguish fakes, stubs and mocks by what each one checks
- Reproduce a seeded failure on every standard library, and catch a test whose own code has undefined behaviour
Before you start
On this page
Testing C++ well means picking the kind of test by the kind of mistake it can catch. Unit tests check one function or class in microseconds, with its slow or unpredictable dependencies, such as the clock, the network or the file system, replaced by fakes. Integration tests run several parts together with real files or processes, and a few end-to-end tests run the whole program as its users do. Two more kinds find what hand-picked examples miss. A property test checks a rule that must hold for every input, such as “a merge of two sorted lists is sorted and loses nothing”, on hundreds of generated inputs. A golden-file test compares a whole output with a file that a person reviewed once. All of them should also run in sanitizer builds, where undefined behaviour: What a program does when it breaks a rule for which the C++ standard sets no requirements, such as reading past the end of an array. The compiler may assume it never happens, so the result can change with the compiler and the optimisation level. More in “Object lifetime and undefined behaviour in C++” is reported instead of hiding behind a lucky result.
Each kind of test catches a different mistake
Each kind exercises a different amount of the program, at a different cost in time:
| Kind of test | What it exercises | Typical speed | How many | What C++ adds |
|---|---|---|---|---|
| Unit | One function or class, its dependencies replaced | Microseconds to milliseconds | Most of the suite | A seam for every dependency the test replaces |
| Integration | Several parts with real files, processes or a database | Milliseconds to seconds | Fewer | Real libraries, linking, temporary folders |
| End-to-end | The installed program, as users run it | Seconds to minutes | A few | Release flags, the real configuration |
| Property | One function over hundreds of generated inputs | Milliseconds | One per rule worth stating | Seeds that reproduce on every standard library |
| Golden file | A whole output against a reviewed file | Milliseconds | One per output format | Masking what varies between runs |
Checkpoint
Check: Which kind of test for which mistake
A seam lets a test replace a dependency
A seam is a place where a test can swap what the code calls without editing the code. C++ offers three.
Three seams where a test can replace the clock
Text description of the diagram
The diagram has three panels, one under the other. In each, the code under test has two arrows: one to the real clock, used in production, and one to a fake, used in the test.
- Interface seam, chosen at run time: SessionStore holds a const Clock&. In production it refers to a SystemClock; in the test, to a FakeClock.
- Template seam, chosen at compile time: BasicSessionStore<C> holds a C by value. Production uses C = SteadyClock; the test uses C = FakeTime.
- Link seam, chosen by the linker: code calls clock_now(). The program links clock_real.cpp; the test links clock_fake.cpp.
- An interface seam passes the dependency as a reference to an abstract class, so a test passes a fake that derives from it.
- A template seam makes the dependency a template parameter, so a test instantiates the code with a fake type.
- A link seam declares a function in a header and lets the build link either the real definition or a fake one.
The interface seam is the simplest to read and costs a virtual call. The template seam costs nothing at run time but moves the code into a header. The link seam changes no code at all, which makes it the usual way into legacy code, but a test binary can hold one-definition rule: The rule that a program holds exactly one definition of each ordinary function it uses. An inline function may be defined in every translation unit that uses it, as long as all the definitions are identical. More in “How C++ compiles: preprocess, compile and link”.
Checkpoint
Check: Seams that replace a dependency
A property test finds a bug that example tests missed
merge_sorted() merges two sorted lists. Three example tests, the kind most suites start with, pass. The property
test generates 500 pairs of short sorted lists from a fixed seed: The value that sets a random number engine's starting state, such as 42 in std::mt19937 engine(42). The same seed gives the same sequence again; a seed from std::random_device makes each run different. More in “Random numbers in C++: engines, distributions, seeds” and checks the rule against a simple model: the
result must equal both lists put together and sorted.
merge/merge.h
// Merges two sorted lists into one sorted list. It has a bug that the example tests below never reach.
#pragma once
#include <cstddef>
#include <vector>
inline std::vector<int> merge_sorted(const std::vector<int>& a, const std::vector<int>& b) {
std::vector<int> out;
out.reserve(a.size() + b.size());
std::size_t i = 0, j = 0;
while (i < a.size() && j < b.size()) {
if (a[i] < b[j]) {
out.push_back(a[i++]);
} else if (b[j] < a[i]) {
out.push_back(b[j++]);
} else { // equal heads: take one and move both on
out.push_back(a[i++]);
++j;
}
}
while (i < a.size()) out.push_back(a[i++]);
while (j < b.size()) out.push_back(b[j++]);
return out;
}merge/merge_examples.cpp
// Three example tests of merge_sorted(), written the usual way: chosen inputs and their expected outputs.
#include <vector>
#include "check.h"
#include "merge.h"
int main() {
CHECK(merge_sorted({1, 4, 9}, {2, 3, 10}) == std::vector<int>({1, 2, 3, 4, 9, 10}));
CHECK(merge_sorted({}, {5, 6}) == std::vector<int>({5, 6}));
CHECK(merge_sorted({7}, {}) == std::vector<int>({7}));
std::println("example tests: {} failed of 3", failures);
return failures == 0 ? 0 : 1;
}merge/merge_properties.cpp
// The property of merge_sorted(): for any two sorted lists, the result is sorted and holds exactly the elements of
// both. Cases are two short sorted lists of small numbers, so equal elements are common.
#include <algorithm>
#include <format>
#include <random>
#include <utility>
#include <vector>
#ifdef FIXED
#include "merge_fixed.h"
#else
#include "merge.h"
#endif
#include "proptest.h"
using Case = std::pair<std::vector<int>, std::vector<int>>;
Case generate(std::mt19937& engine) {
Case c;
for (auto* list : {&c.first, &c.second}) {
const int length = prop::pick(engine, 0, 6);
for (int k = 0; k < length; ++k) list->push_back(prop::pick(engine, 0, 9));
std::ranges::sort(*list);
}
return c;
}
bool holds(const Case& c) {
const std::vector<int> merged = merge_sorted(c.first, c.second);
std::vector<int> expected = c.first;
expected.insert(expected.end(), c.second.begin(), c.second.end());
std::ranges::sort(expected);
return merged == expected; // sorted, and the same elements as both lists together
}
// Smaller candidates: drop one element from either list, lower one element by one (re-sorted), or lower every
// element of both lists by one when none is 0.
std::vector<Case> shrink(const Case& c) {
std::vector<Case> out;
const auto positive = [](const std::vector<int>& l) { return std::ranges::all_of(l, [](int x) { return x > 0; }); };
if (positive(c.first) && positive(c.second) && !(c.first.empty() && c.second.empty())) {
Case lower = c;
for (int& x : lower.first) --x;
for (int& x : lower.second) --x;
out.push_back(lower);
}
for (int side = 0; side < 2; ++side) {
const std::vector<int>& list = side == 0 ? c.first : c.second;
for (std::size_t k = 0; k < list.size(); ++k) {
Case smaller = c;
auto& l = side == 0 ? smaller.first : smaller.second;
l.erase(l.begin() + static_cast<std::ptrdiff_t>(k));
out.push_back(smaller);
if (list[k] > 0) {
Case lower = c;
auto& m = side == 0 ? lower.first : lower.second;
--m[k];
std::ranges::sort(m);
out.push_back(lower);
}
}
}
return out;
}
std::string show(const Case& c) {
return std::format("a = {}, b = {}, merged = {}", c.first, c.second, merge_sorted(c.first, c.second));
}
int main() { return prop::for_all<Case>("merge_sorted", 2026, 500, generate, holds, shrink, show) ? 0 : 1; }merge/proptest.h
// A small property-test runner: generate random cases from a seed, check a property on each, and shrink the first
// failing case to a smallest one that still fails. Values come straight from the engine (mt19937 produces the same
// numbers with every standard library), so a seed reproduces a failure on every compiler.
#pragma once
#include <cstdint>
#include <optional>
#include <print>
#include <random>
#include <string>
namespace prop {
// A random integer in [lo, hi], computed from the engine's output by this code, not by a standard distribution.
inline int pick(std::mt19937& engine, int lo, int hi) {
return lo + static_cast<int>(engine() % static_cast<std::uint32_t>(hi - lo + 1));
}
// Runs `property` on `cases` values from `generate`. On the first failure, keeps replacing the case with any smaller
// candidate from `shrink` that still fails, then reports both. Returns true if every case passed.
template <class Case, class Generate, class Property, class Shrink, class Show>
bool for_all(const char* name, std::uint32_t seed, int cases, Generate generate, Property property, Shrink shrink,
Show show) {
std::mt19937 engine(seed);
for (int n = 1; n <= cases; ++n) {
Case c = generate(engine);
if (property(c)) continue;
std::println("{}: failed on case {} of {} (seed {})", name, n, cases, seed);
std::println(" first failing case: {}", show(c));
int steps = 0;
for (bool smaller = true; smaller;) {
smaller = false;
for (const Case& candidate : shrink(c)) {
if (!property(candidate)) {
c = candidate;
smaller = true;
++steps;
break;
}
}
}
std::println(" shrunk in {} steps to: {}", steps, show(c));
return false;
}
std::println("{}: passed {} cases (seed {})", name, cases, seed);
return true;
}
} // namespace propfrom session import run
# The example tests, then the property test, on both compilers, then the property test against the fixed merge.
run("""
clang++ -std=c++23 -Wall -Wextra merge_examples.cpp -o examples && ./examples
clang++ -std=c++23 -Wall -Wextra merge_properties.cpp -o properties && ./properties
g++-16 -std=c++23 -Wall -Wextra merge_properties.cpp -o properties-gcc && ./properties-gcc
clang++ -std=c++23 -Wall -Wextra -DFIXED merge_properties.cpp -o properties-fixed && ./properties-fixed
""")Output
$ clang++ -std=c++23 -Wall -Wextra merge_examples.cpp -o examples && ./examples example tests: 0 failed of 3 $ clang++ -std=c++23 -Wall -Wextra merge_properties.cpp -o properties && ./properties merge_sorted: failed on case 3 of 500 (seed 2026) first failing case: a = [2, 4, 8], b = [3, 6, 7, 8, 9, 9], merged = [2, 3, 4, 6, 7, 8, 9, 9] shrunk in 15 steps to: a = [0], b = [0], merged = [0] [exit status 1] $ g++-16 -std=c++23 -Wall -Wextra merge_properties.cpp -o properties-gcc && ./properties-gcc merge_sorted: failed on case 3 of 500 (seed 2026) first failing case: a = [2, 4, 8], b = [3, 6, 7, 8, 9, 9], merged = [2, 3, 4, 6, 7, 8, 9, 9] shrunk in 15 steps to: a = [0], b = [0], merged = [0] [exit status 1] $ clang++ -std=c++23 -Wall -Wextra -DFIXED merge_properties.cpp -o properties-fixed && ./properties-fixed merge_sorted: passed 500 cases (seed 2026)
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 merge_session.py
Step through the same run and watch what the runner does after the first failure:
The drawing shows the whole scene; its 7 steps are listed below.
Every step as text
- A case from the seed. The runner seeds its engine with 2026 and builds the first case: two short sorted lists of values from 0 to 9.
- The rule holds. The property merges the two lists and compares the result with both lists put together and sorted. Cases 1 and 2 pass, so the runner moves on.
- Case 3 fails. Case 3 is a = [2, 4, 8] and b = [3, 6, 7, 8, 9, 9]. The merge returns [2, 3, 4, 6, 7, 8, 9, 9]: one of the two 8s is missing, so the property fails.
- Shrinking. The runner stops generating. It tries smaller variants of the failing case, checks each one with the same property and keeps any that still fails.
- The smallest case. After 15 steps no smaller variant fails, and the report ends with a = [0], b = [0], merged = [0]: one value in both lists, and one copy of it lost.
- The cause, named. That smallest case names the bug at once: on equal heads, merge_sorted() takes one element and skips the other. Both compilers found the same case at the same step.
- Fixed. The fixed merge takes both equal heads. Run again from the same seed, it passes all 500 cases.
The property failed on the third generated case, where 8 appears in both lists and the result holds only one 8. The
runner then shrank that case, keeping any smaller variant that still failed, until it reached a = [0], b = [0],
merged into [0]. That smallest case names the bug at once: on equal heads the function takes one element and
skips the other. Both compilers found the same case at the same step, because the runner computes its random values
from the engine’s output instead of using a standard random number distribution: An object such as std::uniform_int_distribution<int>(1, 6) that turns an engine's raw numbers into numbers of the range and shape you want. The standard says what it must produce, not how, so libraries may give different numbers. More in “Random numbers in C++: engines, distributions, seeds”. The fixed merge passed all 500 cases.
Property tests in C++ need no special library, although RapidCheck provides generators and automatic shrinking, and Catch2’s generators run one test over many values.
Key idea
A property states a rule for every input, and shrinking turns its first failure into the smallest case that still fails, which usually names the bug.
Checkpoint
Check: Property tests and shrinking
Fakes, stubs, mocks and golden files check different things
The first three are Any object that stands in for a real dependency in a test, so that the code under test can run without it: a fake, a stub or a mock.:
- A fake is a working, simplified implementation, such as a clock whose time moves only when the test says so.
- A stub returns fixed answers and checks nothing.
- A mock also checks how it was called against what the test expects, such as how many times the clock was read.
- A golden file holds an output that a person reviewed, and the test fails when the output changes.
- A fuzzer such as libFuzzer feeds a function a stream of generated inputs, looking for crashes rather than checking a stated rule.
- A sanitizer build runs the same tests with AddressSanitizer or UndefinedBehaviorSanitizer, which report each bad access or undefined operation where it happens.
The three doubles side by side:
| Criterion | Fake | Stub | Mock |
|---|---|---|---|
| What it is | A working, simplified implementation | Fixed answers and nothing more | A double that records its calls and checks them |
| Checks how it was called | No | No | Yes |
| A clock as each one | Time moves when the test says so | Always returns the same time | Counts how often it is read |
Checkpoint
Check: Fakes, stubs, mocks and golden files
Practise: choose the test, then the double
Twelve questions, one at a time and weakest idea first; a wrong choice tells you what it missed.
Practice that adapts to you
Practise this lesson
12 questions about 7 ideas of this lesson. Each answer is explained at once, and what you get right is kept in this browser.
Fakes through three seams, and a mock that fails a correct refactoring Pro: opens with a Pro, Premium or Ultimate pass
A clock is the classic dependency to replace: tests that wait for real time are slow and flaky. This part tests session expiry through each seam, then shows what a mock costs when the code changes.
In this part:
- The session store and its test doubles
- Thirty minutes of session time in a few microseconds
- A mock that fails a correct refactoring, reproduced
- The link seam, and the build that links both clocks
- Choosing a seam
- Which dependencies to replace, and with what
1 diagram · 2 runnable examples (Python, clock.h, fakes.h, sessions.h, sessions_test.cpp) · about 6 minutes
Opening this part with your pass…
This part opens with any MySmartCoPilot pass.
- Pro pass · 1 week₹79$2.99
- 30 days₹179$5.99
- 1 year₹1,699$45
One payment; a pass never renews. The price shown is the total you pay: no fees or charges are added at checkout.
Passes can’t be bought right now, so this part can’t be opened at the moment. The free part above stands on its own. If you have a pass, enter your licence key.
It also opens every other in-depth part in Learn and every Pro tool. Sales are final, except where the law gives you a right to withdraw, for example in the EU and the UK (refund policy).
Shrinking, seeds that reproduce, and a property that passed by accident Pro: opens with a Pro, Premium or Ultimate pass
A property test is only as good as its failures are reproducible. This part shows a seed that gives two compilers different cases, the two-bug interval exercise, and a test whose own undefined behaviour changed its verdict.
In this part:
- How the runner generates and shrinks
- One seed, two standard libraries
- Exercise: properties that expose two planted bugs
- A property that passed because of a dangling reference
6 runnable examples (C++, Python, interval_set.h, properties_starter.cpp) · about 6 minutes
Opening this part with your pass…
This part opens with any MySmartCoPilot pass.
- Pro pass · 1 week₹79$2.99
- 30 days₹179$5.99
- 1 year₹1,699$45
One payment; a pass never renews. The price shown is the total you pay: no fees or charges are added at checkout.
Passes can’t be bought right now, so this part can’t be opened at the moment. The free part above stands on its own. If you have a pass, enter your licence key.
It also opens every other in-depth part in Learn and every Pro tool. Sales are final, except where the law gives you a right to withdraw, for example in the EU and the UK (refund policy).
Golden files that survive two compilers and a code review Pro: opens with a Pro, Premium or Ultimate pass
A golden-file test is one comparison of a whole output, so it is cheap to write and catches any change at all. The work is keeping it stable across machines and reviewable when it must change.
In this part:
- The report and its golden test
- Hash order differs between libc++ and libstdc++
- Rules that keep golden files useful
- Line endings and other traps outside the code
1 diagram · 2 runnable examples (Python, expected/report.txt, golden_test.cpp, report.h) · about 6 minutes
Opening this part with your pass…
This part opens with any MySmartCoPilot pass.
- Pro pass · 1 week₹79$2.99
- 30 days₹179$5.99
- 1 year₹1,699$45
One payment; a pass never renews. The price shown is the total you pay: no fees or charges are added at checkout.
Passes can’t be bought right now, so this part can’t be opened at the moment. The free part above stands on its own. If you have a pass, enter your licence key.
It also opens every other in-depth part in Learn and every Pro tool. Sales are final, except where the law gives you a right to withdraw, for example in the EU and the UK (refund policy).
Interview questions
-
Mid to senior
A test gives a session a lifetime of 2 seconds, sleeps 1.9 seconds and asserts that the session is still valid, then sleeps again and asserts that it expired. The first assertion fails in about one CI run in fifty. Explain why, and redesign the code and the test.
-
Mid to staff
A property test failed on the Linux CI job and printed seed 81723. On your Mac the same seed passes, and the code under test is deterministic. What is the likely cause, how do you confirm it, and how do you make seeds reproduce everywhere?
-
Mid to senior
Your team writes a code generator that emits C++ headers. Would you test it with golden files or with assertions on the output? Describe the setup, what you would normalise, and how you would stop golden files from being updated without review.
Key takeaways
- Unit tests with fakes form most of a suite; integration and end-to-end tests cover what fakes cannot.
- A seam lets a test replace a dependency: an interface, a template parameter or the linker.
- A property test checks a rule on many generated inputs and shrinks a failure to its smallest case.
- A golden-file test compares a whole output with a reviewed file, after masking what varies between runs.
- Every suite also belongs in a sanitizer build, where undefined behaviour is reported instead of passing by luck.
References
- gMock for Dummies: what is gMock, fakes and mocks (GoogleTest project)
- Catch2: data generators (Catch2 project)
- RapidCheck: property based testing for C++ (RapidCheck project)
- Working Draft, Programming Languages: C++ (N4950): [rand.dist.general], [rand.predef], [unord.req.general] (ISO/IEC JTC1/SC22/WG21)
- AddressSanitizer (Clang documentation) (LLVM Project)
- UndefinedBehaviorSanitizer (Clang documentation) (LLVM Project)
- Options Controlling C++ Dialect: -Wdangling-reference (GNU Project)
- gitattributes: defining attributes per path (text, eol) (Git project)
- std::filesystem::temp_directory_path (cppreference) (cppreference.com)
- libFuzzer: a library for coverage-guided fuzz testing (LLVM Project)
- C++ Core Guidelines: I.25, prefer empty abstract classes as interfaces to class hierarchies (Standard C++ Foundation)
Related tools
Report a problem with this lesson
Kept only in this browser. Your Learn progress