Unit tests with GoogleTest or Catch2 and CTest

In the Fraction class of this lesson, an operator+ that adds the denominators instead of multiplying them compiles, runs and returns 1/1 for 1/6 + 1/3. A unit test is a small program that calls one piece of your code, a function or a class, and checks what it returns, so that mistake fails a test. In C++ the job is split between two tools. A test framework provides the macros you write tests with and the report when a check fails; GoogleTest and Catch2 are two open-source frameworks for this, and doctest is a lighter third. CTest, which ships with CMake, is the runner: it keeps the list of every test in the project, runs them, in parallel if you ask, and says which ones failed, with an exit statusexit status: The number a program hands back when it ends, the value main returns. 0 means success; a shell keeps the last status in $?, and && runs the next command only after a status of 0. More in “How C++ compiles: preprocess, compile and link” that a CI job can act on.

This lesson writes unit tests with GoogleTest and CTest for that Fraction class, reads the failing run, and shows what the same tests look like in Catch2.

A framework writes the checks, CTest runs the tests

CMake joins the two tools: FetchContent downloads a pinned framework version at configure time, enable_testing() turns testing on, and gtest_discover_tests (or the Catch2 and doctest equivalents) registers each test with CTest. The three frameworks compare like this:

GoogleTest’s checks name what they compare: EXPECT_EQ keeps going after a failure, ASSERT_EQ stops the test, and EXPECT_THROW checks that an exception is thrown. Catch2 and doctest look at the whole expression instead, so a plain CHECK(a == b) reports both values when it fails. CTest knows nothing about any framework: a test is a command, and it passes when that command exits with status 0. Discoverytest discovery: Asking a built test program for its list of tests, as gtest_discover_tests does after each build, so that each test becomes a CTest test of its own that can be run, selected and timed alone. asks the built test program for its list of tests, so each TEST or TEST_CASE becomes its own CTest test that can be run, selected and timed alone.

The diagram follows one test file from its source to CTest’s report:

More that the runner and the frameworks can do

• Catch2’s SECTIONs run one test case once per section, each time from a fresh copy of its setup code.
• doctest is a single header that compiles quickly, and its tests may sit in the same file as the code.
• Test presets in CMakePresets.json give a CTest run a name, so developers and CI run exactly the same suite.
• Sanitizersanitizer: A compiler option, such as -fsanitize=address or -fsanitize=undefined, that builds extra checks into a program, so that memory errors and undefined behaviour are reported, with their source line, while it runs. More in “Finding memory bugs with AddressSanitizer and UBSan” builds run the same tests with AddressSanitizer and UBSan, which turn hidden memory errors and undefined behaviour into reports.
• ctest -j 8 runs tests in parallel, -R selects them by name, and --repeat until-fail:50 repeats them to expose a flaky test.
• Fixtures and parameterised tests (TEST_F, TEST_P) share setup between tests and run one test over many inputs.

Six GoogleTest tests catch the operator+ bug

Fraction keeps a fraction in lowest terms with a positive denominator and throws std::invalid_argument for a zero denominator. The tests check one behaviour each, and their names say which:

#pragma once

#include <cstddef>
#include <ostream>
#include <string>

namespace frac {

// A fraction kept in lowest terms, with a positive denominator: Fraction(2, -4) is -1/2.
class Fraction {
public:
    Fraction(long long numerator, long long denominator);  // throws std::invalid_argument for a zero denominator

    long long numerator() const { return num_; }
    long long denominator() const { return den_; }
    std::string to_string() const;

    friend Fraction operator+(const Fraction& a, const Fraction& b);
    friend Fraction operator*(const Fraction& a, const Fraction& b);
    friend bool operator==(const Fraction& a, const Fraction& b) = default;

private:
    long long num_;
    long long den_;
};

std::ostream& operator<<(std::ostream& out, const Fraction& f);

// A hash, so that fractions can go into an unordered_set.
std::size_t hash_value(const Fraction& f);

}  // namespace frac

#include "fraction/fraction.hpp"

#include <numeric>
#include <stdexcept>

namespace frac {

Fraction::Fraction(long long numerator, long long denominator) {
    if (denominator == 0) throw std::invalid_argument("denominator is zero");
    if (denominator < 0) {
        numerator = -numerator;
        denominator = -denominator;
    }
    const long long g = std::gcd(numerator, denominator);
    num_ = numerator / g;
    den_ = denominator / g;
}

std::string Fraction::to_string() const { return std::to_string(num_) + "/" + std::to_string(den_); }

Fraction operator+(const Fraction& a, const Fraction& b) {
#ifdef FRACTION_INJECT_BUG
    // The bug a test should catch: the denominators are added instead of multiplied.
    return Fraction(a.num_ * b.den_ + b.num_ * a.den_, a.den_ + b.den_);
#else
    return Fraction(a.num_ * b.den_ + b.num_ * a.den_, a.den_ * b.den_);
#endif
}

Fraction operator*(const Fraction& a, const Fraction& b) { return Fraction(a.num_ * b.num_, a.den_ * b.den_); }

std::ostream& operator<<(std::ostream& out, const Fraction& f) { return out << f.to_string(); }

std::size_t hash_value(const Fraction& f) {
    // Signed arithmetic: overflows, which is undefined behaviour, once the numerator passes about 9.2e12.
    const long long h = f.numerator() * 1'000'003 + f.denominator();
    return static_cast<std::size_t>(h);
}

}  // namespace frac

#include <stdexcept>

#include <gtest/gtest.h>

#include "fraction/fraction.hpp"

using frac::Fraction;

TEST(Fraction, StoresLowestTerms) {
    const Fraction f(6, 8);
    EXPECT_EQ(f.numerator(), 3);
    EXPECT_EQ(f.denominator(), 4);
}

TEST(Fraction, KeepsTheSignInTheNumerator) {
    EXPECT_EQ(Fraction(2, -4), Fraction(-1, 2));
}

TEST(Fraction, AddsWithDifferentDenominators) {
    EXPECT_EQ(Fraction(1, 6) + Fraction(1, 3), Fraction(1, 2));
}

TEST(Fraction, Multiplies) {
    EXPECT_EQ(Fraction(2, 3) * Fraction(9, 4), Fraction(3, 2));
}

TEST(Fraction, RejectsAZeroDenominator) {
    EXPECT_THROW(Fraction(1, 0), std::invalid_argument);
}

TEST(Fraction, EqualFractionsHashEqual) {
    EXPECT_EQ(frac::hash_value(Fraction(1, 2)), frac::hash_value(Fraction(2, 4)));
    EXPECT_EQ(frac::hash_value(Fraction(9'500'000'000'000, 7)), frac::hash_value(Fraction(19'000'000'000'000, 14)));
}

cmake_minimum_required(VERSION 3.28...4.4)
project(fraction LANGUAGES CXX)

set(CMAKE_CXX_EXTENSIONS OFF)
set(CMAKE_CXX_SCAN_FOR_MODULES OFF)

add_library(fraction src/fraction.cpp)
target_include_directories(fraction PUBLIC include)
target_compile_features(fraction PUBLIC cxx_std_23)
option(FRACTION_INJECT_BUG "Build operator+ with a bug, to see a failing test" OFF)
if(FRACTION_INJECT_BUG)
  target_compile_definitions(fraction PRIVATE FRACTION_INJECT_BUG)
endif()

# Tests: only when this is the top-level project, so projects that add fraction as a dependency do not build them.
if(PROJECT_IS_TOP_LEVEL)
  enable_testing()
  include(FetchContent)

  # GoogleTest 1.18.0, pinned by its archive's SHA-256: a changed download fails the configure step.
  FetchContent_Declare(googletest
    URL https://github.com/google/googletest/releases/download/v1.18.0/googletest-1.18.0.tar.gz
    URL_HASH SHA256=6e3191c1455468b3fc35a417fb565c1c5071aee1b7e7f85e30cf48a98d37d8b5)
  set(BUILD_GMOCK OFF CACHE BOOL "" FORCE)       # this suite uses no mocks
  set(INSTALL_GTEST OFF CACHE BOOL "" FORCE)     # nothing of GoogleTest is installed with the project
  set(gtest_force_shared_crt ON CACHE BOOL "" FORCE)  # Windows: the same C run-time library as the project
  FetchContent_MakeAvailable(googletest)

  add_executable(fraction_gtest tests/fraction_gtest.cpp)
  target_link_libraries(fraction_gtest PRIVATE fraction GTest::gtest_main)
  include(GoogleTest)
  gtest_discover_tests(fraction_gtest)

  # The same tests with Catch2 3.16.1 and doctest 2.5.3, when asked for.
  option(FRACTION_CATCH2 "Also build the Catch2 version of the tests" OFF)
  if(FRACTION_CATCH2)
    FetchContent_Declare(Catch2
      URL https://github.com/catchorg/Catch2/archive/refs/tags/v3.16.1.tar.gz
      URL_HASH SHA256=d16bd1b8b2364918bd472cb216394acb93bde6784890df0d91bccfb3f28d777d)
    FetchContent_MakeAvailable(Catch2)
    add_executable(fraction_catch2 tests/fraction_catch2.cpp)
    target_link_libraries(fraction_catch2 PRIVATE fraction Catch2::Catch2WithMain)
    list(APPEND CMAKE_MODULE_PATH ${catch2_SOURCE_DIR}/extras)
    include(Catch)
    catch_discover_tests(fraction_catch2 TEST_PREFIX "catch2: ")
  endif()

  option(FRACTION_DOCTEST "Also build the doctest version of the tests" OFF)
  if(FRACTION_DOCTEST)
    FetchContent_Declare(doctest
      URL https://github.com/doctest/doctest/archive/refs/tags/v2.5.3.tar.gz
      URL_HASH SHA256=174ebc4e769928959614789c5b4e9c3d0a0f81a62bb608756b127bfebfb21331)
    FetchContent_MakeAvailable(doctest)
    add_executable(fraction_doctest tests/fraction_doctest.cpp)
    target_link_libraries(fraction_doctest PRIVATE fraction doctest::doctest_with_main)
    include(${doctest_SOURCE_DIR}/scripts/cmake/doctest.cmake)
    doctest_discover_tests(fraction_doctest TEST_PREFIX "doctest: ")
  endif()
endif()

FetchContent_Declare names the GoogleTest 1.18.0 release archive and its SHA-256, so the configure step fails if the download ever differs. PROJECT_IS_TOP_LEVEL builds the tests only when this is the project being built, not when another project adds fraction as a dependency. gtest_main supplies main, and gtest_discover_tests runs the test program after each build to list its tests for CTest.

from session import run

# Configure (CMake downloads GoogleTest 1.18.0 from GitHub here: this session needs the network), build, run the tests.
run("""
cd fraction
cmake -S . -B build -G Ninja
""", keep=r"^(?!-- (?:The|Detecting|Check|Configuring|Generating|Performing|Found))")
run("""
cmake --build build
ctest --test-dir build --output-on-failure
""")
# The same tests against a library whose operator+ has a bug.
run("""
cmake -S . -B build -DFRACTION_INJECT_BUG=ON
cmake --build build
ctest --test-dir build --output-on-failure
""", keep=r"^(?!-- (?:Configuring|Generating|Build files))", max_lines=60)

$ cd fraction
$ cmake -S . -B build -G Ninja
-- Build files have been written to: build
[17 lines not shown]
$ cmake --build build
[1/8] Building CXX object _deps/googletest-build/googletest/CMakeFiles/gtest.dir/src/gtest-all.cc.o
[2/8] Building CXX object CMakeFiles/fraction.dir/src/fraction.cpp.o
[3/8] Linking CXX static library lib/libgtest.a
[4/8] Building CXX object _deps/googletest-build/googletest/CMakeFiles/gtest_main.dir/src/gtest_main.cc.o
[5/8] Linking CXX static library libfraction.a
[6/8] Building CXX object CMakeFiles/fraction_gtest.dir/tests/fraction_gtest.cpp.o
[7/8] Linking CXX static library lib/libgtest_main.a
[8/8] Linking CXX executable fraction_gtest
$ ctest --test-dir build --output-on-failure
Test project build
    Start 1: Fraction.StoresLowestTerms
1/6 Test #1: Fraction.StoresLowestTerms ...............   Passed    … sec
    Start 2: Fraction.KeepsTheSignInTheNumerator
2/6 Test #2: Fraction.KeepsTheSignInTheNumerator ......   Passed    … sec
    Start 3: Fraction.AddsWithDifferentDenominators
3/6 Test #3: Fraction.AddsWithDifferentDenominators ...   Passed    … sec
    Start 4: Fraction.Multiplies
4/6 Test #4: Fraction.Multiplies ......................   Passed    … sec
    Start 5: Fraction.RejectsAZeroDenominator
5/6 Test #5: Fraction.RejectsAZeroDenominator .........   Passed    … sec
    Start 6: Fraction.EqualFractionsHashEqual
6/6 Test #6: Fraction.EqualFractionsHashEqual .........   Passed    … sec

100% tests passed out of 6

Total Test time (real) =   … sec
$ cmake -S . -B build -DFRACTION_INJECT_BUG=ON
[3 lines not shown]
$ cmake --build build
[1/3] Building CXX object CMakeFiles/fraction.dir/src/fraction.cpp.o
[2/3] Linking CXX static library libfraction.a
[3/3] Linking CXX executable fraction_gtest
$ ctest --test-dir build --output-on-failure
Test project build
    Start 1: Fraction.StoresLowestTerms
1/6 Test #1: Fraction.StoresLowestTerms ...............   Passed    … sec
    Start 2: Fraction.KeepsTheSignInTheNumerator
2/6 Test #2: Fraction.KeepsTheSignInTheNumerator ......   Passed    … sec
    Start 3: Fraction.AddsWithDifferentDenominators
3/6 Test #3: Fraction.AddsWithDifferentDenominators ...***Failed    … sec
Running main() from build/_deps/googletest-src/googletest/src/gtest_main.cc
Note: Google Test filter = Fraction.AddsWithDifferentDenominators
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from Fraction
[ RUN      ] Fraction.AddsWithDifferentDenominators
tests/fraction_gtest.cpp:20: Failure
Expected equality of these values:
  Fraction(1, 6) + Fraction(1, 3)
    Which is: 1/1
  Fraction(1, 2)
    Which is: 1/2

[  FAILED  ] Fraction.AddsWithDifferentDenominators (… ms)
[----------] 1 test from Fraction (… ms total)

[----------] Global test environment tear-down
[==========] 1 test from 1 test suite ran. (… ms total)
[  PASSED  ] 0 tests.
[  FAILED  ] 1 test, listed below:
[  FAILED  ] Fraction.AddsWithDifferentDenominators

 1 FAILED TEST

    Start 4: Fraction.Multiplies
4/6 Test #4: Fraction.Multiplies ......................   Passed    … sec
    Start 5: Fraction.RejectsAZeroDenominator
5/6 Test #5: Fraction.RejectsAZeroDenominator .........   Passed    … sec
    Start 6: Fraction.EqualFractionsHashEqual
6/6 Test #6: Fraction.EqualFractionsHashEqual .........   Passed    … sec

83% tests passed, 1 tests failed out of 6

Total Test time (real) =   … sec

The following tests FAILED:
	  3 - Fraction.AddsWithDifferentDenominators (Failed)
Errors while running CTest
[exit status 8]

The same session as a process, from the six TEST blocks to the exit status CTest ended with, first against the correct library and then against the one with the bug:

The first ctest run passed all six tests, each run as its own process. Then the library was rebuilt with a bug in operator+, which adds the denominators instead of multiplying them, and one test failed. With --output-on-failure, CTest printed GoogleTest’s report for that test only: the file and line, the expression Fraction(1, 6) + Fraction(1, 3), the value it produced, 1/1, and the value expected, 1/2. The values are readable because the class has an operator<<; without one, GoogleTest prints the object’s bytes. CTest ended with 83% tests passed, 1 tests failed out of 6 and exit status 8, which fails any CI step that runs it.

Writing unit tests with GoogleTest is mostly writing good expectations: one behaviour per test, inputs at the edges (empty, zero, negative, one past a limit), and names that say what failed before you open the report.

Check yourself

Eight questions about frameworks, CTest and the failing run above, the least familiar idea first, each answered as soon as you choose.

The same tests in Catch2 and doctest, with their build costs measured Pro: opens with a Pro, Premium or Ultimate pass

Running the suite under sanitizers: a bug the tests passed over, caught Pro: opens with a Pro, Premium or Ultimate pass

Tests that find bugs: the StringUtils exercise, solved, with a rubric Pro: opens with a Pro, Premium or Ultimate pass

Interview questions

Key takeaways

• A framework (GoogleTest, Catch2, doctest) writes and checks tests; CTest runs them and reports through its exit status.
• Pin the framework with FetchContent_Declare(... URL ... URL_HASH SHA256=...) and register tests with gtest_discover_tests, so each test becomes its own CTest entry.
• ctest --output-on-failure shows the report of each failing test: the expression, the value it had and the value expected.
• Test one behaviour per test, at the edges of the input, and give it a name that says what broke.