C++ Module 16 – Concurrency: threads, atomics and the memory model
Concurrency-bug hunt: races and deadlocks
Hunt C++ concurrency bugs with a method that holds: reproduce under stress and ThreadSanitizer, find the cause, fix it, and keep a repeatable test.
What you will learn
- Find concurrency bugs with ThreadSanitizer, stress tests and careful reasoning
- Fix data races, deadlocks and lost wake-ups at their cause
- Write repeatable concurrency tests with stress loops, timeouts and test hooks
Before you start
What you’ll learn · Before you start
What you will learn
- Find concurrency bugs with ThreadSanitizer, stress tests and careful reasoning
- Fix data races, deadlocks and lost wake-ups at their cause
- Write repeatable concurrency tests with stress loops, timeouts and test hooks
Before you start
On this page
A job queue hangs at shutdown about once in every few thousand restarts, and ThreadSanitizer (TSan): A sanitizer of Clang and GCC, turned on with -fsanitize=thread, that watches a program's memory accesses while it runs and reports data races, naming both accesses and the threads that made them. More in “Data races, mutexes and lock guards in C++” reports nothing; a test that shut it down again and again made it hang in round 115. A concurrency-bug hunt follows the same loop as any bug hunt, with two changes, because these bugs depend on timing. Reproduce the bug with a A test that repeats one operation thousands of times, often with more threads than cores, so that a bug that depends on timing shows up in seconds instead of once in thousands of runs. that repeats the suspect operation thousands of times, with a timeout so that a hang counts as a failure, and run it under ThreadSanitizer, which reports a data race: Two accesses to the same memory location from different threads, at least one of them a write, with nothing such as a mutex or an atomic operation ordering them. A program with a data race has undefined behaviour. More in “Data races, mutexes and lock guards in C++” wherever two threads touch the same memory, one of them writing, without synchronization. Find the cause by asking, for each piece of shared state, what protects it and who changes it. Fix the cause, often by sharing less, rather than adding locks around the symptom. Then re-run many more times than the bug needed to appear, and keep the test.
Six steps find a bug that depends on timing
- Reproduce. Write a test that runs the operation in a loop, with more threads than cores, and treats a call that takes too long as a hang.
- Ask ThreadSanitizer. Build with
-fsanitize=thread -g -O1and run the same test. A report names both accesses of a data race; silence in a run that hit the bug points elsewhere. - Find the cause. For each shared variable, name the mutex or atomic that protects it, and look for the window in which another thread can act between two steps that must belong together.
- Make it fail every time. A test hook that holds one thread inside that window turns a rare failure into a certain one.
- Fix the cause, then re-run the stress test, the hook and ThreadSanitizer on both compilers.
- Keep the tests, so that the bug cannot return unnoticed.
Step through them with this lesson’s job queue:
The drawing shows the whole scene; its 7 steps are listed below.
Every step as text
- Reproduce. Shutdowns of the job queue hang once in a few thousand restarts. A test that shuts it down again and again, and counts a shutdown over 100 ms as a hang, failed in round 115.
- Ask ThreadSanitizer. The same test under ThreadSanitizer hung too, and printed no report. Had it reported a data race, the report would name both accesses, and the fix would start there.
- The shortcut. When ThreadSanitizer does report, the arrow aside applies: the report names both accesses and their threads, so fix that race, then re-run everything.
- Find the cause. Silence points to the order of the operations. For each shared variable, name what protects it: closed_ is atomic, but close() changes it without the queue's mutex.
- Make it fail every time. A test hook between the worker's check and its wait holds the worker in that window while the test calls close(): 20 of 20 shutdowns hung.
- Fix and re-run. close() now changes closed_ while holding the mutex. The hook test: 0 of 20 hung. The stress test: 10,000 shutdowns with each compiler and 1,000 under ThreadSanitizer, none hung.
- Keep the tests. Both tests stay in the suite: the hook test for this exact interleaving, the stress test for the ones nobody has thought of.
Each kind of concurrency bug has its own detector
| Bug | What you see | What finds it |
|---|---|---|
| Data race | Wrong values now and then, crashes, or nothing at all | ThreadSanitizer |
| deadlock: Two or more threads that each hold a lock and wait for one that another of them holds, so that none of them can go on. It almost always comes from taking the same mutexes in different orders. More in “Data races, mutexes and lock guards in C++” | A hang with the processor idle | A stack dump of every thread |
| lost wake-up: A wait that never ends because its notification came at the wrong moment: before the thread started waiting, as when the state changes between the waiter's check and its wait, or before the state it checks had changed. Nothing wakes the thread again. | A hang now and then, often at shutdown | A stress test with a timeout, then reasoning |
| Two steps that must belong together, such as checking for a key and then inserting it, each done under a lock but not under one lock together, so that another thread can act between them. | Work done twice, or duplicates | A stress test that counts |
| Ordering bug between atomics | Stale data, rarely, on some processors | A litmus test or a proof |
Only the first row is a data race, the kind ThreadSanitizer reports; the other four need a stack dump, a stress test or an argument from the rules. In a deadlock, every stuck thread waits in a lock that another stuck thread holds, so a stack dump of every thread shows the cycle at once.
Checkpoint
Check: The six steps for a timing bug · Which tool finds which kind of bug
A worked hunt: why does the job queue hang at shutdown?
JobQueue hands jobs to worker threads; close() says that no more will come, and each worker’s pop() then returns
no job once the queue is empty.
// A queue of jobs for worker threads. close() says that no more jobs will come: the workers finish the jobs that are
// left, and then pop() returns no job. This version has a bug that makes a shutdown hang now and then.
#pragma once
#include <atomic>
#include <condition_variable>
#include <deque>
#include <functional>
#include <mutex>
#include <optional>
class JobQueue {
public:
void push(int job) {
{
std::lock_guard lock(mutex_);
jobs_.push_back(job);
}
ready_.notify_one();
}
void close() {
closed_ = true; // atomic, so it seemed to need no lock
ready_.notify_all();
}
std::optional<int> pop() {
std::unique_lock lock(mutex_);
while (jobs_.empty()) {
if (closed_) return std::nullopt;
if (before_wait) before_wait(); // a test hook between the check and the wait
ready_.wait(lock);
}
int job = jobs_.front();
jobs_.pop_front();
return job;
}
std::function<void()> before_wait; // set only by tests, before any worker starts
private:
std::mutex mutex_;
std::condition_variable ready_;
std::deque<int> jobs_;
std::atomic<bool> closed_{false};
};shutdown_test.cpp does steps 1 and 2: it starts four workers, gives them four jobs, closes the queue and waits up to
100 ms, again and again, until a shutdown hangs:
// A stress test: start 4 workers, give them 4 jobs, close the queue, and repeat until a shutdown hangs or the number
// of rounds on the command line is reached. A shutdown that has not finished after 100 ms counts as a hang; close() is
// then called again so that the test can end. Build with -DFIXED to test the fixed queue.
#ifdef FIXED
#include "job_queue_fixed.h"
#else
#include "job_queue.h"
#endif
#include <atomic>
#include <chrono>
#include <cstdio>
#include <cstdlib>
#include <future>
#include <thread>
#include <vector>
using namespace std::chrono_literals;
bool shutdown_hangs() {
JobQueue queue;
std::atomic<int> running{4};
std::promise<void> all_stopped;
auto stopped = all_stopped.get_future();
std::vector<std::thread> workers;
for (int w = 0; w < 4; ++w)
workers.emplace_back([&] {
while (queue.pop()) {
}
if (running.fetch_sub(1) == 1) all_stopped.set_value();
});
for (int job = 1; job <= 4; ++job) queue.push(job);
queue.close();
bool hung = stopped.wait_for(100ms) == std::future_status::timeout;
while (stopped.wait_for(1ms) == std::future_status::timeout) queue.close(); // wake the stuck workers
for (auto& w : workers) w.join();
return hung;
}
int main(int argc, char** argv) {
int rounds = argc > 1 ? std::atoi(argv[1]) : 10000;
for (int round = 1; round <= rounds; ++round) {
if (shutdown_hangs()) {
std::printf("round %d: the shutdown hung\n", round);
return 1;
}
}
std::printf("no hang in %d shutdowns\n", rounds);
}from session import run
# Step 1 and step 2: the stress test of the job queue as it is, then the same test under ThreadSanitizer.
# Which round hangs first changes from run to run.
run("""
clang++ -std=c++23 -O2 shutdown_test.cpp -o shutdown_test && ./shutdown_test 50000
clang++ -std=c++23 -g -O1 -fsanitize=thread shutdown_test.cpp -o shutdown_test-tsan && ./shutdown_test-tsan 2000
""")Output
$ clang++ -std=c++23 -O2 shutdown_test.cpp -o shutdown_test && ./shutdown_test 50000 round 115: the shutdown hung [exit status 1] $ clang++ -std=c++23 -g -O1 -fsanitize=thread shutdown_test.cpp -o shutdown_test-tsan && ./shutdown_test-tsan 2000 round 444: the shutdown hung [exit status 1]
This output changes from run to run: which round hangs first changes from run to run
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 stress_session.py
The stress test turned a rare production hang into one that appears within seconds. ThreadSanitizer reported nothing,
which is a finding too: closed_ is atomic, so no access here is a data race, and the bug must be in the order of the
operations.
Checkpoint
Check: Reproducing under stress and ThreadSanitizer
Step 3: the check and the wait leave a window
How the job queue loses its wake-up
Text description of the diagram
The diagram is a sequence diagram with two lifelines, a worker thread inside pop(), which holds the queue's mutex, and a thread inside close(), which holds no lock. Time runs from top to bottom.
1 and 2. The worker sees that jobs_ is empty and reads closed_, which is still false. 3. The other thread sets closed_ to true. It does not take the mutex, so nothing stops it from running now. 4. It calls notify_all(). No thread is waiting on the condition variable yet, so the notification wakes nobody. 5. The worker calls ready_.wait(lock) and starts waiting. closed_ is already true and the notification has already been sent, so nothing will ever wake it, and the shutdown hangs.
A worker checks closed_ at line 29 and starts waiting at line 31, holding the mutex for both. close() changes
closed_ without the mutex, so it can run between the two lines: the worker has just seen false, the notification
goes out while nobody is waiting yet, and the worker then waits for a notification that already happened. This is a
lost wake-up. Making closed_ atomic removed the data race but not the window. Step through the same five events,
then the fixed close():
The drawing shows the whole scene; its 7 steps are listed below.
Every step as text
- The check. The worker holds the mutex, finds jobs_ empty and reads closed_ at line 29: false, so it is about to wait.
- The window. Between line 29 and the wait at line 31 there is a window. The test hook runs exactly here; in production it is a few nanoseconds wide.
- close() runs in the window. close() sets closed_ to true. It takes no mutex, so nothing stops it from running now. closed_ is atomic, so this is not a data race.
- A notification for nobody. close() calls notify_all(). No thread is waiting on ready_ yet, so the notification wakes nobody and is gone.
- The wait that never ends. The worker calls ready_.wait(lock) on its old answer. closed_ is already true and the notification has already been sent, so nothing will wake it: the shutdown hangs.
- The fix. The fixed close() locks the mutex before it sets closed_. The worker holds that mutex from its check until wait() releases it, in the same step as it starts to sleep, so the change lands before the check or after the wait, never in between.
- Woken, and it sees the change. Now notify_all() reaches a waiting worker. wait() returns with the mutex held, the loop checks closed_ again and pop() returns no job: 0 of 20 shutdowns hung with the hook.
Key idea
Change the state a waiter checks while holding the waiter’s mutex, then notify: an atomic flag removes the data race, not the window between the check and the wait.
Checkpoint
Check: The lost wake-up and its cause
Steps 4 to 6: a test hook makes it certain, and the fix holds
hook_test.cpp sets the queue’s
A function a test can set and the code calls at a chosen point, here between the check and the wait, so that the test can hold a thread there and force a rare order of events every time.,
which runs between the check and the wait. The hook counts down a latch and sleeps, and the test calls close() only
after the latch opens, so close always lands in the window. The fix, in job_queue_fixed.h, changes closed_ while
holding the mutex; closed_ no longer needs to be atomic:
hook_test.cpp
// The same shutdown, 20 times, with a test hook that holds one worker between its check of closed_ and its wait
// until close() has run. Build with -DFIXED to test the fixed queue.
#ifdef FIXED
#include "job_queue_fixed.h"
#else
#include "job_queue.h"
#endif
#include <chrono>
#include <cstdio>
#include <future>
#include <latch>
#include <thread>
using namespace std::chrono_literals;
int main() {
int hung = 0;
for (int round = 0; round < 20; ++round) {
JobQueue queue;
std::latch in_window(1);
queue.before_wait = [&in_window] {
in_window.count_down(); // tell the test the worker has checked closed_
std::this_thread::sleep_for(5ms); // and give close() time to run before the worker waits
};
auto stopped = std::async(std::launch::async, [&queue] {
while (queue.pop()) {
}
});
in_window.wait();
queue.close();
hung += stopped.wait_for(100ms) == std::future_status::timeout;
while (stopped.wait_for(1ms) == std::future_status::timeout) queue.close();
}
std::printf("%d of 20 shutdowns hung\n", hung);
}job_queue_fixed.h
// The job queue with the fix: close() changes closed_ while it holds the mutex, so the change cannot fall between a
// worker's check and its wait. closed_ no longer needs to be atomic: the mutex protects it.
#pragma once
#include <condition_variable>
#include <deque>
#include <functional>
#include <mutex>
#include <optional>
class JobQueue {
public:
void push(int job) {
{
std::lock_guard lock(mutex_);
jobs_.push_back(job);
}
ready_.notify_one();
}
void close() {
{
std::lock_guard lock(mutex_);
closed_ = true;
}
ready_.notify_all();
}
std::optional<int> pop() {
std::unique_lock lock(mutex_);
while (jobs_.empty()) {
if (closed_) return std::nullopt;
if (before_wait) before_wait();
ready_.wait(lock);
}
int job = jobs_.front();
jobs_.pop_front();
return job;
}
std::function<void()> before_wait;
private:
std::mutex mutex_;
std::condition_variable ready_;
std::deque<int> jobs_;
bool closed_ = false;
};from session import run
# Step 4: the test hook makes the bug happen every time, and the fixed queue survives it. Step 6: the stress test
# again with the fixed queue, with both compilers and under ThreadSanitizer.
run("""
clang++ -std=c++23 -O2 hook_test.cpp -o hook_test && ./hook_test
clang++ -std=c++23 -O2 -DFIXED hook_test.cpp -o hook_test-fixed && ./hook_test-fixed
clang++ -std=c++23 -O2 -DFIXED shutdown_test.cpp -o shutdown_test-fixed && ./shutdown_test-fixed 10000
g++-16 -std=c++23 -O2 -DFIXED shutdown_test.cpp -o shutdown_test-fixed-gcc && ./shutdown_test-fixed-gcc 10000
clang++ -std=c++23 -g -O1 -fsanitize=thread -DFIXED shutdown_test.cpp -o shutdown_test-fixed-tsan && ./shutdown_test-fixed-tsan 1000
""")Output
$ clang++ -std=c++23 -O2 hook_test.cpp -o hook_test && ./hook_test 20 of 20 shutdowns hung $ clang++ -std=c++23 -O2 -DFIXED hook_test.cpp -o hook_test-fixed && ./hook_test-fixed 0 of 20 shutdowns hung $ clang++ -std=c++23 -O2 -DFIXED shutdown_test.cpp -o shutdown_test-fixed && ./shutdown_test-fixed 10000 no hang in 10000 shutdowns $ g++-16 -std=c++23 -O2 -DFIXED shutdown_test.cpp -o shutdown_test-fixed-gcc && ./shutdown_test-fixed-gcc 10000 no hang in 10000 shutdowns $ clang++ -std=c++23 -g -O1 -fsanitize=thread -DFIXED shutdown_test.cpp -o shutdown_test-fixed-tsan && ./shutdown_test-fixed-tsan 1000 no hang in 1000 shutdowns
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 fix_session.py
With the hook, every shutdown of the old queue hung and none of the fixed one did. The stress test of the fix ran 10,000 shutdowns with each compiler and 1,000 under ThreadSanitizer without a hang; the old queue had hung within a few hundred. Keep both tests: the hook test proves this exact interleaving is handled, and the stress test watches for the ones nobody has thought of.
Checkpoint
Check: A test hook, the fix and its proof
Practise: the method, the detectors and the worked hunt
Thirteen questions follow, the weakest idea first.
Practice that adapts to you
Practise this lesson
13 questions about 8 ideas of this lesson. Each answer is explained at once, and what you get right is kept in this browser.
Deadlocks: from a hang to every thread's stack, the lock order and the fix Pro: opens with a Pro, Premium or Ultimate pass
A deadlock leaves no report and no crash, only a process that has stopped. This part catches one in the act, reads the stack of every thread to find the cycle, shows which tool stays silent, and fixes the lock order.
In this part:
- Reading a deadlock from the stacks of a live process
- The fix: one order for every lock
1 diagram · 2 runnable examples (C++, Python) · about 6 minutes
Opening this part with your pass…
This part opens with any MySmartCoPilot pass.
- Pro pass · 1 week₹79$2.99
- 30 days₹179$5.99
- 1 year₹1,699$45
One payment; a pass never renews. The price shown is the total you pay: no fees or charges are added at checkout.
Passes can’t be bought right now, so this part can’t be opened at the moment. The free part above stands on its own. If you have a pass, enter your licence key.
It also opens every other in-depth part in Learn and every Pro tool. Sales are final, except where the law gives you a right to withdraw, for example in the EU and the UK (refund policy).
Eight broken concurrent programs to repair, with the first symptom of each Pro: opens with a Pro, Premium or Ultimate pass
Each program below does something services do every day and has one concurrency bug. They cover four of the five rows of the table at the start of the lesson, and only three of them are data races that ThreadSanitizer can report.
In this part:
- Your turn: eight hunts
- Where to look
- Compare your fixes
5 runnable examples (1 hits, 2 cache, 3 close, 4 two waiters, 5 refresh, 6 listeners, 7 shared settings, 8 name, C header, Python) · about 5 minutes
Opening this part with your pass…Pro part. It opens with the same pass as the part above.
Making concurrency tests repeatable, with the numbers that decide how long to run them Pro: opens with a Pro, Premium or Ultimate pass
A concurrency test that passes once proves little. This part works out how many runs a rare bug needs, shows why a random delay is no substitute for a test hook, and measures what ThreadSanitizer costs, so that you can budget a test run.
In this part:
- How many runs, and what a clean run proves: the numbers
- A checklist for a concurrency test that stays repeatable
2 runnable examples (Python, runs_needed.cpp, tsan_cost.cpp, widen.cpp) · about 4 minutes
Opening this part with your pass…Pro part. It opens with the same pass as the part above.
Interview questions
-
Fresher to mid · Warm-up
What does ThreadSanitizer find, and which concurrency bugs can it miss?
ThreadSanitizer finds data races: two threads that access the same memory, at least one of them writing, with no happens-before between the accesses. It reports them where they happen in the run it watches, even when the output looks right, and names both accesses. It cannot find bugs that are not data races: a lost wake-up through an atomic flag, a check-then-act whose steps are each locked, or an ordering bug between atomics that are all accessed atomically. Its separate check for locks taken in opposite orders finds only that one kind of deadlock. It only sees code that runs during the test, and code that was not compiled with -fsanitize=thread can make it miss races or report false ones, as the Clang documentation warns. It also makes programs several times slower, so pair it with stress loops, timeouts and reasoning.
-
Mid to senior
A service hangs at shutdown about once in every few thousand deploys. Its job queue's workers wait on a condition variable, and close() sets an atomic flag and calls notify_all(). How do you find the bug, prove the fix, and keep it from coming back?
-
Senior to staff
A concurrency test fails in CI about once in 300 runs and nobody can reproduce it locally. What do you do, step by step, and how do you decide that a fix worked?
-
Senior to staff
A production process stops responding: the processor is idle and requests time out, and you may keep it alive for a few minutes. What do you collect, how do you recognise a lock-order deadlock in it, and how do you fix and prevent it?
Key takeaways
- Reproduce timing bugs with a stress loop that has a timeout, and run it under ThreadSanitizer.
- Silence from ThreadSanitizer means no data race showed in the code that ran; lost wake-ups, deadlocks and check-then-act bugs need other tools.
- Change the state a waiter checks while holding its mutex, then notify.
- Turn a rare bug into a certain one with a test hook, fix the cause, and keep both tests.
Terms in this lesson
- lost wake-up
- A wait that never ends because its notification came at the wrong moment: before the thread started waiting, as when the state changes between the waiter's check and its wait, or before the state it checks had changed. Nothing wakes the thread again.
References
- ThreadSanitizer (Clang documentation) (LLVM Project)
- Working Draft, Programming Languages: C++ (N4950): [intro.races], [thread.condition.condvar], [thread.lock.scoped], [thread.lock.algorithm] (ISO/IEC JTC1/SC22/WG21)
- C++ Core Guidelines: CP.42 Don't wait without a condition (Standard C++ Foundation)
- C++ Core Guidelines: CP.21 Use std::lock() or std::scoped_lock to acquire multiple mutexes (Standard C++ Foundation)
- C++ Core Guidelines: CP.22 Never call unknown code while holding a lock (Standard C++ Foundation)
- C++ Core Guidelines: CP.3 Minimize explicit sharing of writable data (Standard C++ Foundation)
- LLDB tutorial (process attach, thread backtrace all) (LLVM Project)
- Debugging with GDB: Debugging Programs with Multiple Threads (GNU Project)
- ctest(1): --repeat (Kitware (CMake documentation))
- Hardening Modes (libc++ documentation) (LLVM Project)
Related tools
Report a problem with this lesson
Kept only in this browser. Your Learn progress