Python Module 15 – Async programming with asyncio
Debugging asyncio with 3.14 introspection tools
See what a stuck asyncio program waits for: python -m asyncio ps and pstree, call graphs from inside the program, and await at the pdb prompt.
What you will learn
- Inspect running asyncio programs with python -m asyncio ps and pstree
- Display a task's call graph with asyncio.print_call_graph and capture it as objects
- Explain what attaching to another process needs on Linux, macOS and Windows
- Diagnose a stuck program from its await tree, including an await cycle
- Debug coroutines with pdb.set_trace_async and await at the debugger prompt
Before you start
What you’ll learn · Before you start
What you will learn
- Inspect running asyncio programs with python -m asyncio ps and pstree
- Display a task's call graph with asyncio.print_call_graph and capture it as objects
- Explain what attaching to another process needs on Linux, macOS and Windows
- Diagnose a stuck program from its await tree, including an await cycle
- Debug coroutines with pdb.set_trace_async and await at the debugger prompt
Before you start
On this page
When an asyncio program hangs, pressing Ctrl-C cancels the main task and shows where that one task waited, not which of the other tasks, one or a thousand, waits for what. Python 3.14 adds tools that show the await graph: The graph of which asyncio task waits for which: each task points to the tasks awaiting it. Python 3.14's call-graph functions and python -m asyncio ps and pstree show it, with the line where each task is suspended.: which task is suspended, on which line, and which task is waiting for it.
From outside a running process, python -m asyncio ps PID lists every pending task in a table and
python -m asyncio pstree PID draws the same information as a tree; both read the process’s memory without running
any code in it. From inside a program, asyncio.print_call_graph() prints the graph of a task,
asyncio.format_call_graph() returns it as text and asyncio.capture_call_graph() returns it as objects you can walk.
And for stepping through coroutines, await pdb.set_trace_async() opens the debugger: A program that runs your program under its control, so that you can stop it at a line, run it one line at a time and look at its variables. Python's own debugger, pdb, is in the standard library. More in “Debugging with print, breakpoint() and pdb” with
await allowed at its prompt. Together they turn “it is stuck” into “this task waits on this line for that one”.
A task’s await graph names everyone waiting for it
An order checkout runs two price lookups in a TaskGroup, inside another TaskGroup in main(). The lookup for tea
prints the graph of its own task, then walks the same graph as objects:
import asyncio
import re
def tidy(text):
"""Task ids are memory addresses and paths depend on the computer: shorten both."""
text = re.sub(r"id=0x[0-9a-f]+", "id=...", text)
return re.sub(r"File '(?:[^']*/)?([^'/]+\.py)'", r"File '\1'", text)
async def price(item):
await asyncio.sleep(0.01)
if item == "tea":
print(tidy(asyncio.format_call_graph())) # the graph of the task running this line
graph = asyncio.capture_call_graph() # the same graph as objects
chain = []
while graph:
chain.append(graph.future.get_name())
graph = graph.awaited_by[0] if graph.awaited_by else None
print("await chain:", " <- ".join(chain))
return 30
async def checkout(items):
async with asyncio.TaskGroup() as tg:
for item in items:
tg.create_task(price(item), name=f"price:{item}")
async def main():
asyncio.current_task().set_name("main") # a name instead of the default Task-1
async with asyncio.TaskGroup() as tg:
tg.create_task(checkout(["tea", "rice"]), name="checkout")
asyncio.run(main())Output
* Task(name='price:tea', id=...)
+ Call stack:
| File 'call_graph.py', line 14, in async price()
+ Awaited by:
* Task(name='checkout', id=...)
+ Call stack:
| File 'taskgroups.py', line 121, in async TaskGroup._aexit()
| File 'taskgroups.py', line 72, in async TaskGroup.__aexit__()
| File 'call_graph.py', line 25, in async checkout()
+ Awaited by:
* Task(name='main', id=...)
+ Call stack:
| File 'taskgroups.py', line 121, in async TaskGroup._aexit()
| File 'taskgroups.py', line 72, in async TaskGroup.__aexit__()
| File 'call_graph.py', line 32, in async main()
await chain: price:tea <- checkout <- main
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 call_graph.py
Runs on this device, in your browser. The first run downloads Python (about 13.5 MB), which is kept for the next runs.
Your run, in this browser
Here is the graph it printed, read from the bottom up:
The await tree of call_graph.py, read from the bottom up
Text description of the diagram
The diagram is read from the bottom up, following the "awaited by" links that the call-graph tools print.
1. At the bottom are the two tasks that checkout's TaskGroup created: price:tea, running price() at line 14, where it prints the graph, and price:rice, asleep in price(). 2. Both are awaited by the checkout task, which runs checkout() and waits for them in TaskGroup.__aexit__. 3. The checkout task is awaited by the main task, the task that asyncio.run() created and that named itself "main", which runs main() and waits in its own TaskGroup.__aexit__.
The graph starts at the current task, price:tea, running line 14, and follows the “awaited by” links upwards.
The checkout task waits for it inside its TaskGroup, and main, the task that asyncio.run() created and that
renamed itself from the default Task-1, waits for checkout. Each task shows its own call stack, from the innermost
frame down to the task’s coroutine, so TaskGroup._aexit and TaskGroup.__aexit__ appear above the lines of your
code that entered the group. The example masks task ids, which are memory addresses, and shortens file paths, so the
output reads the same everywhere.
capture_call_graph() returned the same graph as data, and three lines of code turned it into a one-line chain,
price:tea <- checkout <- main, of the kind worth putting in a log message. Naming tasks, as name="checkout" does
here, is what makes such output readable. Step through those three lines, one link at a time:
The drawing shows the whole scene; its 6 steps are listed below.
Every step as text
- The task that asks. price("tea") reaches line 14 in the task price:tea and prints format_call_graph(), the graph of the task that runs the line, starting with its own call stack.
- The same graph as objects. Line 15 asks capture_call_graph() for the same graph as objects. Its future is price:tea, its call_stack holds price() at line 14, and its awaited_by lists the graphs of the tasks waiting for it.
- One link up. The loop appends the task's name and moves to awaited_by[0], the graph of checkout, which waits for its two price tasks in TaskGroup.__aexit__.
- Another link up. checkout is awaited by main, the task that asyncio.run() created and that renamed itself from Task-1 with set_name().
- The top of the graph. The awaited_by list of main is empty, so graph becomes None and the loop ends. The program prints await chain, price:tea <- checkout <- main.
- A sibling is not a link. price:rice waits in price() all this time, but tasks of one TaskGroup do not await each other, so no link leads from price:tea to it and the output never mentions it.
Checkpoint
Check: A task's await graph
Outside tools need permission, inside tools need a line of code
Five tools show the same graph from two places:
| Tool | Runs | Shows | Needs |
|---|---|---|---|
python -m asyncio pstree PID |
Outside the process | Every task, nested under the task that awaits it | Permission to read the process’s memory |
python -m asyncio ps PID |
Outside the process | One row per task: name, coroutine stack, awaiter | The same permission |
asyncio.print_call_graph() |
Inside the program | The graph of one task, printed | A line of code in the program |
asyncio.capture_call_graph() |
Inside the program | The same graph as objects | A line of code in the program |
await pdb.set_trace_async() |
Inside the program | The debugger, with await allowed |
A terminal attached to the program |
The outside tools suit a production process that you cannot restart: they inspect without stopping it. The inside tools suit code that can watch itself, such as a timeout handler that logs the graph before it gives up.
Checkpoint
Check: Which tool, when
Attaching from outside needs permission
Reading another process’s memory is something every operating system guards. On Linux you can usually inspect your
own processes, unless the kernel’s
The Linux system call through which one process can read and change another process's memory, as debuggers do. The kernel's own rules decide which process may use it on which.
restrictions or a container’s system-call filter forbid it. macOS forbids it by default: the Python documentation
describes giving the Python binary an ad-hoc signature with a debugging entitlement, or running the tool with sudo.
Windows needs an administrator prompt. Inside containers, current Docker and containerd versions allow the needed
system call on recent Linux kernels, while older ones need it allowed by hand, which weakens the container’s isolation.
Step through one inspection and the check each system makes:
The drawing shows the whole scene; its 6 steps are listed below.
Every step as text
- A request to read memory. The command runs in a process of its own. To list the target's tasks it must read the target's memory, and only the operating system can allow that.
- Linux. On Linux you can usually inspect your own processes, unless the kernel's ptrace restrictions or a container's system-call filter forbid it.
- macOS. macOS forbids it by default. The Python documentation describes giving the Python binary an ad-hoc signature with a debugging entitlement, or running the tool with sudo.
- Windows. Windows needs an administrator prompt. In every case the decision belongs to the operating system, not to Python.
- Reading, not running. Once allowed, the tool reads the target's tasks from its memory and prints the table. Nothing runs inside the target, so it carries on exactly as before.
- Switching the interface off. A process can switch its side of the remote debugging interface off with PYTHON_DISABLE_REMOTE_DEBUG=1 or -X disable-remote-debug. That stops tools that make it run code, such as pdb -p, but not ps, which only reads its memory; the operating system's rules stay in force.
A process can switch its side of the remote debugging interface: The interface PEP 768 added in Python 3.14: a tool that may already read and write a process's memory asks the interpreter to run a Python file at a safe point. python -m pdb -p uses it, and PYTHON_DISABLE_REMOTE_DEBUG=1 switches it off. off with
PYTHON_DISABLE_REMOTE_DEBUG=1 or -X disable-remote-debug, a sensible default for hardened deployments that never
attach a debugger. That interface is what lets python -m pdb -p PID attach the debugger to a running program, by
making the program run code; ps and pstree run no code in it, so the switch does not stop them, and the operating
system’s own debugging rules stay in force either way. Two failures need more than one look at the graph: a queue that
nobody drains, and two tasks that await each other, which pstree reports as a cycle.
Checkpoint
Check: Permission to attach from outside
Check yourself
One question at a time on the await graph, the five tools and attaching, the idea you know least first.
Practice that adapts to you
Practise this lesson
8 questions about 10 ideas of this lesson. Each answer is explained at once, and what you get right is kept in this browser.
Reading ps and pstree on stuck programs, with two failures reproduced Pro: opens with a Pro, Premium or Ultimate pass
The commands are only as useful as your reading of their output. This part runs two stuck programs, shows what ps
and pstree print for each, and turns the readings into a checklist.
In this part:
- A queue that nobody drains: the missing task is the clue
- Two tasks that await each other: pstree names the cycle
- Reading a table of a thousand tasks
- From what you see to the cause: a checklist with the trade-offs
2 diagrams · 3 runnable examples (Python, lost_consumer.py, selfinspect.py) · about 11 minutes
Opening this part with your pass…
This part opens with any MySmartCoPilot pass.
- Pro pass · 1 week₹79$2.99
- 30 days₹179$5.99
- 1 year₹1,699$45
One payment; a pass never renews. The price shown is the total you pay: no fees or charges are added at checkout.
Passes can’t be bought right now, so this part can’t be opened at the moment. The free part above stands on its own. If you have a pass, enter your licence key.
It also opens every other in-depth part in Learn and every Pro tool. Sales are final, except where the law gives you a right to withdraw, for example in the EU and the UK (refund policy).
A stuck-task watchdog and await at the debugger prompt, with an exercise Pro: opens with a Pro, Premium or Ultimate pass
Permission to attach is not always available, so a service should be able to report its own await tree. This part
builds a watchdog from the documented capture_call_graph(), steps through a coroutine with set_trace_async() and
ends with an exercise.
In this part:
- A watchdog that logs the await tree when progress stops
- What building the tree costs, measured
- An await at the debugger prompt, recorded
- Exercise: unstick a packing pipeline
- Unstick a packing pipeline that hangs
2 diagrams · 3 runnable examples (Python, awaittree.py, watchdog.py) · 1 exercise with sample tests · about 10 minutes
Opening this part with your pass…
This part opens with any MySmartCoPilot pass.
- Pro pass · 1 week₹79$2.99
- 30 days₹179$5.99
- 1 year₹1,699$45
One payment; a pass never renews. The price shown is the total you pay: no fees or charges are added at checkout.
Passes can’t be bought right now, so this part can’t be opened at the moment. The free part above stands on its own. If you have a pass, enter your licence key.
It also opens every other in-depth part in Learn and every Pro tool. Sales are final, except where the law gives you a right to withdraw, for example in the EU and the UK (refund policy).
Interview questions
-
Mid to senior · Warm-up
What do python -m asyncio ps and python -m asyncio pstree show, and what do they need in order to work?
Both commands, new in Python 3.14, inspect the asyncio tasks of another running Python process, given its process id, without running any code in it. ps prints a flat table with one row per pending task: the thread, the task's id and name, its coroutine stack, and the task that awaits it. pstree prints the same information as a tree, every task nested under the task that awaits it, with file names and line numbers, which shows at a glance where a stuck program waits; when tasks await each other it prints the cycle instead. Both need permission to read the other process's memory: usually available for your own processes on Linux, an entitlement on the Python binary or sudo on macOS, and administrator rights on Windows.
-
Mid to senior
A production asyncio service has stopped answering requests. CPU use is 0 %, nothing new is in the logs, and you may not restart it until you know why. Walk through how you find the cause with Python 3.14's tools.
-
Senior to staff
python -m asyncio pstree reports that the await graph of a stuck service contains cycles. What does the cycle mean, how can code create one, and how do you get the service out of it and keep it from happening again?
-
Senior to staff
Your security team asks whether Python 3.14's remote debugging interface from PEP 768 is a new risk for production containers, and what you would configure. Answer them.
Key takeaways
- A task’s await graph starts at the task and follows the “awaited by” links up to the task that waits for all of it.
python -m asyncio pstree PIDdraws every task under the task that awaits it;ps PIDlists the same as a table.- The commands read another process’s memory, which needs permission: Linux ptrace rules, an entitlement or
sudoon macOS, an administrator prompt on Windows. - Inside a program,
print_call_graph(),format_call_graph()andcapture_call_graph()show the graph of a task, andawait pdb.set_trace_async()opens the debugger withawaitallowed at its prompt.
Terms in this lesson
- await graph
- The graph of which asyncio task waits for which: each task points to the tasks awaiting it. Python 3.14's call-graph functions and python -m asyncio ps and pstree show it, with the line where each task is suspended.
- remote debugging interface
- The interface PEP 768 added in Python 3.14: a tool that may already read and write a process's memory asks the interpreter to run a Python file at a safe point. python -m pdb -p uses it, and PYTHON_DISABLE_REMOTE_DEBUG=1 switches it off.
References
- What's New In Python 3.14: asyncio introspection capabilities (Python Software Foundation)
- asyncio: Call graph introspection (Python Software Foundation)
- asyncio: Command-line introspection tools (Python Software Foundation)
- Remote debugging attachment protocol: permission requirements (Python Software Foundation)
- pdb: pdb.set_trace_async() (Python Software Foundation)
- Command line and environment: -X disable-remote-debug and PYTHON_DISABLE_REMOTE_DEBUG (Python Software Foundation)
- PEP 768: Safe external debugger interface for CPython (Python Software Foundation)
- asyncio: queues (Queue.join and task_done) (Python Software Foundation)
- asyncio Runners: handling keyboard interruption (Python Software Foundation)
- What's New In Python 3.14: pdb remote attaching with sys.remote_exec() (Python Software Foundation)
Related tools
Report a problem with this lesson
Kept only in this browser. Your Learn progress