Operating Systems Module 9 – I/O systems, disks and disk scheduling

Buffering, caching and zero-copy I/O

Where the kernel buffers, caches and spools data, why the buffer size sets the cost of a copy, measured, and how sendfile and splice skip copies.

Intermediate
20 minutes
Examples run with Python 3.14.8, Pyodide 314.0.7, Clang 21.0.0 and GCC 16.2.0
By MySmartCoPilot

Free: Buffers, caches and spools hold data on its way · The buffer size decides how many system calls a copy makes · Zero-copy: the kernel moves the data, the program never touches it · Practise: predict what a copy costs. With a pass: 3 in-depth parts and 3 interview answers, about 13 minutes.

What you will learn

Explain buffering, caching and spooling in the kernel's I/O subsystem
Measure how the buffer size changes the number of system calls and the speed of a copy
Compare blocking, non-blocking and asynchronous I/O by what a call does when data is not ready
Use sendfile and splice to move data between descriptors without copying it through user space

Before you start

I/O hardware: polling, interrupts and DMA

What you’ll learn · Before you start

What you will learn

Explain buffering, caching and spooling in the kernel's I/O subsystem
Measure how the buffer size changes the number of system calls and the speed of a copy
Compare blocking, non-blocking and asynchronous I/O by what a call does when data is not ready
Use sendfile and splice to move data between descriptors without copying it through user space

Before you start

I/O hardware: polling, interrupts and DMA

On this page

Copying a file with a one-byte buffer makes two system calls for every byte, 2,097,152 for each mebibyte; with a buffer of 1 MiB it makes two. On the recorded Mac the large buffer copied tens of thousands of times faster. Between a program and its disk the kernel keeps data in buffers and in a cache, and it can move data from one file to another without the program touching it.

This lesson shows where the kernel’s buffering and caching happen, measures what the size of a buffer costs, and explains how zero-copy I/O with sendfile and splice avoids copies.

Buffers, caches and spools hold data on its way

A line a program prints does not go straight to the disk. The C library collects output in a buffer of its own, and POSIX says standard output is fully buffered unless it refers to an interactive device. When the buffer is written, write copies it into the kernel and returns before the disk has seen anything. Follow both directions:

Where data waits between a program and the drive

Text description of the diagram

The figure has eight boxes in two columns: the write path runs down the left, the read path up the right.

Top left: printf and fwrite put output in the C library's buffer, inside the program.
When that buffer is full (or at the end of a line, for a terminal), the library makes one write system call, which copies the buffer into the page cache.
In the page cache the written page is marked dirty. The call has returned, but the drive has nothing yet.
The dirty page is written back later by the kernel's flusher threads, or when the program calls fsync, and reaches the drive by DMA.
On the right, from the bottom: a read that misses the page cache makes the drive fill a page of the cache by DMA.
Once the page is in the cache, the read is a hit: the data is copied from memory into the program's buffer, and later reads of the same data never touch the drive.

Three ideas share the work, and the kernel uses all of them:

buffer: Memory that holds data once on its way between a producer and a consumer that work at different speeds or in different sizes, such as the C library's output buffer. holds data between a producer and a consumer that work at different speeds or in different sizes. The C library’s buffering turns many small prints into one write.
Caching keeps a copy so that a later access can skip the slow device. The kernel’s caching of files happens in the page cache: in its own words, “whenever a file is read, the data is put into the page cache”; a write lands there too, as a dirty page: A page in memory that has been written since it was read from the drive, so the drive's copy is out of date until the page is written back., and reaches the drive later.
A spooling: Queuing whole jobs on disk for a device that can serve only one job at a time, as a print scheduler keeps print jobs in its spool directory. queues whole jobs on disk for a device that serves one at a time. The CUPS print scheduler keeps the files of each waiting job in its spool directory, usually /var/spool/cups.

The difference between buffering and caching is what happens to the data afterwards: a buffer’s data is used once, on its way through, and a cache’s data is kept because it may be wanted again. One page cache: The kernel's cache of file data in memory: pages read from a file stay there for later reads, and written pages wait there before they reach the drive. page can be both, since a dirty page buffers a write and meanwhile answers reads of the same data.

Checkpoint

Check: Buffering, caching and spooling

Question 1 of 2 Which of these is a cache rather than a buffer?
Choose one answer.

A pipe holding bytes that its reader has not taken yet The C library collecting printf output until it has a full buffer A print job waiting in /var/spool/cups The page cache keeping a file's pages after a read, so that later reads skip the drive

Show the answer: Question 1 of 2

Answer: The page cache keeping a file's pages after a read, so that later reads skip the drive

A buffer holds data on its way between two parts that work at different speeds or sizes; a cache keeps a copy for the next access. A dirty page in the page cache is both at once.
Question 2 of 2 A program prints 100 lines of 30 bytes each to a file, and the C library's buffer holds 4,096 bytes. When does the kernel first see any of the lines?
Choose one answer.

When the disk is ready to take them As soon as each printf returns, because printf is a system call Only when the program calls fflush or exits, because 3,000 bytes never fill the buffer After every line, when printf sees the newline

Show the answer: Question 2 of 2

Answer: Only when the program calls fflush or exits, because 3,000 bytes never fill the buffer

100 × 30 = 3,000 bytes fit in one 4,096-byte buffer, so the library makes a single write when the stream is flushed or closed. POSIX says standard output is fully buffered unless it refers to an interactive device.

The buffer size decides how many system calls a copy makes

A copy with read and write makes two system calls per buffer, so the buffer’s size sets the number of calls. The program copies a file with buffers from 1 byte to 1 MiB, giving each size enough data to time. It uses Python’s os.read and os.write, low-level functions that make one system call each:

One file copied with six buffer sizes Python · copy_sizes.py

# Copies a file with os.read and os.write, which make one system call each,
# one buffer size at a time, and reports the calls per mebibyte and the speed.
import os
import time

MIB = 1024 * 1024
TOTAL = 16 * MIB
SIZES = [1, 16, 256, 4096, 65536, MIB]


def copy(src: int, dst: int, size: int, nbytes: int) -> int:
    """Copies nbytes from src to dst in pieces of size; returns the calls made."""
    calls = 0
    os.lseek(src, 0, os.SEEK_SET)
    os.lseek(dst, 0, os.SEEK_SET)
    done = 0
    while done < nbytes:
        piece = os.read(src, size)
        os.write(dst, piece)
        calls += 2
        done += len(piece)
    return calls


def label(size: int) -> str:
    return f"{size // 1024:4} KiB" if size >= 1024 else f"{size:6} B"


src = os.open("source.bin", os.O_RDWR | os.O_CREAT | os.O_TRUNC, 0o600)
dst = os.open("copy.bin", os.O_WRONLY | os.O_CREAT | os.O_TRUNC, 0o600)
block = bytes(i % 26 + 97 for i in range(256)) * (MIB // 256)
for _ in range(TOTAL // MIB):
    os.write(src, block)
copy(src, dst, MIB, TOTAL)  # untimed: gives the copy its pages first

print("buffer   calls per MiB       MB/s")
speeds = []
for size in SIZES:
    # Small buffers copy less data, so that no size takes much over a second.
    nbytes = {1: 64 * 1024, 16: MIB}.get(size, TOTAL)
    start = time.perf_counter()
    calls = copy(src, dst, size, nbytes)
    speed = nbytes / 1e6 / (time.perf_counter() - start)
    speeds.append(speed)
    print(f"{label(size)} {calls * MIB // nbytes:15} {speed:10.2f}")
print(f"the fastest copy ran {max(speeds) / speeds[0]:.0f} times as fast as the 1-byte copy")
os.close(src)
os.close(dst)
os.unlink("source.bin")
os.unlink("copy.bin")

Output

buffer   calls per MiB       MB/s
     1 B         2097152       0.17
    16 B          131072       2.98
   256 B            8192      53.76
   4 KiB             512     620.56
  64 KiB              32    2573.42
1024 KiB               2    5030.02
the fastest copy ran 29582 times as fast as the 1-byte copy

This output changes from run to run: the speeds depend on the machine and on what else it is running; the counts of system calls do not

Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 copy_sizes.py

The calls fall exactly as the buffer grows, from 2,097,152 per mebibyte to 2, and the speed rises as they fall, from well under 1 MB a second to several thousand. With tiny buffers almost all the time is the fixed cost of each call, however few bytes it moves. With large ones copying the bytes dominates: from 64 KiB to 1 MiB the calls fell sixteenfold, the speed far less. This is why the C library buffers: a thousand short printf calls become a few writes.

A call can also differ in how it waits when no data is ready. A blocking call sleeps until it can do its work. A non-blocking call returns at once with the error EAGAIN. An asynchronous call starts the work and reports its completion afterwards.

Key idea

Every read or write pays a fixed cost to enter the kernel, so the buffer size decides how often a copy pays it: two calls per byte with a one-byte buffer, two per mebibyte with a 1 MiB one.

Checkpoint

Check: System calls: the buffer size and how a call waits

Question 1 of 2 How many system calls does copying 1 MiB with read and write and a 4 KiB buffer make?
Type a number.
Your answer calls

A hint

Count the buffer-fulls, then remember that each one takes two calls.

Show the answer: Question 1 of 2

Answer: 512 calls

1 MiB / 4 KiB = 256 buffer-fulls, each one read and one write: 512 calls, the number the copy program printed for 4 KiB.
Question 2 of 2 From the 64 KiB buffer to the 1 MiB buffer the calls fall from 32 to 2 per mebibyte, yet the speed grows far less than sixteenfold. Why?
Choose one answer.

Most of the time now goes on copying the bytes themselves, which no buffer size removes The kernel caps every read at 64 KiB The drive cannot deliver data any faster A 1 MiB buffer is slower to allocate

Show the answer: Question 2 of 2

Answer: Most of the time now goes on copying the bytes themselves, which no buffer size removes

Time per mebibyte is the calls times the cost of a call plus the bytes times the cost of copying them. Large buffers shrink the first term until only the second is left.

Zero-copy: the kernel moves the data, the program never touches it

A file server that answers with read and write copies every block twice through its own memory: read copies it from the page cache into the program’s buffer, and write copies it back into the kernel, into a socket’s buffer. The program never looks at the bytes, so both copies are waste. zero-copy: Moving data from one descriptor to another inside the kernel, as sendfile and splice do, so that the bytes never pass through the program's memory. I/O leaves the data in the kernel instead. Step through one block both ways:

Interactive · message by message One block sent to a socket, with read and write, then with sendfile Step through one block of a file on its way to a socket. Count the copies the CPU makes and the system calls.

The drawing shows the whole scene; its 5 steps are listed below.

Every step as text

The block reaches memory. The program calls read. The block is not in the page cache yet, so the drive puts it there by DMA; so far the CPU has copied nothing.
Copy 1: into the program. read then copies the block from the page cache into the program's buffer. This is the first copy the CPU makes, across the line between kernel memory and the program's memory.
Copy 2: back into the kernel. To send the block, the program calls write on the socket, and the CPU copies the same bytes back into the kernel, into the socket's buffer. Two system calls and two copies, and the program never looked at the data.
sendfile instead. With sendfile the program makes one call that names the file and the socket. The kernel moves the block from the page cache to the socket itself; the copying, the manual page says, is done within the kernel.
What it saved. The block never entered the program's memory, so both copies through user space are gone, and one call covers what took two. splice does the same between any two descriptors, as long as one of them is a pipe.

Two calls give zero-copy I/O on Linux. sendfile copies between two descriptors inside the kernel, which its manual page calls “more efficient than the combination of read(2) and write(2)”. splice moves data between two descriptors without copying it between kernel and user space, provided that one of them is a pipe. Sending a file to a socket with splice therefore takes two calls, into a pipe and out of it.

Checkpoint

Check: What zero-copy saves

One question A server sends a file that is already in the page cache to a socket. How often does the CPU copy each byte through the program's memory with read and write, and how often with sendfile?
Choose one answer.

Not at all with either, because the file is cached Once each way Twice with both, because the data has to reach the socket Twice with read and write, not at all with sendfile

Show the answer: One question

Answer: Twice with read and write, not at all with sendfile

Zero-copy calls remove the trip through user space. The kernel itself may still move the data from the page cache into the socket's buffer, but the program's memory never holds it.

Practise: predict what a copy costs

Turn the measurements into a model that predicts the calls and the time of a copy. The sample tests include a one-byte buffer and a file that does not fill its last buffer:

Exercise · Easy · Python

Predict the system calls and the time of a copy

A program copies a file with read and write through a buffer of its own: each read fills the buffer (the last one may fill it only partly) and each write writes what the read returned. Write predict(file_bytes, buffer_bytes, call_us, byte_ns) in copy_cost.py. It returns a pair (calls, seconds):

calls is the number of system calls: one read and one write for each buffer-full, counting a last, partly filled buffer as a full one. A file of 0 bytes needs none.
seconds is the predicted time: every call costs call_us microseconds, and every byte is copied twice, into the program's buffer and back out, at byte_ns nanoseconds a copy.

For example, predict(1_048_576, 1_048_576, 2.0, 0.1) copies 1 MiB in one buffer: 2 calls, which cost 4 µs, and 2 × 1,048,576 copied bytes, which cost about 210 µs, so it returns (2, 0.0002137152).

Starter code · copy_cost.py

def predict(file_bytes, buffer_bytes, call_us, byte_ns):
    """(system calls, seconds) for copying file_bytes with a buffer of buffer_bytes."""
    # Replace this line with your code.
    return 0, 0.0

The sample tests · test_copy_cost.py

import math

from copy_cost import predict


def test_one_buffer_for_the_whole_file():
    """makes one read and one write, and spends most of the time copying"""
    calls, seconds = predict(1_048_576, 1_048_576, 2.0, 0.1)
    assert calls == 2
    assert math.isclose(seconds, 0.0002137152)


def test_a_one_byte_buffer():
    """makes two calls for every byte, so the calls cost far more than the copying"""
    calls, seconds = predict(1_048_576, 1, 2.0, 0.1)
    assert calls == 2_097_152
    assert math.isclose(seconds, 4.1945137152)


def test_a_last_buffer_that_is_not_full():
    """counts a partly filled last buffer as one more read and write"""
    calls, seconds = predict(10_000, 4096, 1.0, 0.0)
    assert calls == 6
    assert math.isclose(seconds, 6e-06)
    calls, seconds = predict(8192, 4096, 1.0, 0.0)
    assert calls == 4
    assert math.isclose(seconds, 4e-06)


def test_an_empty_file():
    """needs no system call and no time"""
    calls, seconds = predict(0, 4096, 5.0, 1.0)
    assert calls == 0
    assert seconds == 0


def test_the_copies_cost_the_same_whatever_the_buffer():
    """charges every byte twice, so with free calls the time does not depend on the buffer"""
    small = predict(1_000_000, 100, 0.0, 0.5)[1]
    large = predict(1_000_000, 1_000_000, 0.0, 0.5)[1]
    assert math.isclose(small, 0.001)
    assert math.isclose(large, 0.001)

A hint

The number of buffers is math.ceil(file_bytes / buffer_bytes), or with whole numbers only (file_bytes + buffer_bytes - 1) // buffer_bytes; each one takes a read and a write. For the time, add calls * call_us / 1e6 and 2 * file_bytes * byte_ns / 1e9.

The sample tests run on this device, in your browser (Pyodide): nothing is sent to mysmartcopilot.com. The first run downloads Python (about 13.5 MB), which is kept for the next runs. A check in your browser is feedback for you, not proof that the code is right for every input.

Then answer the questions below, weakest idea first.

The page cache, modelled and measured Pro: opens with a Pro, Premium or Ultimate pass

Reads that find their pages in memory skip the drive, and writes return before the drive has them. This part builds a page cache with write-back in code, measures cached reads and the price of making a write durable on the recorded Mac, and shows the calls that keep data out of the cache.

In this part:

A page cache in code: hits, dirty pages and write-back
Cached reads and the price of fsync, measured

2 runnable examples (Python) · about 10 minutes

Opening this part with your pass…

This part opens with any MySmartCoPilot pass.

Pro pass · 1 week₹79$2.99
30 days₹179$5.99
1 year₹1,699$45

One payment; a pass never renews. The price shown is the total you pay: no fees or charges are added at checkout.

Choose a pass I have a licence key

It also opens every other in-depth part in Learn and every Pro tool. Sales are final, except where the law gives you a right to withdraw, for example in the EU and the UK (refund policy).

Blocking, non-blocking and asynchronous calls compared Pro: opens with a Pro, Premium or Ultimate pass

A call that cannot be served at once has three choices: wait, refuse, or accept the job and report later. This part runs the refusal on a real pipe, compares the three, and shows why the asynchronous interfaces POSIX defined gave way to newer ones.

In this part:

What each call does when the data is not ready
Asynchronous I/O: POSIX AIO, then a ring shared with the kernel

1 runnable example (C) · about 5 minutes

Opening this part with your pass…Pro part. It opens with the same pass as the part above.

sendfile measured against read and write, and splice Pro: opens with a Pro, Premium or Ultimate pass

Saving two copies sounds small until a server sends files all day. This part measures sendfile against read and write on the recorded Mac, counts what each saves, and shows how splice gets the same result through a pipe on Linux.

In this part:

sendfile against read and write, measured
splice: zero-copy through a pipe

1 diagram · 1 runnable example (Python) · about 6 minutes

Opening this part with your pass…Pro part. It opens with the same pass as the part above.

Interview questions

Fresher · Warm-up

What is the difference between buffering and caching in an operating system's I/O?

A buffer holds data once, on its way between two parts that work at different speeds or in different sizes; a cache keeps a copy of data because it may be needed again. The C library's output buffer is a buffer: it collects small prints and hands them to the kernel in one write, and the data is gone from it afterwards. The kernel's page cache is a cache: pages read from a file stay in memory, so a second read never touches the drive. The two often meet in one place, since a dirty page in the page cache buffers a write on its way to the drive and answers reads of the same data meanwhile. A third idea, spooling, queues whole jobs on disk for a device that serves one at a time, as a print scheduler does.
Fresher to mid

A backup tool copies files with read and write 512 bytes at a time, and a 10 GB file takes far longer than the drive's speed allows. A colleague proposes a 1 MiB buffer. Predict the effect, explain where the time goes, and say when a larger buffer stops helping.

Model answer, Pro. Model answers open with any pass; prices are in the notice on this page. First step and follow-ups
Mid to senior

A service appends each order to a log file and replies "saved" as soon as write returns. After a power cut, the last few seconds of orders are missing. Explain why, and how you would make every order durable without making each request wait a full trip to the drive.

Model answer, Pro. First step and follow-ups
Mid to senior

A file server sends large static files with read and write, and runs out of CPU long before its network link is full. How would you cut the CPU it spends per byte, what exactly does sendfile save, and what can stop zero-copy from applying?

Model answer, Pro. First step and follow-ups

Key takeaways

A buffer holds data once on its way between two parts that work at different speeds or sizes, a cache keeps a copy for the next access, and a spool queues whole jobs for a device that serves one at a time.
Every read or write pays a fixed cost, so the buffer size sets the cost of a copy: 2,097,152 calls per mebibyte with a one-byte buffer against 2 with a 1 MiB buffer, and tens of thousands of times the speed.
Zero-copy calls such as sendfile and splice move data between descriptors inside the kernel, so the program’s memory never holds a copy of it.

Terms in this lesson

buffer: Memory that holds data once on its way between a producer and a consumer that work at different speeds or in different sizes, such as the C library's output buffer.
dirty page: A page in memory that has been written since it was read from the drive, so the drive's copy is out of date until the page is written back.
page cache: The kernel's cache of file data in memory: pages read from a file stay there for later reads, and written pages wait there before they reach the drive.
spooling: Queuing whole jobs on disk for a device that can serve only one job at a time, as a print scheduler keeps print jobs in its spool directory.
zero-copy: Moving data from one descriptor to another inside the kernel, as sendfile and splice do, so that the bytes never pass through the program's memory.

References

Memory management concepts overview (page cache) (The Linux Kernel Organization)
Documentation for /proc/sys/vm/ (dirty data and writeback) (The Linux Kernel Organization)
stdin, stdout, stderr: standard I/O streams (POSIX.1-2024) (The Open Group and IEEE)
open(2): open and possibly create a file (O_DIRECT, O_NONBLOCK, O_SYNC) (Linux man-pages project (man7.org))
fsync(2): synchronize a file with its storage device (Linux man-pages project (man7.org))
readv(2): RWF_DONTCACHE (Linux man-pages project (man7.org))
sendfile(2): transfer data between file descriptors (Linux man-pages project (man7.org))
splice(2): splice data to or from a pipe (Linux man-pages project (man7.org))
os: file descriptor operations (Python 3.14) (Python Software Foundation)
aio(7): POSIX asynchronous I/O overview (Linux man-pages project (man7.org))
aio_read: asynchronous read from a file (POSIX.1-2024) (The Open Group and IEEE)
io_uring(7): asynchronous I/O facility (shared rings and batching) (Linux man-pages project (man7.org))
io_uring_setup(2): IORING_SETUP_SQPOLL, submission without system calls (Linux man-pages project (man7.org))
sendfile(2) (macOS manual page) (Apple)
fcntl(2) (macOS manual page: F_NOCACHE, F_FULLFSYNC) (Apple)
fsync(2) (macOS manual page: the drive cache and F_FULLFSYNC) (Apple)
Kernel TLS (sendfile over TLS records, TLS_TX_ZEROCOPY_RO) (The Linux Kernel Organization)
CUPS design description (the scheduler and its spool directory) (OpenPrinting)
Linux 6.14 (uncached buffered I/O) (KernelNewbies)
Computer Science Curricula 2023: Operating Systems knowledge area (Version Gamma) (ACM, IEEE Computer Society and AAAI (CS2023 Task Force))

Report a problem with this lesson

In this module: I/O systems, disks and disk scheduling

I/O hardware: polling, interrupts and DMA Free: 20 minutes
Buffering, caching and zero-copy I/O Free core, Pro depth: 7 of 20 minutes free
Many connections: epoll, kqueue, io_uring Free core, Pro depth: 7 of 20 minutes free
Disks, SSDs and disk-scheduling algorithms Free core, Pro depth: 7 of 20 minutes free
RAID, checksums and data integrity Free core, Pro depth: 7 of 20 minutes free

Your country

Buffering, caching and zero-copy I/O

What you will learn

Before you start

What you will learn

Before you start

Buffers, caches and spools hold data on its way

The buffer size decides how many system calls a copy makes

Zero-copy: the kernel moves the data, the program never touches it

Practise: predict what a copy costs

Predict the system calls and the time of a copy

Results of the sample tests

The page cache, modelled and measured Pro: opens with a Pro, Premium or Ultimate pass

Blocking, non-blocking and asynchronous calls compared Pro: opens with a Pro, Premium or Ultimate pass

sendfile measured against read and write, and splice Pro: opens with a Pro, Premium or Ultimate pass

Interview questions

What is the difference between buffering and caching in an operating system's I/O?

A backup tool copies files with read and write 512 bytes at a time, and a 10 GB file takes far longer than the drive's speed allows. A colleague proposes a 1 MiB buffer. Predict the effect, explain where the time goes, and say when a larger buffer stops helping.

A service appends each order to a log file and replies "saved" as soon as write returns. After a power cut, the last few seconds of orders are missing. Explain why, and how you would make every order durable without making each request wait a full trip to the drive.

A file server sends large static files with read and write, and runs out of CPU long before its network link is full. How would you cut the CPU it spends per byte, what exactly does sendfile save, and what can stop zero-copy from applying?

Key takeaways

References