Protocols, layers and encapsulation

A protocol is an agreement between two programs about the messages they exchange: what each message looks like, in which order the messages go, and what each side does when a message arrives or fails to arrive. Networks use many protocols, and each one does a single job. They are stacked in layers: a protocol uses only the service of the layer below it and offers its own service to the layer above. On the way down, each layer wraps the data it is given in a header of its own, and that wrapping is called encapsulation; the receiver unwraps the layers in the opposite order.

This lesson compares the two models people use to describe the layers, builds one real Ethernet frame byte by byte, and works out how much of what crosses the wire is your data.

What a protocol defines

Every protocol pins down three things:

• The format of each message: which fields it has, how long they are and what their values mean. An IPv4 header, for example, starts with a 4-bit version field that is always 4.
• The order of messages: TCP’s SYN must come before any data, and a DNS answer only makes sense after a question.
• The actions: what to do when a message arrives, when it is malformed, and when nothing arrives before a timer runs out.

Two programs that speak the same protocol are called peers. Peers seem to talk directly, browser to web server, but every message actually travels down the sender’s stack, across the links between them and up the receiver’s stack.

Two models of the layers

Two models describe the same idea with different numbers of layers. The Open Systems Interconnection (OSI) reference model has seven (ITU-T X.200). The Internet’s own model, written down in RFC 1122, has four: link, internet, transport and application. RFC 1122 notes that its application layer essentially combines OSI’s presentation and application layers.

Each OSI layer has one job:

• Application (7) offers services to programs, such as fetching a page or sending mail.
• Presentation (6) decides how data is represented: character encodings, compression and encryption. On the Internet these live inside programs and libraries; TLS’s encryption is the best-known example.
• Session (5) manages dialogues between programs, such as a login session or a resumed TLS session; on the Internet, again, inside programs and libraries.
• Transport (4) delivers data between programs on two hosts.
• Network (3) addresses and forwards packets between networks.
• Data link (2) moves frames between neighbours on one link.
• Physical (1) sends bits as signals on copper, fibre or radio.

In the Internet model, each layer has protocols you will meet throughout this track, and its own name for the unit of data:

• Application: HTTP, DNS and SMTP, with TLS between TCP and HTTP. The unit is a message.
• Transport: TCP, UDP and QUIC. A segment for TCP, a datagram for UDP.
• Internet: IPv4, IPv6 and ICMP. A packet, also called a datagram.
• Link: Ethernet and Wi-Fi. A frame.

Use the OSI numbers as a vocabulary: people say “a layer 2 switch” or “a layer 7 load balancer”. Use the Internet model to understand real protocols, which were designed with it in mind. You will also meet a five-layer version that splits the link layer into data link and physical; it is the same idea.

Layering buys independence. Your laptop can switch from Wi-Fi to Ethernet without TCP noticing, and a site can move from HTTP/1.1 to HTTP/2 without any router changing, because each layer depends only on the service below it, not on how that service works.

Encapsulation, step by step

Here is what happens to a 1,000-byte HTTP response on its way down the server’s stack:

Each header carries what its layer needs. TCP’s header holds the two port numbers, the sequence and acknowledgement numbers and a checksum (RFC 9293). IPv4’s holds the source and destination addresses, a time to live that each router lowers by one, and a number that says which protocol is inside (RFC 791). Ethernet’s holds the two MAC addresses and an EtherType that says which network protocol is inside, and its trailer, the frame check sequence (FCS), lets the receiver detect damaged frames (IEEE 802.3).

The program below builds that frame for real, with every checksum computed. It uses addresses reserved for documentation (RFC 5737, RFC 9542), so the frame describes no real machine:

"""Builds one Ethernet frame by hand: an HTTP response inside TCP, inside IPv4, inside Ethernet.

The addresses are reserved for documentation (MAC 00:00:5e:00:53:xx, IPv4 192.0.2.0/24 and 198.51.100.0/24),
so this frame describes no real machine.
"""
import ipaddress
import struct
import zlib


def internet_checksum(data):
    """The checksum of IPv4 and TCP: the ones' complement of the ones' complement sum of 16-bit words."""
    if len(data) % 2:
        data += b"\x00"
    total = sum(struct.unpack(f"!{len(data) // 2}H", data))
    while total > 0xFFFF:
        total = (total & 0xFFFF) + (total >> 16)
    return ~total & 0xFFFF


def tcp_segment(src_ip, dst_ip, payload):
    """A 20-byte TCP header (no options) plus the payload, with its checksum filled in."""
    header = struct.pack("!HHIIHHHH", 443, 51514, 1000, 2000, (5 << 12) | 0x18, 65535, 0, 0)  # flags ACK+PSH
    pseudo = src_ip.packed + dst_ip.packed + struct.pack("!BBH", 0, 6, len(header) + len(payload))
    check = internet_checksum(pseudo + header + payload)
    return header[:16] + struct.pack("!H", check) + header[18:] + payload


def ipv4_packet(src_ip, dst_ip, segment):
    """A 20-byte IPv4 header (no options) plus the segment, with its header checksum filled in."""
    header = struct.pack("!BBHHHBBH4s4s", 0x45, 0, 20 + len(segment), 0x1C46, 0x4000, 64, 6, 0, src_ip.packed, dst_ip.packed)
    return header[:10] + struct.pack("!H", internet_checksum(header)) + header[12:] + segment


def hex_bytes(data):
    return " ".join(f"{b:02x}" for b in data)


body = b"<p>" + b"x" * 928 + b"</p>"
payload = b"HTTP/1.1 200 OK\r\nContent-Type: text/html\r\nContent-Length: %d\r\n\r\n" % len(body) + body
server, client = ipaddress.ip_address("198.51.100.20"), ipaddress.ip_address("192.0.2.10")

segment = tcp_segment(server, client, payload)
packet = ipv4_packet(server, client, segment)
ethernet = bytes.fromhex("00005e005301" "00005e005302" "0800")  # destination MAC, source MAC, EtherType IPv4
frame = ethernet + packet
fcs = struct.pack("<I", zlib.crc32(frame))  # Ethernet's frame check sequence is a CRC-32, sent low byte first
frame += fcs

print(f"Ethernet header ({len(ethernet)} bytes): {hex_bytes(ethernet)}")
print(f"IPv4 header     ({len(packet) - len(segment)} bytes): {hex_bytes(packet[:20])}")
print(f"TCP header      ({len(segment) - len(payload)} bytes): {hex_bytes(segment[:20])}")
print(f"Payload       ({len(payload)} bytes): {payload.split(b'\r\n')[0].decode()} ...")
print(f"FCS              ({len(fcs)} bytes): {hex_bytes(fcs)}")
print(f"Frame: {len(frame):,} bytes; on the wire, with 8 bytes of preamble and 12 of gap: {len(frame) + 20:,}")
print(f"Payload share: {len(payload) / len(frame):.1%} of the frame, {len(payload) / (len(frame) + 20):.1%} of the wire")
print(f"Receiver's check: CRC-32 of the frame without its FCS equals the FCS: {struct.pack('<I', zlib.crc32(frame[:-4])) == frame[-4:]}")

print()
print("How the receiver finds the next layer:")
print(f"  Ethernet: EtherType {frame[12:14].hex()} -> the payload is an IPv4 packet")
print(f"  IPv4: protocol {packet[9]} -> the payload is a TCP segment")
print(f"  TCP: destination port {segment[2] << 8 | segment[3]} -> the browser's connection")
print()
nat_client = ipaddress.ip_address("100.72.160.9")
moved = tcp_segment(server, nat_client, payload)
print(f"TCP checksum to {client}: {segment[16:18].hex()}; after NAT rewrites the address to {nat_client}: {moved[16:18].hex()}")

Ethernet header (14 bytes): 00 00 5e 00 53 01 00 00 5e 00 53 02 08 00
IPv4 header     (20 bytes): 45 00 04 10 1c 46 40 00 40 06 2e 50 c6 33 64 14 c0 00 02 0a
TCP header      (20 bytes): 01 bb c9 3a 00 00 03 e8 00 00 07 d0 50 18 ff ff ff 28 00 00
Payload       (1000 bytes): HTTP/1.1 200 OK ...
FCS              (4 bytes): 1b 73 07 5f
Frame: 1,058 bytes; on the wire, with 8 bytes of preamble and 12 of gap: 1,078
Payload share: 94.5% of the frame, 92.8% of the wire
Receiver's check: CRC-32 of the frame without its FCS equals the FCS: True

How the receiver finds the next layer:
  Ethernet: EtherType 0800 -> the payload is an IPv4 packet
  IPv4: protocol 6 -> the payload is a TCP segment
  TCP: destination port 51514 -> the browser's connection

TCP checksum to 192.0.2.10: ff28; after NAT rewrites the address to 100.72.160.9: bce1

Read the headers against the diagram. The IPv4 header starts with 45: version 4 and a header of five 32-bit words, 20 bytes. Its bytes 3 and 4, 04 10, are the total length, 1,040; byte 9, 40, is the time to live of 64, and byte 10, 06, says TCP is inside. The TCP header starts with the ports 01 bb (443) and c9 3a (51514). Both checksums use the same Internet checksum algorithm (RFC 1071), and the frame’s FCS is a CRC-32 that the receiver recomputes.

On the way up, the receiver works in reverse, and every header tells it what comes next: the EtherType 0800 says “an IPv4 packet”, the IP protocol number 6 says “a TCP segment”, and the destination port picks the connection. This handing of data to the right protocol or program above is called demultiplexing.

How much of the wire is your data?

Headers cost bytes in every frame. On standard Ethernet, the minimum sizes are a 14-byte header, a 4-byte FCS, a 20-byte IPv4 header and a 20-byte TCP header, and an IP packet may be at most 1,500 bytes long, the link’s maximum transmission unit (MTU). The wire also carries an 8-byte preamble with its start-of-frame delimiter before each frame, and stays idle for 12 byte times after it. A frame shorter than 64 bytes is padded up to 64. UDP’s header is only 8 bytes (RFC 768), and when TCP’s timestamp option is in use it makes the TCP header 32 bytes (RFC 7323).

This program works out the share of the wire that carries payload for one packet of different sizes:

"""How many of the bytes on an Ethernet wire are your data, for one packet of different sizes.

Per frame on the wire: an 8-byte preamble and start-of-frame delimiter, a 14-byte Ethernet header, the IP packet,
a 4-byte frame check sequence (FCS), then a gap of 12 byte times before the next frame. A frame (header to FCS) is
at least 64 bytes: shorter packets are padded. The IP packet is at most the MTU, 1,500 bytes on standard Ethernet.
"""
PREAMBLE, HEADER, FCS, GAP, MIN_FRAME = 8, 14, 4, 12, 64
IPV4, TCP, TCP_WITH_TIMESTAMPS, UDP = 20, 20, 32, 8


def wire_bytes(payload, transport):
    """Bytes on the wire for one IPv4 packet that carries `payload` bytes over the given transport header."""
    frame = max(HEADER + IPV4 + transport + payload + FCS, MIN_FRAME)
    return PREAMBLE + frame + GAP


cases = [
    ("1-byte keystroke over TCP", 1, TCP),
    ("20-byte sensor reading over UDP", 20, UDP),
    ("20-byte sensor reading over TCP", 20, TCP),
    ("200 bytes: ten readings in one packet", 200, TCP),
    ("1,000-byte response over TCP", 1000, TCP),
    ("1,460 bytes: a full TCP segment", 1460, TCP),
    ("1,448 bytes: full, with TCP timestamps", 1448, TCP_WITH_TIMESTAMPS),
]
print(f"{'one packet carrying':40} {'wire bytes':>10} {'efficiency':>10}")
for name, payload, transport in cases:
    wire = wire_bytes(payload, transport)
    print(f"{name:40} {wire:10,} {payload / wire:10.1%}")

one packet carrying                      wire bytes efficiency
1-byte keystroke over TCP                        84       1.2%
20-byte sensor reading over UDP                  86      23.3%
20-byte sensor reading over TCP                  98      20.4%
200 bytes: ten readings in one packet           278      71.9%
1,000-byte response over TCP                  1,078      92.8%
1,460 bytes: a full TCP segment               1,538      94.9%
1,448 bytes: full, with TCP timestamps        1,538      94.1%

Three lessons hide in that table:

• Small messages are mostly overhead. A single keystroke over TCP fills an 84-byte slot on the wire, so 1.2 % of it is the keystroke; a 20-byte sensor reading is only 20 to 23 % useful, over TCP or over UDP.
• Batching fixes it. Ten readings in one packet are 71.9 % useful instead of 20.4 %.
• Full packets are close to 95 %. A full TCP segment of 1,460 bytes uses 94.9 % of the wire, and the timestamp option costs 0.8 percentage points of it.

A payload larger than 1,460 bytes is split into several packets, and each one pays all the headers again. The exercise at the end of this lesson works that case out.

Where the layers bend

Strict layering is a model, and real networks bend it in useful and sometimes painful ways:

• TLS sits between TCP and HTTP. It is neither a transport protocol nor an application; it is usually built into a library that the application calls, and it adds encryption, a presentation job in OSI’s terms.
• The TCP checksum covers the IP addresses. It includes a pseudo-header with the source and destination addresses (RFC 9293), so the transport header depends on the network layer. The last line of the frame-building program shows the consequence: when the destination address changes, the TCP checksum changes too.
• NAT rewrites both layers. A home router that translates addresses usually rewrites port numbers as well, and has to fix the IP and TCP checksums of every packet it changes (RFC 3022).
• QUIC builds a transport on top of a transport. It provides reliable, ordered streams itself and runs over UDP (RFC 9000). It also encrypts most of its own header, so devices on the path cannot come to depend on its fields, which is what made several changes to TCP hard to deploy (RFC 9065).

Interview questions

Fresher · How do the OSI reference model and the Internet’s four-layer model differ?

OSI has seven layers: physical, data link, network, transport, session, presentation and application. The Internet model of RFC 1122 has four: link, internet, transport and application. Its application layer combines OSI’s top layers, so encryption, encodings and sessions live inside applications and libraries such as TLS, and its link layer covers both the data link and the physical layer. OSI is mostly used as a vocabulary (“layer 2”, “layer 7”), while the Internet’s protocols were designed with the four-layer model. A good answer adds the units of data: frames at the link layer, packets at the internet layer, segments or datagrams at the transport layer.

Key takeaways

• A protocol defines message formats, their order and the actions on receiving them; layering lets each protocol rely only on the service of the layer below.
• OSI has seven layers and the Internet model of RFC 1122 has four; RFC 1122’s application layer combines OSI’s presentation and application layers.
• Encapsulation adds a header at each layer (and a trailer at the link layer); each header says what is inside, which is how the receiver hands data upwards.
• On Ethernet with IPv4 and TCP, a frame costs at least 78 bytes of headers, preamble and gap, so small messages are mostly overhead and full packets reach about 95 % efficiency.
• Real networks bend the layers: TLS sits between TCP and HTTP, TCP’s checksum covers IP addresses, NAT rewrites ports and QUIC runs a transport over UDP.