Design a logging pipeline

A logging pipeline carries a fleet’s log lines, 120,000 a second in this lesson, from every process to a store where engineers can search them a minute later, at a cost the business can carry and within the rules on what logs may contain and how long they are kept. Agents on each node read and ship the lines, a replicated log buffers them through bursts and outages, aggregators parse, redact and route them, and storage is split into a hot tier for recent, searchable lines and an archive in object storage for the rest of each line’s retention.

Requirements and estimate: storage is the large number

A shopping and payments app runs 40 services on 2,000 pods. Engineers search its logs in every incident, auditors ask for a year of access records, and the storage bill grows faster than traffic. Ask first:

• How soon after a line is written must it be searchable?
• How many lines a second, of what size, and how high is the peak?
• Which lines carry personal data or secrets, and which rules set each kind’s retention?
• May the pipeline ever drop lines under load, and which ones first?
• Do searches look for words in messages, or for fields such as service and level?

The answers here: within a minute; 120,000 lines a second on average, three times that at the evening peak, about 400 bytes each; access and audit records kept a year, application logs 180 days; nothing dropped silently; most searches filter by service, time and level first.

Functional requirements: parse JSON and key-value lines from every pod; redact before storage; search by fields and words; tail one service live; keep each class for its retention, then erase it. Non-functional requirements: searchable within 60 seconds at p99; no line lost to a burst, a lost node or a two-hour store outage; one noisy service never delays the others; storage that grows slower than traffic.

The first estimate:

# A first estimate for the logging pipeline of this lesson. Every input is an assumption to replace with
# measurements: line counts from the agents' own counters, line sizes from a day of real logs, and compression
# ratios from loading a sample day into the store you plan to use.
PODS = 2_000
LINES_PER_SECOND = 120_000      # fleet average over a day
PEAK_FACTOR = 3                 # the evening peak against that average
BYTES_PER_LINE = 400            # one JSON line with its fields
STORE_COMPRESSION = 8           # raw bytes to bytes on the hot store's disks
ARCHIVE_COMPRESSION = 10        # raw bytes to bytes in object storage (large blocks, zstd)
HOT_COPIES = 2                  # the hot store keeps a replica of every shard
HOT_DAYS = 14
RETAIN_DAYS = 365
OUTAGE_HOURS = 2                # the buffer must hold a store outage this long, at peak
WIRE_COMPRESSION = 4            # batches are compressed on their way into the buffer
BUFFER_COPIES = 3               # replicas of each buffer partition

TB, GB, MB = 10**12, 10**9, 10**6
raw_day = LINES_PER_SECOND * BYTES_PER_LINE * 86_400
peak_lines = LINES_PER_SECOND * PEAK_FACTOR
peak_bytes = peak_lines * BYTES_PER_LINE
hot = raw_day / STORE_COMPRESSION * HOT_COPIES * HOT_DAYS
cold = raw_day / ARCHIVE_COMPRESSION * RETAIN_DAYS
buffer = peak_bytes * OUTAGE_HOURS * 3600 / WIRE_COMPRESSION * BUFFER_COPIES

print(f"lines: {LINES_PER_SECOND:,} a second on average, {peak_lines:,} at peak")
print(f"raw volume: {raw_day / TB:.2f} TB a day; {peak_bytes / MB:.0f} MB a second at peak")
print(f"per pod: {LINES_PER_SECOND / PODS:.0f} lines a second, {raw_day / PODS / GB:.2f} GB a day")
print(f"hot tier: {HOT_DAYS} days x {HOT_COPIES} copies at {STORE_COMPRESSION}x = {hot / TB:.1f} TB of disk")
print(f"cold tier: {RETAIN_DAYS} days at {ARCHIVE_COMPRESSION}x = {cold / TB:.0f} TB in object storage")
print(f"buffer: {OUTAGE_HOURS} hours at peak, {WIRE_COMPRESSION}x on the wire, {BUFFER_COPIES} copies = {buffer / GB:,.0f} GB")
print(f"cold share of stored bytes: {cold / (hot + cold):.0%}")

lines: 120,000 a second on average, 360,000 at peak
raw volume: 4.15 TB a day; 144 MB a second at peak
per pod: 60 lines a second, 2.07 GB a day
hot tier: 14 days x 2 copies at 8x = 14.5 TB of disk
cold tier: 365 days at 10x = 151 TB in object storage
buffer: 2 hours at peak, 4x on the wire, 3 copies = 778 GB
cold share of stored bytes: 91%

Storage is the large number: 4.15 TB of raw lines a day becomes 151 TB in the archive over a year, while two weeks in the hot tier take 14.5 TB. A store outage at the peak sets the buffer: about 778 GB of replicated log.

Follow one line through it:

Open choices, one sentence each:

• Node agents: a small process on each node that reads log files and ships parsed lines in batches.
• A replicated log as the buffer: a topic that keeps every line for hours, so a slow store loses nothing.
• Aggregators: a stateless tier that parses, redacts, routes and batches lines before storage.
• Full-text indexes: stores such as OpenSearch that index every word of every line.
• Label indexes: stores such as Loki that index a few labels and keep compressed chunks in object storage.
• Columnar stores: databases such as ClickHouse that keep each field as a compressed column.
• Hot and cold tiers: recent lines on fast disks, every line in object storage until its retention ends.
• Redaction: masking personal data and secrets in a line before anything stores it.

The deep dives that decide the design:

1. What happens to lines written during a burst, a node failure or a two-hour store outage?
2. Where are lines parsed, and how does one malformed or huge line not stall the rest?
3. Which store fits the searches engineers run, and how is it partitioned and sorted?
4. How is personal data removed before storage, and how do you know it stays removed?
5. How long does each class live, in which tier, and what erases it on time?
6. What does a day of logs cost, and which levers cut it without blinding investigations?

Answers are judged on the estimate, ingestion and buffering, redaction, storage and search, retention, cost and failures.

Every line becomes one record with two times Pro: opens with a Pro, Premium or Ultimate pass