Design a metrics and alerting system

A metrics and alerting system collects the numbers services expose, such as request counts, error ratios and queue lengths, stores each as a time series and pages a person when a rule over them stays true too long. The fleet in this lesson exposes 1.61 million series, each sampled every 15 seconds; to design a metrics and alerting system for it, size it by series, not by traffic. Scrapers collect the samples, a sharded store keeps recent series in memory and older blocks in object storage, compression cuts each sample to one or two bytes, and rule evaluators send alerts to notifiers that group and deduplicate them.

Requirements and estimate: series, not bytes, set the size

A shopping and payments company runs 40 services on 2,000 pods and 250 nodes. Each team runs its own scraper with 15 days of data, nobody can query across clusters, and an alert once stayed silent through an outage because the scraper evaluating it had died. Before you design a metrics and alerting system for it, ask:

• How many targets, how many series each, and how often are they scraped?
• How far back must queries reach, and at what resolution?
• How many dashboards and rules query it, and how fresh must they be?
• Who may add labels, and what stops one team from flooding the store?
• What must still work when part of the system fails, above all the alerting?

The answers here: 2,250 targets with about 1.6 million active series, scraped every 15 seconds; 13 months of history; about 400 rules evaluated every 30 seconds; teams add their own metrics; and an alert must reach a person even when a scraper, an ingester or a notifier is down.

Functional requirements: pull from services and accept pushes from short jobs; store labelled series; query ranges with functions; precompute aggregates; route alerts with grouping, silences and inhibition. Non-functional requirements: queries over the last hour in under a second; no samples lost when one storage node fails; one tenant’s label mistake never slows another; and an alerting path that reports its own failure.

The first estimate:

# A first estimate for the metrics system of this lesson. Every input is an assumption to replace with
# measurements: series per target from one scrape of each kind of target, and bytes per sample from the store's
# own statistics after a day of real data. The sizing rules for the ingest tier are Grafana Mimir's published
# capacity-planning minimums; other systems publish their own.
PODS = 2_000
SERIES_PER_POD = 660          # 10 routes x 3 status classes x 17 histogram series, plus 150 runtime series
NODES = 250
SERIES_PER_NODE = 1_000       # each node's CPU, memory, disk and network series
CLUSTER_SERIES = 40_000       # one cluster-wide exporter describing pods and deployments
SCRAPE_SECONDS = 15
BYTES_PER_SAMPLE = 1.5        # compressed; Prometheus documents an average of 1 to 2 bytes
REPLICATION = 3               # copies of every recent series in the ingest tier
GB_PER_300K = 2.5             # ingester memory for every 300,000 series held in memory
CORES_PER_300K = 1
SAMPLES_PER_DISTRIBUTOR_CORE = 25_000
HEADROOM = 1.5                # 50 % extra for peaks
LONG_DAYS = 395               # 13 months kept in object storage, at full resolution

series = PODS * SERIES_PER_POD + NODES * SERIES_PER_NODE + CLUSTER_SERIES
samples = series / SCRAPE_SECONDS
day = samples * 86_400 * BYTES_PER_SAMPLE
in_memory = series * REPLICATION

print(f"active series: {series:,}")
print(f"samples: {samples:,.0f} a second, {samples * 86_400 / 1e9:.2f} billion a day")
print(f"compressed samples: {day / 1e9:.1f} GB a day; {LONG_DAYS} days = {day * LONG_DAYS / 1e12:.1f} TB")
print(f"ingesters: {in_memory:,} series in memory with {REPLICATION} copies -> "
      f"{in_memory / 300_000 * GB_PER_300K * HEADROOM:,.0f} GB of memory and "
      f"{in_memory / 300_000 * CORES_PER_300K * HEADROOM:.0f} cores, with headroom")
print(f"distributors: {samples / SAMPLES_PER_DISTRIBUTOR_CORE * HEADROOM:.1f} cores for {samples:,.0f} samples a second")
histogram = PODS * 10 * 3 * 17
print(f"a label with 100 values added to the request histogram: {histogram:,} series become {histogram * 100:,}")

active series: 1,610,000
samples: 107,333 a second, 9.27 billion a day
compressed samples: 13.9 GB a day; 395 days = 5.5 TB
ingesters: 4,830,000 series in memory with 3 copies -> 60 GB of memory and 24 cores, with headroom
distributors: 6.4 cores for 107,333 samples a second
a label with 100 values added to the request histogram: 1,020,000 series become 102,000,000

Bytes are the small number: 13.9 GB of samples a day is 5.5 TB over 13 months. Series are the large one: 1.61 million active seriesA time series that is still receiving samples, so the store keeps it in memory. Such a store is sized by its active series rather than by bytes, because each series costs memory however small its samples are., held three times in memory, need about 60 GB of ingest memory with headroom, and one label with 100 values on the request histogram would multiply its series a hundredfold.

Follow one sample through it, from a pod to a page:

Open choices, one sentence each:

• Pull: scrapers fetch each target’s metrics on a schedule, finding targets by service discovery.
• Push: short jobs and agents send samples to a receiver instead of waiting to be scraped.
• Remote writeForwarding every sample a scraper collects to a central store, which keeps the samples longer than the scraper could and answers queries across clusters.: a scraper forwards every sample to a central store that keeps it longer.
• Delta-of-delta and XOR encoding: each timestamp and value stored by how little it differs from the previous one.
• Rollups: old samples rewritten as 5-minute and 1-hour aggregates so that long queries stay fast.
• CardinalityThe number of distinct series a metric has: one for every combination of its label values. A new label with 100 values multiplies the metric's series by 100. limits: caps on samples per scrape and series per tenant that contain a label explosion.
• For-durations: a condition must hold for a set time before its alert fires.
• Grouping and deduplication: the notifier merges related alerts into one message and drops copies.
• A dead man’s switch: an alert that always fires, so an outside service notices when alerts stop.

The deep dives that decide whether the design holds:

1. How do samples get from 2,250 targets into the store, and what happens when one target sends a million series?
2. How does a sample fit in one or two bytes, and what makes some series compress badly?
3. How are series sharded and replicated, and what does losing one storage node cost?
4. How do queries over 13 months stay fast without keeping every raw sample?
5. How does a rule become one message, without blips, duplicates or silence?
6. Who watches the alerting system, and how does anyone learn that it is down?

Answers are judged on the estimate, collection and limits, storage and compression, sharding and replication, retention and rollups, alert evaluation and notification, and watching the watcher.

Each sample takes five steps and lands on three ingesters Pro: opens with a Pro, Premium or Ultimate pass