Alerting on SLOs: burn rates and on-call

A checkout service with a 99.9% objective over 30 days may fail one request in a thousand, about 43 minutes of total outage a month. Alerting on that SLO means raising an alert when the service spends its error budgeterror budget: The share of valid events that may be bad under an objective, 100 % minus the SLO. At 99.9 %, 3,000 of 3,000,000 requests may fail, or 43.2 minutes of full outage in 30 days. More in “SLIs, SLOs, SLAs and error budgets” too fast, rather than when a metric crosses a fixed line. The speed is the burn rateburn rate: How fast an error budget is being spent, compared with spending it evenly over the window: the observed bad share divided by the share the objective allows. At a burn rate of 10, a 30-day budget lasts 3 days. More in “SLIs, SLOs, SLAs and error budgets”: the error ratio right now divided by the ratio the objective allows. A burn rate of 1 spends the budget exactly over the objective’s window. A burn rate of 14.4 sustained for one hour spends 2% of a 30-day budget, because 14.4 × 1 hour ÷ 720 hours = 0.02. Fast burns page a person, slow burns open a ticket, and burns below 1 raise nothing, so one rule covers outages of every size by asking how much of the promise to users is at risk.

Why a burn rate beats a fixed threshold

A fixed threshold, such as “page when more than 1% of requests fail over five minutes”, treats every crossing alike. It wakes someone for a three-minute blip that used a sliver of the month’s budget, and it stays silent while 0.4% of requests fail for a day and a half, which spends a fifth of the budget (4 × 36 hours ÷ 720 hours). A burn rate measures the quantity the objective protects: the error budget.

The table turns burn rates into time for a 99.9% objective over 30 days; each row divides the 30 days by the burn rate.

Burn rate	Error ratio	The budget is gone in
1	0.1%	30 days
2	0.2%	15 days
6	0.6%	5 days
14.4	1.44%	50 hours
36	3.6%	20 hours
1,000, every request failing	100%	43 minutes

The rows show why one alert cannot serve every case. A rule tuned to see a total outage within a minute ignores a burn of 6 that empties the budget in five days, and a rule sensitive enough for that burn fires at every blip. Google’s SRE workbook judges alerting rules by four properties: precision, the share of alerts that matter; recall, the share of real incidents that alert; detection time; and reset time, how long an alert keeps firing after the problem has gone (alerting on SLOs). The rest of the lesson builds alerts from these numbers:

• Multiwindow, multi-burn-rate alerts pair a long window that measures budget spent with a short one that checks the burn is still going on.
• A paging policy sends fast burns to a person now and slow burns to a queue for working hours.
• Low-traffic services need extra signal, because one failed request can look like a fast burn.
• On-call load is counted in pages per shift. A rotation stays healthy only while that count stays within limits its people can sustain.

Paired windows page fast and stop soon after the fix Pro: opens with a Pro, Premium or Ultimate pass

In this part:

• Three formulas choose the windows and thresholds
• When one failed request looks like an outage
• Only symptoms that threaten the budget page a person
• On-call load that people can sustain, in numbers
• Rules generated from the objective stay in step with it
• Practise: a page, a ticket or silence
  • Decide between a page, a ticket and silence
• Interview questions
  • What is an error-budget burn rate, and what do a burn rate of 1 and a burn rate of 14.4 mean for a 30-day SLO?
  • Checkout has a 99.9% availability SLO over 30 days. Design its alerting: which conditions page, which open a ticket, and how would you show the design is right before the next incident?
  • An alert on "error rate above 1% for 5 minutes" pages your team about 40 times a month, yet the 99.9% SLO is met every month. What do you replace it with, and how do you prove the replacement is better?
  • Your rotation of five engineers gets about 30 pages a week, most of them at night, and two people want to leave the rotation. How do you bring the load down without hiding real problems?
• Key takeaways

2 diagrams · 4 runnable examples (Python) · 1 exercise with sample tests · 3 interview answers · about 51 minutes

Opening this part with your pass…

This part opens with any MySmartCoPilot pass.

• Pro pass · 1 week · ₹79 · $2.99
• 30 days · ₹179 · $5.99
• 1 year · ₹1,699 · $45

One payment; a pass never renews. The price shown is the total you pay: no fees or charges are added at checkout.

Choose a passI have a licence key

It also opens every other in-depth part in Learn and every Pro tool. Sales are final, except where the law gives you a right to withdraw, for example in the EU and the UK (refund policy).

Read a complete sample first: Design a unique ID generator