Your country

Tools that support it use your country for local currency, number formats, units and paper size. Your choice is saved only in this browser.

Type a name or a two-letter code. Use the up and down arrow keys to move through the countries, Enter to choose one and Escape to close.

IT Support & Help Desk Fundamentals  Module 12 – Service desk practice

Incidents, problems and major incidents

Sort incidents, service requests, problems and changes, run a major incident with one owner and a half-hourly status cadence, then write a blameless review.

  • Intermediate
  • 20 minutes
  • Examples run with Python 3.14.8 and Pyodide 314.0.7
  • By MySmartCoPilot
Free lesson, Pro answers: 20 minutes
Free: the whole lesson. With a pass: the model answers to its 2 interview questions.

What you will learn

  • Distinguish incidents, service requests, problems and changes
  • Run a major incident with one owner, a bridge call, a status cadence and a timeline
  • Record a problem with its workaround and cause, and recognise when it has become a known error
  • Write a blameless post-incident review whose actions each carry an owner and a date

Before you start

What you’ll learn · Before you start

What you will learn

  • Distinguish incidents, service requests, problems and changes
  • Run a major incident with one owner, a bridge call, a status cadence and a timeline
  • Record a problem with its workaround and cause, and recognise when it has become a known error
  • Write a blameless post-incident review whose actions each carry an owner and a date

Before you start

On this page

At 09:12 on a Monday the payroll service slows to a crawl. Twenty-three people write or ring in the next fifteen minutes, the desk opens twenty-three separate tickets, and three hours later nobody can say which of them was picked up first or what had already been tried. The slowdown was never the hard part. The record was.

Four kinds of record keep that morning readable. The slowdown is an incident: a service working worse than it should, ended when people can work again. The fault behind it is a problem, a cause that keeps producing incidents until somebody removes it. That many tickets about one service in fifteen minutes cross the agreed trigger for a major incident, which runs differently: one owner, a bridge call, duplicates linked to one parent and an update every thirty minutes. Service requests and changes complete the four. Two programs replay that Monday, up to a post-incident review whose actions each name an owner and a date.

Four records, four owners, four endings

Twenty-three tickets for one outage is a filing mistake: twenty-three people each wait for their own answer and twenty-three technicians each start the same diagnosis. The cure is deciding what kind of record a report is before any work starts. An incident: An unplanned interruption of a service, or a drop in its quality, that stops people working as they should. It is judged on how quickly the service becomes usable again, not on whether the cause is understood. is an unplanned interruption or a drop in quality of a service people use. A service request: A normal, planned ask for something the service catalogue already offers, such as a laptop, a mailbox or access to a folder. Nothing is broken, so it follows an authorisation route rather than an outage route. is a normal, planned ask that the catalogue already offers. A problem: The record that owns the cause behind one or more incidents. It is judged on whether the cause is understood and gone, so a restart that restores a service ends an incident but never a problem. is the cause behind one or more incidents. A change adds, alters or removes something in the live service. Each has its own owner and its own ending:

Record A Monday example Who owns it What ends it
Incident 23 callers cannot open a payslip the desk, then one incident manager the service is usable again
Service request a second monitor for a new joiner the desk, plus whoever authorises it the thing is delivered
Problem an overnight job holds database locks a problem owner, often off the desk the cause is gone or accepted
Change moving that job to 02:00 a change owner, after an assessment the change has run and held

The last column matters most on a bad morning. An incident is judged on how quickly people work again, so a restart that gets payroll moving counts as a success even though nothing is understood; a problem is judged on whether the cause is gone, so the same restart counts for nothing. Mixing the two is how a desk restarts one service every month-end for a year. The four records are a shared industry idea: PeopleCert’s ITIL catalogue, which lists ITIL 4 and ITIL (Version 5) courses, has separate practice courses for incident management, service request management, problem management and change enablement, whose course covers assessing a change’s risk, authorising it and managing the change schedule. This track teaches the same ideas in its own words.

Step through one Monday report and watch the four questions decide where it lands:

Interactive · step by step Sorting one report into the right kind of record Step through one Monday report and watch it land in one of the four records a desk keeps.
Sorting one report into the right kind of record, step 6 of 6. Showing A user reports, Incident (loses work time), Service request (nothing is broken), Problem (owns the cause), Change (assessed first). In focus: A user reports.A user reportsIncidentloses work timeService requestnothing is brokenProblemowns the causeChangeassessed first

The drawing shows the whole scene; its 6 steps are listed below.

Every step as text
  1. One report arrives. A caller says the payslip page times out. Nothing is sorted yet. Four questions decide which record this becomes, and the answers are not always exclusive: one report can open an incident now and a problem later.
  2. Is a service working worse?. First question. A service that is down, or slow enough to stop work, is an incident: an unplanned interruption or a drop in quality. The payslip timeout answers yes, so an incident is opened.
  3. Or is it a normal ask?. Second question. A new laptop, a mailbox, access to a folder: nothing is broken, somebody wants something the catalogue already offers. That is a service request, and it follows its own authorisation path, not an outage path.
  4. Is the cause still unknown?. Third question. When the same symptom keeps coming back, or one incident has no explanation yet, the desk raises a problem record. It owns the cause, while the incident owns getting the user working again.
  5. Does something have to change?. Fourth question. Removing the cause almost always means adding, altering or removing something: a setting, a schedule, a version. That work is a change, assessed and scheduled before anyone touches production.
  6. Monday, sorted. The 23 payslip calls become one incident with its duplicates. A colleague asking for a second monitor is a service request. The long-running reporting job behind the slowness is a problem. Rescheduling that job is a change.

The step order is the order to ask in: a service hurting now outranks everything, because somebody cannot work while you file. One report can also produce two records, an incident today and a problem this week, which is normal rather than double counting.

Closing an incident as if it were the problem

Wrong: “restarted the reporting job, payroll fine, closed” and nothing else. Right: close the incident on “payroll usable at 11:10” and open a problem record for the job that keeps holding locks. The first version throws away the only evidence the next outage had.

Running a major incident takes one owner and one clock

A major incident: An incident declared on a trigger agreed in advance, such as ten tickets about one service inside fifteen minutes, and then worked differently: one named owner, a bridge call, duplicates linked to one parent and updates on a clock. is not a worse incident, it is a different way of working. The trigger is agreed in writing before the day it is needed, so nobody debates it at 09:19: for example, ten tickets about one service inside fifteen minutes, or any outage that stops money moving. Once it is declared, five things change at once. One incident manager: The one person who owns a declared major incident: they decide what is tried next, speak for the incident and keep its timeline. Nobody else reassigns the work while they hold it. owns the incident and nobody else reassigns the work. A bridge call: One open call that holds the service desk and the technical teams together for the length of a major incident, so that decisions are made in one place and written into one ticket. puts the desk and the technical teams on one line. Duplicate tickets are linked to one parent, so the queue shows one piece of work and the affected people behind it. Updates go out on a fixed status cadence: The fixed interval, often half an hour, at which a major incident's updates are published whether or not anything has changed, because silence is read as nobody working on it.. And one person keeps a timestamped incident timeline: The time-ordered list of what happened during an incident, written as the events occur rather than from memory afterwards, each entry carrying a timestamp with its UTC offset. in the parent ticket as events happen, not from memory afterwards.

Walk the six stages, including what happens when the half-hour comes round and nothing has moved:

Interactive · stage by stage The first half hour of a major incident Walk the six stages a desk works through once many tickets point at one service.
The first half hour of a major incident, step 7 of 7. Showing Spot the pattern, Declare it, Open the bridge, Link duplicates, Publish every 30 min, Close and hand on. In focus: Close and hand on.not fixed yetSpot the patternmany calls, one appDeclare itone incident managerOpen the bridgeone call, one recordLink duplicatesparent and childrenPublish every 30 mineven with no changeClose and hand ontimeline, then review

The drawing shows the whole scene; its 7 steps are listed below.

Every step as text
  1. Many calls, one service. Ten payroll calls in seven minutes is a pattern, not ten faults. A desk that reads its own queue sees it before any monitor does, because the callers are the monitor. The threshold is agreed in advance so nobody argues at 09:19.
  2. Declare early. Declaring costs a phone call; not declaring costs two hours of 23 people working separately. One named incident manager takes the incident, decides what is tried next and speaks for it. Nobody else reassigns the work.
  3. One bridge, one record. A bridge call puts the desk, the application team and the platform team in one place, and one person writes every action into the parent ticket as it happens. Side chats are where facts go to die.
  4. One parent, many children. The first ticket becomes the parent. Every later call about the same service is linked to it as a child, so the queue shows one piece of work with 22 linked children, and each caller still gets their own update at the end.
  5. Updates on the clock. Every 30 minutes the incident manager publishes what is known, what is being tried and when the next update comes, whether or not anything moved. A silent hour is read as nobody is working on it.
  6. When nothing has moved. An update that says the same as the last one is still an update. If the fix is no nearer, the bridge keeps the same shape and the clock starts again, with wider help pulled in rather than the cadence dropped.
  7. Close, then learn. Service back, watched for one cadence, then closed with a timestamped timeline in the parent ticket. The cause goes on as a problem record, and the review is booked while everyone still remembers the order of events.

The cadence is the part most desks drop first and regret most. An update that repeats the last one still tells several hundred people that someone competent is on it; silence tells them all to ring the desk again. Each timeline entry carries an ISO 8601 timestamp: A date and time written in the international ISO 8601 order, from the largest unit to the smallest, with a UTC offset after the time, as in a time written 14:05+05:30, or a trailing Z for UTC. More in “Ticket notes and handoffs people can use” with its offset, as ticket notes do, because readers tomorrow may sit in another zone; convert an entry from another desk with the timestamp converter or the timezone converter.

The program below is that Monday. It reads forty invented tickets arriving over twenty minutes, groups them by application, applies the ten-in-fifteen-minutes rule, links the duplicates and publishes the cadence:

Forty tickets, one major incident, six updates Python · outage-ladder/outage_ladder.py
"""A Monday morning on the service desk: 40 invented tickets arrive over
20 minutes. The program groups them by application, declares a major
incident when one service passes the threshold, links the duplicates to a
parent ticket and publishes a status update every 30 minutes.

Change THRESHOLD or WINDOW and run it again: the declaration, the parent
and the updates follow. Times are shown with a fixed +05:30 offset. Every
ticket, name and application here is invented.
"""

from datetime import datetime, timedelta, timezone

OFFSET = timezone(timedelta(hours=5, minutes=30), "+05:30")
OPENED = datetime(2027, 2, 15, 9, 0, tzinfo=OFFSET)

# How many tickets about one service, inside how long, declare a major incident.
THRESHOLD = 10
WINDOW = timedelta(minutes=15)
CADENCE = timedelta(minutes=30)

# (minutes after 09:00, application, what the caller said)
ARRIVALS = [
    (5, "email", "attachment will not open"),
    (6, "vpn", "disconnects every few minutes"),
    (7, "printing", "job stuck in the queue"),
    (8, "email", "search returns nothing"),
    (9, "vpn", "cannot reach the file share"),
    (10, "crm", "contact search is slow"),
    (11, "email", "signature image missing"),
    (12, "payroll", "payslip page times out"),
    (12, "payroll", "payslips will not load"),
    (13, "payroll", "spinning wheel on payslips"),
    (13, "payroll", "payroll says please try later"),
    (13, "printing", "duplex option greyed out"),
    (14, "payroll", "cannot open my payslip"),
    (14, "email", "calendar invite bounced"),
    (15, "payroll", "payroll very slow"),
    (15, "payroll", "payslip download fails"),
    (16, "payroll", "payroll times out at sign-in"),
    (16, "vpn", "slow on the guest network"),
    (17, "payroll", "month-end batch will not start"),
    (18, "email", "mailbox almost full warning"),
    (19, "payroll", "payroll unusable"),
    (19, "payroll", "payslip page blank"),
    (19, "printing", "wrong tray selected"),
    (20, "payroll", "payroll stuck at loading"),
    (20, "payroll", "cannot submit timesheets"),
    (20, "vpn", "prompts for the code twice"),
    (21, "payroll", "payroll error page"),
    (21, "payroll", "timesheet save fails"),
    (21, "crm", "report export is slow"),
    (22, "payroll", "payslips still down"),
    (22, "payroll", "payroll will not load at all"),
    (23, "payroll", "no payslip for this month"),
    (23, "payroll", "payroll keeps timing out"),
    (23, "email", "cannot send to a group"),
    (24, "payroll", "payroll down for my team"),
    (24, "payroll", "payslip page never finishes"),
    (24, "vpn", "drops when the laptop sleeps"),
    (25, "payroll", "payroll is still slow"),
    (25, "payroll", "cannot see my payslip"),
    (25, "printing", "prints only the first page"),
]

# The first update says the incident is owned; later ones say what was learned.
DECLARED = "{} is slow for everyone. Cause unknown. One ticket owns it now."

# What the responders learned, by minute after 09:00.
NOTES = {
    49: "Database lock waits are long. The overnight reporting job is still running.",
    109: "Reporting job named as the holder of the locks. Stopping it under an emergency change.",
    139: "Job stopped at 10:52. Response times normal since 11:10. Watching for 30 minutes.",
    169: "Normal for 30 minutes. Incident closed. A problem record carries the cause on.",
}
CLOSED_MINUTE = 169


def clock(moment):
    return moment.strftime("%H:%M") + "+05:30"


def minute_of(moment):
    return int((moment - OPENED).total_seconds() // 60)


def tickets():
    """One ticket per arrival, numbered in the order they reached the desk."""
    rows = []
    for number, (minute, app, text) in enumerate(ARRIVALS, start=1):
        rows.append({
            "id": "T-" + str(1400 + number),
            "at": OPENED + timedelta(minutes=minute),
            "app": app,
            "text": text,
        })
    return rows


def group_by_app(rows):
    groups = {}
    for row in rows:
        groups.setdefault(row["app"], []).append(row)
    return groups


def declaration(group):
    """The ticket whose arrival puts THRESHOLD tickets inside WINDOW, or None."""
    times = [row["at"] for row in group]
    for index in range(THRESHOLD - 1, len(times)):
        if times[index] - times[index - THRESHOLD + 1] <= WINDOW:
            return group[index]
    return None


def cadence(start, closed):
    """Update times: the declaration, then every CADENCE until the close."""
    moments = []
    moment = start
    while moment < closed:
        moments.append(moment)
        moment = moment + CADENCE
    moments.append(closed)
    return moments


def latest_note(after, upto):
    """The newest note learned after one update and up to the next, or None."""
    learned = [m for m in NOTES if minute_of(after) < m <= minute_of(upto)]
    return NOTES[max(learned)] if learned else None


rows = tickets()
groups = group_by_app(rows)

print("Tickets reaching the desk:", len(rows), "between",
      clock(rows[0]["at"]), "and", clock(rows[-1]["at"]))
print()
print("By application")
declared_app = None
declared_at = None
for app in sorted(groups, key=lambda name: -len(groups[name])):
    group = groups[app]
    trigger = declaration(group)
    verdict = "under the threshold"
    if trigger is not None:
        verdict = "MAJOR INCIDENT at " + clock(trigger["at"])
        if declared_at is None or trigger["at"] < declared_at:
            declared_app, declared_at = app, trigger["at"]
    print("  {:9} {:2} tickets  {}".format(app, len(group), verdict))

if declared_app is None:
    print()
    print("No application reached", THRESHOLD, "tickets inside",
          int(WINDOW.total_seconds() // 60), "minutes: no major incident is declared.")
else:
    group = groups[declared_app]
    parent = group[0]
    children = group[1:]
    print()
    print("Parent ticket", parent["id"], "opened for", declared_app,
          "- first report", clock(parent["at"]), "-", parent["text"])
    if children:
        print("Duplicates linked to it:", len(children),
              "(from", children[0]["id"], "to", children[-1]["id"] + ")")
    else:
        print("Duplicates linked to it: 0")
    print("The queue now shows 1 parent +", len(children), "linked children for",
          declared_app + ", and", len(rows) - len(group), "unrelated tickets.")
    print()
    print("Status updates, every 30 minutes")
    closed = OPENED + timedelta(minutes=CLOSED_MINUTE)
    moments = cadence(declared_at, closed)
    previous = None
    for moment in moments:
        if previous is None:
            note = DECLARED.format(declared_app.capitalize())
        else:
            note = latest_note(previous, moment)
            if note is None:
                note = "No change since " + clock(previous) + ". The bridge is still open."
        print(" ", clock(moment), note)
        previous = moment
    print()
    print("Updates published:", len(moments),
          "over", minute_of(closed) - minute_of(declared_at), "minutes.")

Output

Tickets reaching the desk: 40 between 09:05+05:30 and 09:25+05:30

By application
  payroll   23 tickets  MAJOR INCIDENT at 09:19+05:30
  email      6 tickets  under the threshold
  vpn        5 tickets  under the threshold
  printing   4 tickets  under the threshold
  crm        2 tickets  under the threshold

Parent ticket T-1408 opened for payroll - first report 09:12+05:30 - payslip page times out
Duplicates linked to it: 22 (from T-1409 to T-1439)
The queue now shows 1 parent + 22 linked children for payroll, and 17 unrelated tickets.

Status updates, every 30 minutes
  09:19+05:30 Payroll is slow for everyone. Cause unknown. One ticket owns it now.
  09:49+05:30 Database lock waits are long. The overnight reporting job is still running.
  10:19+05:30 No change since 09:49+05:30. The bridge is still open.
  10:49+05:30 Reporting job named as the holder of the locks. Stopping it under an emergency change.
  11:19+05:30 Job stopped at 10:52. Response times normal since 11:10. Watching for 30 minutes.
  11:49+05:30 Normal for 30 minutes. Incident closed. A problem record carries the cause on.

Updates published: 6 over 150 minutes.

Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 outage_ladder.py

Payroll crosses the line at 09:19, on its tenth ticket, while email at six tickets and the virtual private network at five stay ordinary incidents. The parent takes 22 linked children, so the queue holds eighteen pieces of work instead of forty, and six updates cover the 150 minutes to the close. Press Edit, set THRESHOLD to 15 and run it again: the declaration moves to 09:21, five payroll callers later, which is the argument to have in a quiet week rather than during an outage.

Key idea

A major incident is declared on an agreed trigger, owned by one named person, worked on one bridge, filed as one parent with its duplicates linked, and published on a clock that does not stop when the news is “no change”.

Checkpoint

Check: The four records and what ends each · Declaring and running a major incident

  1. Question 1 of 2 A caller says the shared drive has been slow all week and is slow again now. Nobody on the desk knows why. Which pair of records fits the morning best?

    Choose one answer.

    A hint

    One record gets the caller working; a second one owns the cause for later.

    Show the answer: Question 1 of 2

    Answer: An incident now, and a problem record for the repeating slowness

    The two questions are separate: what gets this person working again (an incident) and what stops the week repeating (a problem record that owns the cause). The change comes later, once the problem names something to alter.

  2. Question 2 of 2 Thirty minutes into a declared major incident the engineers have ruled out two causes and found nothing. The next status update is due. What do you publish?

    Choose one answer.

    Show the answer: Question 2 of 2

    Answer: An update saying what has been ruled out, that users are still affected, and when the next update comes

    A status cadence is a promise about frequency, not about news. The program on this page publishes six updates over 150 minutes, and the one at 10:19 says only that nothing has changed since 09:49: that is the cadence working as intended.

A problem record owns the cause, not the restart

Three closed incidents with the same symptom mean the desk is paying for one fault again and again. A problem record is where that cause goes to live: the symptoms seen, the evidence from each incident, the workaround: A safe, temporary way for a user to keep working while the real fix is still to come. It does not remove the cause, so it is logged as a known issue until the cause is fixed. More in “Difficult calls, VIPs and clean escalations” people are using meanwhile, and eventually the root cause: The condition that makes a fault happen. Fixing it stops the fault from coming back, while fixing only a symptom, or working around it, leaves the cause in place. More in “A troubleshooting method that holds under pressure”. It is deliberately slower than an incident, which wants somebody working again in minutes.

Two words are worth getting exactly right. A workaround is a safe, temporary way to keep working that leaves the cause in place: retry payroll after 11:00, run the batch in the afternoon. A known error: A problem whose cause is understood and whose workaround is written down, so the next caller can be answered in a minute instead of being diagnosed from the beginning again. is a problem whose cause is understood and whose workaround is written down, which is what lets the next caller be answered in a minute instead of diagnosed from scratch. Follow the same slowdown from three tickets to the work that ends it:

Interactive · step by step From three incidents to one cause removed Follow the same slowdown from three separate tickets to the change that ends it.
From three incidents to one cause removed, step 6 of 6. Showing Three incidents, Problem record (job holds locks), Known error (answer in a minute), Workaround (safe, temporary), Change request, Cause removed. In focus: Cause removed.Three incidentsProblem recordjob holds locksKnown erroranswer in a minuteWorkaroundsafe, temporaryChange requestCause removed

The drawing shows the whole scene; its 6 steps are listed below.

Every step as text
  1. The same symptom, thrice. Payroll crawled on three mornings in five weeks. Each time the desk restored service and closed the ticket, so three incidents are closed and nothing has been learned. Closed incidents do not stop a fourth morning.
  2. Raise the cause as a record. One problem record now owns the cause for all three. It is not measured by how fast a user got working again, which is what an incident is judged on, but by whether the cause is understood and gone.
  3. Name the cause. Lock waits during the overnight reporting job explain all three mornings. The problem record now holds a cause that someone can act on, with the evidence from each incident attached to it.
  4. A documented workaround. Until the job moves, the desk tells callers to retry after 11:00 and runs the month-end batch in the afternoon. That is a workaround: it costs people time, so it is written down rather than remembered.
  5. A problem with a workaround. A problem whose cause is understood and whose workaround is written down is a known error. The desk can answer the next caller in one minute instead of diagnosing the same thing a fourth time.
  6. Only a change removes it. Moving the job to 02:00 and restoring the dropped index is a change: assessed for risk, scheduled, with a way back if it goes wrong. The problem closes when the change has run and the mornings stay quiet.

Nothing in a problem record removes a cause by itself. Moving the job to 02:00 and restoring a dropped index is work on the live service, so it goes through change management: assessed for risk, scheduled, with a way back if it goes wrong. The problem closes when that change has run and the quiet mornings have held, and the known error is kept so the desk can recognise the shape if it ever returns.

Count the workaround's cost before you accept it

“Retry after 11:00” sounds cheap until you multiply: 300 people losing ten minutes each is 50 hours a month. That number belongs in the problem record, because it is the argument that gets the change scheduled.

A blameless review asks what let it happen

A post-incident review: The document written after an incident: a summary, the impact in numbers, a timeline, the root cause with its contributing factors, what went well, and actions that each carry an owner and a date. is the written record of what happened, what it cost, how it was handled, why it was possible and what will change. Google’s site reliability engineering book, which publishes the practice its teams use, describes a blameless review: A post-incident review that looks for the conditions which let a fault happen rather than the person who acted, on the assumption that everyone did something sensible with the information they had. as one that looks for the contributing causes “without indicting any individual or team”, on the assumption that everyone acted sensibly with the information they had. The practical test is the question you ask: not “who ran the job” but “what let a job run seven hours over its window without anyone being told”.

Six headings make a review usable by a stranger: a summary, the impact in numbers, a timeline, the root cause and its contributing factors, what went well, and what to do with an owner and a date on each action. The last is where most reviews fail. “Improve monitoring” is not an action; “alert when payroll lock waits pass 5 s; owner: platform team; by: 2027-02-23” is, because a date written largest unit first can be checked by anyone and an owner can be asked. The linter below holds both rules and reads two invented reviews:

Linting a weak and a strong post-incident review Python · pir/pir_lint.py
"""Check a post-incident review: the six sections it needs, and an owner
plus an ISO 8601 date on every action.

The reviews below are invented. Only the shape of the document is checked,
never whether its wording is true.
"""

REQUIRED_SECTIONS = [
    "Summary",
    "Impact",
    "Timeline",
    "Root cause and contributing factors",
    "What went well",
    "What to do (owner, date)",
]

ACTION_SECTION = "What to do (owner, date)"

DAYS_IN_MONTH = [31, 29, 31, 30, 31, 30, 31, 31, 30, 31, 30, 31]


def split_sections(review_text):
    """Return the heading of each '## ' section mapped to its body lines."""
    sections = {}
    heading = None
    for raw in review_text.splitlines():
        line = raw.strip()
        if line.startswith("## "):
            heading = line[3:].strip()
            sections[heading] = []
        elif heading is not None and line:
            sections[heading].append(line)
    return sections


def is_iso_date(token):
    """True for an ISO 8601 calendar date, 2027-02-16 and no other shape."""
    if len(token) != 10 or token[4] != "-" or token[7] != "-":
        return False
    parts = token.split("-")
    if not all(part.isdigit() for part in parts):
        return False
    year, month, day = (int(part) for part in parts)
    if not 1 <= month <= 12:
        return False
    limit = DAYS_IN_MONTH[month - 1]
    if month == 2 and not (year % 4 == 0 and (year % 100 != 0 or year % 400 == 0)):
        limit = 28
    return 1 <= day <= limit


def read_action(line):
    """Pull the owner and the date out of one action line."""
    owner = None
    date = None
    for field in line.lstrip("-").split(";"):
        field = field.strip()
        lowered = field.lower()
        if lowered.startswith("owner:"):
            owner = field[len("owner:"):].strip() or None
        elif lowered.startswith("by:"):
            candidate = field[len("by:"):].strip()
            date = candidate if is_iso_date(candidate) else None
    return owner, date


def lint(review_text):
    """Report the sections a review is missing and its faulty action lines."""
    sections = split_sections(review_text)
    missing = [name for name in REQUIRED_SECTIONS if name not in sections]

    actions = [line for line in sections.get(ACTION_SECTION, []) if line.startswith("-")]
    faults = []
    for line in actions:
        owner, date = read_action(line)
        short = line.lstrip("- ").split(";")[0]
        if owner is None and date is None:
            faults.append(short + " -> no owner and no ISO 8601 date")
        elif owner is None:
            faults.append(short + " -> no owner")
        elif date is None:
            faults.append(short + " -> no ISO 8601 date")

    return {
        "missing_sections": missing,
        "actions": len(actions),
        "action_faults": faults,
        "ready": not missing and not faults and bool(actions),
    }


def report(name, review_text):
    result = lint(review_text)
    print(name)
    print("  sections found   :", len(REQUIRED_SECTIONS) - len(result["missing_sections"]),
          "of", len(REQUIRED_SECTIONS))
    if result["missing_sections"]:
        for heading in result["missing_sections"]:
            print("  missing section  :", heading)
    print("  actions checked  :", result["actions"])
    if result["action_faults"]:
        for fault in result["action_faults"]:
            print("  action fault     :", fault)
    print("  ready to publish :", "yes" if result["ready"] else "no")
    print()


WEAK = """
## Summary
Payroll was slow for about two hours on Monday morning.

## Impact
Around 300 staff could not open payslips. Nobody could run the month-end batch.

## Root cause and contributing factors
A reporting job held locks on the payroll database for longer than expected.

## What to do (owner, date)
- Add an alert on lock wait time; owner: platform team; by: soon
- Move the reporting job out of office hours; by: 2027-02-23
- Write a runbook for payroll slowdowns; owner: service desk
"""

STRONG = """
## Summary
Payroll slowed to a crawl for 1 h 58 min on Monday morning and recovered when a
reporting job was stopped.

## Impact
Around 300 staff could not open payslips between 09:12 and 11:10. The month-end
batch started 2 hours late and finished inside its window.

## Timeline
- 09:12 first ticket: payslip page times out
- 09:19 tenth payroll ticket arrives; major incident declared
- 09:20 parent ticket opened on the first report; 22 duplicates linked to it
- 09:45 lock waits traced to the overnight reporting job, still running
- 10:52 reporting job stopped
- 11:10 response times back to normal; incident closed at 11:49

## Root cause and contributing factors
The reporting job ran 7 hours longer than its usual 40 minutes after a new index
was dropped, and held locks the payroll pages needed. Nothing alerted on lock
wait time, so the desk learned about it from users.

## What went well
The first responder scoped the outage in 7 minutes. Linking duplicates to one
parent kept the queue readable for the whole two hours.

## What to do (owner, date)
- Alert when payroll lock waits pass 5 s; owner: platform team; by: 2027-02-23
- Restore the dropped index and record it in the CMDB; owner: database team; by: 2027-02-18
- Run the reporting job at 02:00 under a change request; owner: platform team; by: 2027-03-02
- Add a payroll slowdown runbook with the parent-ticket step; owner: service desk; by: 2027-03-09
"""

report("Weak review", WEAK)
report("Strong review", STRONG)

Output

Weak review
  sections found   : 4 of 6
  missing section  : Timeline
  missing section  : What went well
  actions checked  : 3
  action fault     : Add an alert on lock wait time -> no ISO 8601 date
  action fault     : Move the reporting job out of office hours -> no owner
  action fault     : Write a runbook for payroll slowdowns -> no ISO 8601 date
  ready to publish : no

Strong review
  sections found   : 6 of 6
  actions checked  : 4
  ready to publish : yes

Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 pir_lint.py

The weak review loses two of the six headings, and all three of its actions fail: one gives “soon” as its date, one names no owner and one has no date at all. What the linter cannot judge is whether the cause named is the real one, so a document can pass every structural rule and still be useless. That is also why the strong review’s “what went well” section is not decoration: it keeps the habits that held.

An action that names a person instead of a system

Wrong: “Asha to be more careful when dropping indexes.” Right: “Block index changes outside a change window in the deployment check; owner: database team; by: a date in the next two weeks.” The first asks a person to be different next time; the second asks the system to make the mistake harder, which is the only thing that survives a staff change.

Checkpoint

Check: Problem records, workarounds and known errors · The blameless post-incident review

  1. Question 1 of 2 What turns a problem record into a known error?

    Choose one answer.

    Show the answer: Question 1 of 2

    Answer: Its cause is understood and its workaround is written down

    A known error sits between "we do not know" and "it is fixed": the cause is named, a safe temporary way of working is written down, and the change that removes the cause is still to come.

  2. Question 2 of 2 Which of these four actions in a post-incident review would the linter on this page accept, and would also survive the person who wrote it leaving?

    Choose one answer.

    Show the answer: Question 2 of 2

    Answer: Alert when the nightly job runs past 90 minutes; owner: platform team; by: an ISO 8601 date

    Two structural rules, one judgement. The linter checks that each action line carries an owner and an ISO 8601 date. The judgement it cannot make is whether the action changes the system instead of asking a person to try harder.

A security incident takes a different road

Some incidents must leave the normal path the moment you recognise them. A laptop that encrypts its own files, a sign-in from a country nobody travelled to, a mailbox quietly forwarding to an outside address: these are security incidents, and the desk’s job is to recognise, preserve and hand over, not to investigate. Spotting the hand-off is the skill; the investigation belongs elsewhere. Three habits that help on an ordinary outage are reversed on this path:

On an ordinary incident On a suspected security incident
restore the service as fast as you can preserve the device and the logs as they are
broadcast widely on a cadence tell the named security contact first, on their channel
close when people can work hand over, record what you did, and follow their instructions

NIST’s SP 800-61 Revision 3 sets out how an organisation plans for this; its own aim is to help organisations “prepare for incident responses, reduce the number and impact of incidents”. It arranges the work around the functions of the Cybersecurity Framework, and it is the reference to read before writing your desk’s hand-off page. One of its outcomes, RS.AN-07, wants incident data and its metadata collected with their integrity and provenance kept intact, which is why the desk does not wipe, restart or tidy a suspect machine before the security contact says what to do with it. CompTIA’s A+ Core 2 objectives (220-1202, objective 4.6) put the same duties under incident response: a The written record of who held a piece of evidence, such as a laptop or a copy of its disk, and when, from the moment it was collected, so nobody can later claim it was changed., informing management, preserving the data and documenting the incident.

Three things not to do on a suspected security incident

Do not wipe, reimage or “clean” the device: that destroys the evidence. Do not reuse the same wide status cadence, because an attacker reading it learns what you know. Do not keep investigating once you have handed over, unless the security contact asks you to; a second pair of hands on the same machine ruins a timeline.

Checkpoint

Check: Handing a security incident over

  1. One question A user rings: her files have been renamed with an unfamiliar extension and a note on the desktop demands payment. What is your first action?

    Choose one answer.

    Show the answer: One question

    Answer: Tell the named security contact and leave the machine as it is

    A suspected security incident leaves the normal path at once: preserve the device and the logs, tell the named contact first, record what you did, then follow their instructions.

Practise: lint a post-incident review

Rebuild the review section’s linter yourself. The exercise gives you the six headings and asks for the problems in a fixed order: missing sections first, then each action’s missing owner or date. The sample tests cover a ready review, missing and empty sections, an action with no owner whose only date is “soon”, and dates that only look real, such as a leap day in a year that has none.

Exercise · Medium · Python

Check a post-incident review before you share it

Write lint_pir(review_text) in lint_pir.py. It reads one post-incident review written as plain text and returns a list of its problems, so that the review can be fixed before it is shared. A review that is ready gives an empty list.

The review is split into sections. A section starts with a line ## <heading> (spaces around a line do not matter), and every non-blank line after it belongs to that section until the next heading. REQUIRED_SECTIONS holds the six required headings, in this order:

  1. Summary
  2. Impact
  3. Timeline
  4. Root cause and contributing factors
  5. What went well
  6. What to do (owner, date)

Inside What to do (owner, date), an action is a line that starts with -. After the dash, an action holds fields separated by ;. A field that starts with owner: names the owner, and a field that starts with by: gives the date (the field names in any letter case, the values with the spaces around them removed), as in - Alert when payroll lock waits pass 5 s; owner: platform team; by: 2027-02-23.

Build problems in this order:

  1. For every required heading that is absent, in the order of the list above, add "missing section: <heading>". When the What to do (owner, date) section is absent, stop here and return the list.
  2. When that section has no action lines, add "no actions listed".
  3. For each action, numbered from 1 in the order it appears: when it has no owner (no owner: field, or one with nothing after it), add "action <n>: no owner"; then, when it has no valid date, add "action <n>: no ISO 8601 date".

A valid date is a real calendar day written YYYY-MM-DD: exactly ten characters, a dash at the fifth and the eighth, digits everywhere else, a month from 1 to 12 and a day that exists in that month. February has 29 days in a leap year: a year divisible by 4, except a year divisible by 100 that is not divisible by 400. So 2028-02-29 is valid, 2027-02-29 and 2100-02-29 are not, and neither is soon or 23-02-27.

For example, a review with all six sections whose only action is - Write a runbook; by: soon returns ["action 1: no owner", "action 1: no ISO 8601 date"].

Starter code · lint_pir.py

REQUIRED_SECTIONS = [
    "Summary",
    "Impact",
    "Timeline",
    "Root cause and contributing factors",
    "What went well",
    "What to do (owner, date)",
]

ACTION_SECTION = "What to do (owner, date)"

DAYS_IN_MONTH = [31, 28, 31, 30, 31, 30, 31, 31, 30, 31, 30, 31]


def lint_pir(review_text):
    """Return the problems of one post-incident review as a list of strings.

    An empty list means the review is ready to share.
    """
    problems = []
    # Your code here.
    return problems
The sample tests · test_lint_pir.py
from lint_pir import lint_pir

BODIES = {
    "Summary": "Payroll pages were slow for 1 h 58 min on Monday morning.",
    "Impact": "Around 300 staff could not open payslips.",
    "Timeline": "- 09:12 first ticket\n- 09:19 major incident declared",
    "Root cause and contributing factors": "A reporting job held locks the payroll pages needed.",
    "What went well": "Duplicates were linked to one parent ticket.",
}


def review(actions, drop=()):
    """A review with every section except those in drop; actions are the lines of the action section."""
    parts = [f"## {name}\n{body}" for name, body in BODIES.items() if name not in drop]
    if "What to do (owner, date)" not in drop:
        parts.append("## What to do (owner, date)\n" + "\n".join(actions))
    return "\n\n".join(parts) + "\n"


GOOD = "- Alert when payroll lock waits pass 5 s; owner: platform team; by: 2027-02-23"


def test_ready_review_has_no_problems():
    """a review with all six sections and a complete action is ready"""
    assert lint_pir(review([GOOD])) == []


def test_missing_sections_in_order():
    """missing headings are listed in the order of the required list"""
    text = review([GOOD], drop=("Timeline", "Impact"))
    assert lint_pir(text) == ["missing section: Impact", "missing section: Timeline"]


def test_no_action_section_stops_there():
    """without the action section only the missing headings are reported"""
    assert lint_pir(review([], drop=("What to do (owner, date)",))) == ["missing section: What to do (owner, date)"]


def test_empty_action_section():
    """an action section without a single action line is reported"""
    assert lint_pir(review([])) == ["no actions listed"]


def test_action_without_owner_and_date():
    """an action with no owner and a date of soon has both problems, owner first"""
    text = review([GOOD, "- Write a runbook for payroll slowdowns; by: soon"])
    assert lint_pir(text) == ["action 2: no owner", "action 2: no ISO 8601 date"]


def test_leap_day_only_in_a_leap_year():
    """29 February exists in 2028 but not in 2027"""
    ok = "- Restore the dropped index; owner: database team; by: 2028-02-29"
    bad = "- Move the reporting job; owner: platform team; by: 2027-02-29"
    assert lint_pir(review([ok, bad])) == ["action 2: no ISO 8601 date"]


def test_century_years_follow_the_400_rule():
    """2000 is a leap year and 2100 is not"""
    ok = "- Restore the dropped index; owner: database team; by: 2000-02-29"
    bad = "- Move the reporting job; owner: platform team; by: 2100-02-29"
    assert lint_pir(review([ok, bad])) == ["action 2: no ISO 8601 date"]


def test_a_letter_in_the_date_is_not_a_date():
    """a capital O typed for a zero makes the date invalid"""
    assert lint_pir(review(["- Alert on lock waits; owner: platform team; by: 2027-O2-23"])) == [
        "action 1: no ISO 8601 date"
    ]
A hint

Work in three passes. First split the text into sections: walk review_text.splitlines(), strip each line, start a new list when a line begins with "## " and append every other non-blank line to the current list. Then report each name of REQUIRED_SECTIONS that is not a key, and return early when ACTION_SECTION is missing. Last, keep the lines of the action section that start with "-", split each on ";" and look for the owner: and by: fields. For the date, check its shape first, then the month, then the day against DAYS_IN_MONTH, with 29 for February in a leap year.

The sample tests run on this device, in your browser (Pyodide): nothing is sent to mysmartcopilot.com. The first run downloads Python (about 13.5 MB), which is kept for the next runs. A check in your browser is feedback for you, not proof that the code is right for every input.

The questions below cover all five ideas, from sorting a report to the security hand-off. They adapt as you answer, so a right answer at one level brings a harder one next.

Practice that adapts to you

Practise this lesson

12 questions about 5 ideas of this lesson. Each answer is explained at once, and what you get right is kept in this browser.

  1. Question 1 of 12 A new joiner asks for a second monitor, which the catalogue already offers. Which record is that?

    Choose one answer.

    Show the answer: Question 1 of 12

    Answer: A service request

    Ask whether a service is working worse than it should. If the answer is no and the catalogue already offers the thing, it is a service request and follows the authorisation route rather than an outage route.

  2. Question 2 of 12 Restarting a stuck reporting job gets payroll working within four minutes, and nobody knows why the job stuck. Which record can honestly be closed?

    Choose one answer.

    Show the answer: Question 2 of 12

    Answer: The incident, because people can work again

    Each record has its own ending. Close the incident on "payroll usable", and open or keep a problem record for the job that stuck, so the next outage starts with evidence instead of a blank page.

  3. Question 3 of 12 The desk declares a major incident when 10 tickets about one service arrive inside 15 minutes. In the run of the program on this page, how many minutes after 09:00 is the payroll major incident declared?

    Type a number.

    Show the answer: Question 3 of 12

    Answer: 19 minutes after 09:00

    The first payroll ticket arrives at 09:12 and the tenth at 09:19, which completes ten inside fifteen minutes, so the declaration is at 09:19. The other 13 payroll tickets arrive afterwards, and all 22 tickets after the first are linked to it as duplicates.

  4. Question 4 of 12 Twenty-three tickets about one service are filed as one parent with its duplicates linked to it. How many children does the parent carry?

    Type a number.

    Show the answer: Question 4 of 12

    Answer: 22 children

    The first ticket becomes the parent and the other 22 are linked to it, which is what the program prints: one parent, 22 children, and the 17 unrelated tickets left as ordinary work.

  5. Question 5 of 12 During a declared outage three senior engineers are each trying a different fix, two of them on the same server, and the parent ticket has no entries for forty minutes. Which single change helps most?

    Choose one answer.

    Show the answer: Question 5 of 12

    Answer: Name one incident manager who decides what is tried next and keeps the parent ticket current

    A major incident is a way of working, and its first rule is one owner. The bridge, the parent ticket and the cadence all depend on one person deciding which step is tried now.

  6. Question 6 of 12 What does a workaround do?

    Choose one answer.

    Show the answer: Question 6 of 12

    Answer: It lets people keep working while the cause is still in place

    A workaround buys time and leaves the cause alone. Writing it down, with the time it costs people, is what turns it from a rumour on the desk into an argument for scheduling the change.

  7. Question 7 of 12 The same payroll slowdown has been restored and closed three times in five weeks. Each incident was handled inside its target time. What does that tell you?

    Choose one answer.

    Show the answer: Question 7 of 12

    Answer: The incidents were handled well and the cause has never been owned by anybody

    Incident work and problem work are measured differently on purpose. A run of well-handled, identical incidents is the clearest sign that nobody owns the cause.

  8. Question 8 of 12 The linter on this page checks six headings in a post-incident review. In its recorded output, how many of the six does the weak review carry?

    Type a number.

    Show the answer: Question 8 of 12

    Answer: 4 headings

    The weak review has the summary, the impact, the root cause and the actions, and is missing the timeline and what went well. The linter prints "4 of 6" and lists the two missing headings.

  9. Question 9 of 12 Which question belongs in a blameless post-incident review?

    Choose one answer.

    Show the answer: Question 9 of 12

    Answer: What let a job run seven hours past its window without anyone being told?

    The test is whether the answer can become an action on a system. Google's site reliability engineering book describes the practice as looking for contributing causes without indicting anyone, on the assumption that people acted sensibly with what they knew.

  10. Question 10 of 12 A review's only action reads "the team will watch the overnight jobs more closely". You must rewrite it so it can be checked in three months. What is the best rewrite?

    Choose one answer.

    Show the answer: Question 10 of 12

    Answer: An alert when a nightly job passes a stated runtime, owned by the platform team, with an ISO 8601 date

    A good action names an owner and a date so it can be chased, and changes the system rather than adding human vigilance. The alert satisfies all three tests; the rota and the meeting only satisfy two.

  11. Question 11 of 12 You notice that a user's mailbox has a rule forwarding every message to an outside address that nobody recognises. What do you do first?

    Choose one answer.

    Show the answer: Question 11 of 12

    Answer: Tell the named security contact on their channel

    Recognise, preserve, hand over. The desk's value on a security incident is noticing it early and not spoiling the evidence, not working out what happened.

  12. Question 12 of 12 Which of these habits, good on an ordinary outage, work against you on a suspected security incident? Select all that apply.

    Choose every answer that is right.

    Show the answer: Question 12 of 12

    Answer:

    • Restoring the service as fast as possible, including by reimaging the device
    • Publishing a wide status update on a fixed cadence

    The two reflexes to suspend are speed and breadth. Preserving the machine and keeping the first message narrow are what the separate path exists for; careful records and a single owner carry over unchanged.

Interview questions

Warm-up (fresher): A finance user emails “please give me access to the new reports folder”. Another emails “the reports folder is empty, the whole department has lost its files”. Which one is a service request, which is an incident, and what do you do first with each?

Access to a folder the catalogue already offers is a service request: nothing is broken, so the first move is to raise it on the standard route and ask who authorises that folder, never to grant it myself. An empty folder that a whole department relied on is an incident, and a wide one, so impact comes before cause: I scope who is affected and whether the files are unreachable or really gone, then say the narrowest version back in one sentence for the user to correct. If the whole department is affected I check the major-incident trigger before touching anything, because other callers are probably already dialling. Missing files also raise a question I do not settle alone, whether this is a restore job or a security matter, and that decides who it goes to next.

The two questions below ask for the same skills on other days: a night when you are alone on the desk and one fault hides behind three applications, and the review of an outage that a supplier’s change set off. Plan your opening moves on both before you open either.

Model answers open with any pass: ₹79 for 1 week ($2.99), one payment that never renews.

  1. Mid

    It is 03:10 on a Saturday and you are alone on the desk. In twelve minutes nine tickets from two branch offices say nobody there can sign in to email, the rota app or the payroll site. Your written trigger is ten tickets about one service inside fifteen minutes. Walk me through your first thirty minutes.

    Model answer, Pro. Model answers open with any pass; prices are in the notice on this page. First step and follow-ups

  2. Senior

    A supplier rotated the signing certificate of the hosted sign-in service your staff use, without warning, and for 2 h 40 min nobody could sign in to email or payroll. You cannot see or change the supplier's systems. Draft the post-incident review: how you word the root cause, and which actions your own organisation can own and date.

    Model answer, Pro. First step and follow-ups

Flashcards · 7 cards

Recall it, then check

Say the answer to yourself before you look: recalling it is what makes it stick.

  1. What do an incident, a service request, a problem and a change each do?

    An incident restores a service, a service request delivers something normal, a problem owns the cause behind incidents, and a change alters the live service.

  2. One user asks for access to a reports folder; another says the folder is empty for the whole department. Which is which?

    The access is a service request: raise it on the standard route and ask who authorises the folder, never grant it yourself. The empty folder is an incident, and a wide one, so scope who is affected first and check the major-incident trigger.

  3. How is a major incident run?

    It is declared on a trigger agreed in advance, owned by one named incident manager, worked on one bridge call, filed as one parent with its duplicates linked, and updated on a fixed clock even when there is no news.

  4. Why are major-incident updates published on a fixed cadence, even with nothing new?

    Because silence is read as nobody working on it. An interval of half an hour is common.

  5. What is a known error, and what alone removes its cause?

    A problem whose cause is understood and whose workaround is written down. Only a change, made through change management, removes the cause.

  6. What does a blameless post-incident review ask, and what must every action carry?

    What let the fault happen, rather than who caused it. Every action names an owner and a date, so it can be checked later.

  7. A ticket turns out to be a suspected security incident. What changes?

    It leaves the normal path at once: preserve the device and the logs, tell the named contact, and hand over rather than investigate.

Key takeaways

  • Four records answer four different questions: an incident restores a service, a service request delivers something normal, a problem owns the cause behind incidents, and a change alters the live service.
  • A major incident is declared on a trigger agreed in advance, owned by one named incident manager, worked on a bridge call, filed as one parent with its duplicates linked, and published on a clock whether or not anything has moved, because silence reads as nobody working on it.
  • A problem record carries the symptoms, the workaround and the cause; a known error is a problem whose cause is understood and whose workaround is written down; only a change through change management removes the cause.
  • A blameless post-incident review asks what let the fault happen rather than who caused it, and every action names an owner and a date so it can be checked later.
  • A suspected security incident leaves the normal path at once: preserve the device and the logs, tell the named contact, hand over rather than investigate.

Terms in this lesson

blameless review
A post-incident review that looks for the conditions which let a fault happen rather than the person who acted, on the assumption that everyone did something sensible with the information they had.
bridge call
One open call that holds the service desk and the technical teams together for the length of a major incident, so that decisions are made in one place and written into one ticket.
incident
An unplanned interruption of a service, or a drop in its quality, that stops people working as they should. It is judged on how quickly the service becomes usable again, not on whether the cause is understood.
incident manager
The one person who owns a declared major incident: they decide what is tried next, speak for the incident and keep its timeline. Nobody else reassigns the work while they hold it.
incident timeline
The time-ordered list of what happened during an incident, written as the events occur rather than from memory afterwards, each entry carrying a timestamp with its UTC offset.
known error
A problem whose cause is understood and whose workaround is written down, so the next caller can be answered in a minute instead of being diagnosed from the beginning again.
major incident
An incident declared on a trigger agreed in advance, such as ten tickets about one service inside fifteen minutes, and then worked differently: one named owner, a bridge call, duplicates linked to one parent and updates on a clock.
post-incident review
The document written after an incident: a summary, the impact in numbers, a timeline, the root cause with its contributing factors, what went well, and actions that each carry an owner and a date.
problem
The record that owns the cause behind one or more incidents. It is judged on whether the cause is understood and gone, so a restart that restores a service ends an incident but never a problem.
service request
A normal, planned ask for something the service catalogue already offers, such as a laptop, a mailbox or access to a folder. Nothing is broken, so it follows an authorisation route rather than an outage route.
status cadence
The fixed interval, often half an hour, at which a major incident's updates are published whether or not anything has changed, because silence is read as nobody working on it.

References

Related tools

Report a problem with this lesson

Quick answers and tool search

Type to search tools or to get a quick answer, for example 18% of 2500. Use the up and down arrow keys to move through the results, Enter to choose, and Escape to close.