A troubleshooting method that holds under pressure

An accounting app crashes every time one laptop starts, and 64 programs and services start with it. Turning them back on one per restart can take 64 restarts; testing half of them at a time finds the one to blame in 6. With a user waiting and a queue behind them, a method is what keeps a technician from guessing.

The method has six steps: understand the symptom and say it back, scope it, list the likely causes with the cheapest tests first, test one change at a time, fix and then verify the user’s whole task, and document. You will choose between four strategies, top-down, bottom-up, half-splittinghalf-splitting: Finding one faulty item by testing half of the suspects, or the middle of a path, at a time and keeping the half that still fails. n suspects need at most log2 n tests, rounded up: 6 for 64. and a known-good swapknown-good swap: Replacing one suspect part, such as a cable, a charger or a whole PC, with one of the same kind that is known to work, so that a single test shows whether that part is at fault., try reversible stepsreversible step: A troubleshooting step that can be undone without losing anything, such as a restart, a disabled startup item or an uninstalled update. Reversible steps come before ones with no way back, such as a reset. before anything you cannot undo, and write notes that another technician can pick up and continue.

Six steps, with a loop back when the fix falls short

A method earns its keep when the first idea is wrong, which happens often. These are the six steps of this track:

1. Understand the symptom. Let the user describe it in their own words, then say it back as something you can test: what fails, on which device, during which task. Many lost hours start with a symptom nobody pinned down.
2. Scope it. Ask who else has it (one user, one device, one site or everyone), since when, and what changed just before it started.
3. List the likely causes. Rank them by how probable each is and how cheap it is to test.
4. Test one change at a time. Make one change, repeat the same test, write down the result, and undo the change if it did not help.
5. Fix, then verify the whole task. Ask the user to do what they called about, not only the step you touched. If it still fails, go back to step 3 with what you have just learned.
6. Document. Write down the root causeroot cause: The condition that makes a fault happen. Fixing it stops the fault from coming back, while fixing only a symptom, or working around it, leaves the cause in place., the fix and how to stop it happening again, in words the next technician can follow.

Step through one ticket from the user’s first words to the notes. The drive letter is back after step 4, and step 5 shows why that was not the end:

The first theory was half right. The drive was not mapped, but that was a symptom of the cause, a group the team move dropped, and only the user’s own task, saving the workbook, brought it to light. That is what the loop back from step 5 is for.

CompTIA’s A+ Core 1 (220-1201) V15 exam objectives print a troubleshooting method too, as a “competency standard”, and state that “the methodology itself will not be tested”. Troubleshooting still matters there: hardware and network troubleshooting is the exam’s largest domain, 28 % of it.

Scope the fault before you list its causes

Two questions narrow a fault faster than any test: who has it, and what changed? Who has it tells you what everyone affected has in common, and only a part they share can be the cause:

So the user’s own report is where scoping starts: ask a neighbour to try, try another device, try the same task from another site. “Since when” turns a vague fault into a moment, and a fault that began at a moment usually began with a change: an update, a new device, a moved desk, a password change, a team move, a new policy. Ask what changed even when the user says “nothing”, because an update that ran overnight is easy not to notice.

Scope also guards against the costliest trap: fixing one PC while the same cause hits hundreds more. Say the shared printer on the second floor goes offline for one user at 9:40. A technician removes it from her PC, adds it again by the print server’s new address, and she can print. Then the same ticket comes from a second user and a third: the print server moved that morning, and all 300 PCs that print through it still look for the old address. The fix belongs at the cause, in the printer setting every PC receives, made once. The hand-made copy on the first PC now differs from the other 299, so it has to be undone, or it will fail on its own the next time the server moves.

Test the likely and cheap causes first

Once the scope is known, the list of causes is usually short, and the order of the tests decides how long the user waits. Rank each cause on two things: how likely it is, given the scope and what changed, and what it costs to test, in minutes, money and risk. A likely cause that is cheap to test goes first; an unlikely, costly one, such as replacing a board, goes last, however dramatic it sounds.

The best first checks are cheap and also cut the problem in two. Step through a laptop that will not charge: the first look, at a single light, decides which of the later checks are worth making.

Three checks in under ten minutes found the fault, and the two costly ones were never needed. The order came from cost as much as from likelihood: a failed charger and a worn battery may be about as likely as each other, but one is a five-minute swap and the other a report and perhaps a repair. The light on the connector was worth more than any of them, because one look ruled out half of the power path.

Four strategies, chosen by the clue you have

The clue in front of you also picks a strategy, and there are four:

• Top-down starts where the user sees the fault, the app, the web page or the print job, and works down through the layers it depends on. It is fastest when an error message names a layer, since everything under that layer must at least have started.
• Bottom-up starts at the physical end, power, cables and link lights, and works up. It is slow when the fault is high up, but it never skips a layer, so it is the choice when nothing works at all or there is no clue.
• Half-splitting tests in the middle of the path, or of the list of suspects, and keeps the half that still fails. With many alike suspects, such as startup items, add-ins or updates, it needs far fewer tests than the others.
• A known-good swap replaces one suspect part with one of the same kind that is known to work: a cable, a charger, a keyboard, a whole PC. One test then settles that part, as long as the spare really is good and the swap changes nothing else.

Answer three questions about a fault from your own work, and the table marks the strategies that fit it:

In short: an error message points top-down, a dead device points bottom-up or to a swap, and a long list of alike suspects points to half-splitting. The strategies also combine. A webcam that shows only black in a meeting app is a path of two parts, the camera and the app, and the webcam test uses the camera in the browser without the app. If the picture appears there, the camera and its driver work, and the fault lies with the app or the permission it was given: one test halved the path. Network faults have a chain of their own, which a method for network troubleshooting follows from the link to the application.

Halving finds one culprit among 64 in six restarts

For the laptop of the opening, Microsoft’s support article on a clean bootclean boot: Starting Windows with only essential drivers and startup programs, with the services and startup apps of other software turned off, to show whether that software causes a fault. More in “Windows admin tools: Settings, MMC, msinfo32” gives the first move: turn off every non-Microsoft service and startup app, restart, and run the app. If it now works, one of the items turned off is the culprit, and to find it the article calls testing “half of them at a time” the most efficient way. On a Mac, safe modeOn a Mac, a way of starting up that keeps login items, extensions macOS does not need and fonts it did not install from loading, and checks the startup disk. plays the same part: Apple describes it as a way to tell whether software that loads at startup causes an issue.

Step through the search on eight apps, one of which, cloud, makes the app crash. Each pass turns on the first half of the suspects still in doubt and restarts:

apps = ["sync", "vpn", "chat",
        "pdf", "audio", "cloud",
        "print", "backup"]
bad = "cloud"  # unknown to you
lo, hi = 0, len(apps)
restarts = 0
while hi - lo > 1:
    mid = (lo + hi) // 2
    on = apps[lo:mid]  # enabled
    restarts += 1
    if bad in on:  # still fails
        hi = mid
    else:
        lo = mid
    print(restarts, apps[lo:hi])
print(apps[lo], "after", restarts)

1 ['audio', 'cloud', 'print', 'backup']
2 ['audio', 'cloud']
3 ['cloud']
cloud after 3

Each restart halved the suspects, 8, then 4, then 2, then 1, so three restarts were enough. That is half-splitting: a test whose answer is yes or no rules out half of what is left, so n suspects need at most log₂ n tests, rounded up. Eight need 3, 64 need 6 and 1,000 need 10. The next program searches 64 startup items for three hidden culprits, chosen by a seeded random number generator, and counts its restarts against turning one item on per restart:

"""Find the one startup item that makes an app crash, by halving the suspects.

A clean boot (every non-Microsoft service and startup app turned off) lets the app run, so one of the N items turned
off is the culprit. Each round turns on half of the items still under suspicion, restarts and runs the app once:
if it crashes, the culprit is in the half that is on; if not, it is in the other half. A seeded random choice hides
the culprit, so every run is the same, and checking one item per restart is counted for comparison.
"""

import math
import random

N = 64  # startup items and services left after a clean boot
MINUTES_PER_RESTART = 3  # an assumption: one restart plus one run of the app
SEEDS = [1, 2, 3]

items = [f"item-{i:02d}" for i in range(1, N + 1)]


def half_split(suspects, culprit):
    """Halve the suspects until one is left; return it with one line per restart."""
    rounds = []
    while len(suspects) > 1:
        on = suspects[: len(suspects) // 2]
        crashed = culprit in on  # the restart and the test, simulated
        before = len(suspects)
        suspects = on if crashed else suspects[len(on) :]
        result = "crashes" if crashed else "runs   "
        rounds.append(f"  restart {len(rounds) + 1}: {len(on):>2} of {before:>2} on, app {result} -> {len(suspects):>2} left")
    return suspects[0], rounds


def one_by_one(suspects, culprit):
    """Turn on one item per restart, in list order, until the app crashes."""
    for restarts, item in enumerate(suspects, start=1):
        if item == culprit:
            return restarts
    raise ValueError("no culprit among the suspects")


print(f"{N} startup items, one culprit; a restart and a test take {MINUTES_PER_RESTART} minutes")
for seed in SEEDS:
    culprit = random.Random(seed).choice(items)
    found, rounds = half_split(items, culprit)
    slow = one_by_one(items, culprit)
    print(f"\nSeed {seed}: the culprit is {culprit}")
    print("\n".join(rounds))
    print(f"  halving found {found} in {len(rounds)} restarts ({len(rounds) * MINUTES_PER_RESTART} minutes)")
    print(f"  one at a time: {slow} restarts ({slow * MINUTES_PER_RESTART} minutes)")

average = sum(range(1, N + 1)) / N
print(f"\nHalving: at most {math.ceil(math.log2(N))} restarts, wherever the culprit hides (log2 of {N}, rounded up).")
print(f"One at a time: 1 to {N} restarts, {average} on average ({average * MINUTES_PER_RESTART} minutes).")

64 startup items, one culprit; a restart and a test take 3 minutes

Seed 1: the culprit is item-18
  restart 1: 32 of 64 on, app crashes -> 32 left
  restart 2: 16 of 32 on, app runs    -> 16 left
  restart 3:  8 of 16 on, app crashes ->  8 left
  restart 4:  4 of  8 on, app crashes ->  4 left
  restart 5:  2 of  4 on, app crashes ->  2 left
  restart 6:  1 of  2 on, app runs    ->  1 left
  halving found item-18 in 6 restarts (18 minutes)
  one at a time: 18 restarts (54 minutes)

Seed 2: the culprit is item-08
  restart 1: 32 of 64 on, app crashes -> 32 left
  restart 2: 16 of 32 on, app crashes -> 16 left
  restart 3:  8 of 16 on, app crashes ->  8 left
  restart 4:  4 of  8 on, app runs    ->  4 left
  restart 5:  2 of  4 on, app runs    ->  2 left
  restart 6:  1 of  2 on, app runs    ->  1 left
  halving found item-08 in 6 restarts (18 minutes)
  one at a time: 8 restarts (24 minutes)

Seed 3: the culprit is item-31
  restart 1: 32 of 64 on, app crashes -> 32 left
  restart 2: 16 of 32 on, app runs    -> 16 left
  restart 3:  8 of 16 on, app runs    ->  8 left
  restart 4:  4 of  8 on, app runs    ->  4 left
  restart 5:  2 of  4 on, app runs    ->  2 left
  restart 6:  1 of  2 on, app crashes ->  1 left
  halving found item-31 in 6 restarts (18 minutes)
  one at a time: 31 restarts (93 minutes)

Halving: at most 6 restarts, wherever the culprit hides (log2 of 64, rounded up).
One at a time: 1 to 64 restarts, 32.5 on average (97.5 minutes).

Halving took 6 restarts every time, 18 minutes at the program’s assumed 3 minutes a restart. One item per restart took 18, 8 and 31 restarts for these three culprits; over all 64 places a culprit could be, that averages 32.5 restarts (97.5 minutes), and the worst case is 64. Doubling the suspects adds one restart to halving and doubles the work of checking them one by one.

Halving needs two things: a single culprit, since two would spoil the clean yes or no of each half, and a test that gives the same answer every time it runs. Microsoft’s article adds a habit worth keeping, “Keep track of which items have been Disabled”, because the list of what is on and off is the state of the search. Developers run the same search on a program’s history: Git documents git bisect as a way to “use binary search to find the commit that introduced a bug”.

One change at a time, and a note for each

A test means something only when exactly one thing changed since the last. Here is the log of a ticket about lost sound in video calls on one laptop, one change per line and the same test after each:

The line at 10:05 split the path: the headset works outside the app, so the fault is in the app. The line at 10:11 is the one most people skip: putting the old setting back and watching the fault return proves the setting was the cause, not a coincidence. Anyone who reads this log tomorrow can repeat a line and get the same result, which is the test of a good note. A note that someone else can continue holds:

• the symptom in the user’s words, and how you restated it;
• the scope: who has it, since when, and what changed;
• every test with its exact steps, its result and its time, with its UTC offset;
• every change, and whether it was kept or undone;
• what is still unknown, when the ticket moves on;
• the cause, the fix, the user’s confirmation and the prevention.

When a ticket goes to another tier, this note lets them continue instead of starting again; the lesson on IT support roles, tiers and channels shows where it goes.

Reversible steps first, and a way back before the rest

Some steps cost nothing to undo, and some cannot be undone at all, so try them in that order. The table runs from the first kind down to the second; for the recovery options among them, the keeps-and-loses column is what Microsoft’s documentation says:

Read the table from the top: the lower a row, the more it takes to undo, and the last two have no way back at all. Apple’s safe mode article suggests the third row on a Mac, because a new account does not have the old one’s startup software and settings. Before any step without a way back, make one: a backup of the user’s files, an exported registry key, or a snapshotsnapshot: A saved state of a virtual machine, its settings, its disks and, if it was running, its memory, to which the VM can be restored later. Hyper-V calls it a checkpoint. It is not a backup. More in “Build a practice lab with virtual machines” of a virtual machine when you practise. A step that cannot be undone is sometimes right, but choose it with the evidence in hand, never because the queue is long. The practice lab lesson builds a virtual machine to try the bottom rows on.

Practise: find the culprit with the fewest restarts

Write the search of this lesson as a function. The sample tests count every restart your code asks for, so a version that checks one item at a time finds the culprit and still fails:

def find_culprit(items, still_fails):
    """Return the one item of items that makes the problem happen.

    still_fails(subset) restarts with only the items of subset turned on and
    returns True when the problem still happens. Call it no more than
    ceil(log2(len(items))) times: 3 for 8 items, 6 for 64.
    """
    # This version turns on one item per restart: the right answer, but far
    # too many restarts. Halve the suspects instead.
    for item in items:
        if still_fails([item]):
            return item
    return None

from culprit import find_culprit

APPS = ["sync", "vpn", "chat", "pdf", "audio", "cloud", "print", "backup"]


def restarts(bad, budget):
    """A pretend restart: still_fails(subset) is True when the bad item is on."""
    calls = []

    def still_fails(subset):
        calls.append(list(subset))
        assert len(calls) <= budget, f"more than {budget} restarts"
        return bad in subset

    return still_fails, calls


def test_finds_cloud():
    """finds cloud among the eight apps in 3 restarts"""
    still_fails, calls = restarts("cloud", 3)
    assert find_culprit(list(APPS), still_fails) == "cloud"
    assert len(calls) == 3


def test_every_position_of_eight():
    """finds the culprit wherever it is among the eight apps"""
    for bad in APPS:
        still_fails, calls = restarts(bad, 3)
        assert find_culprit(list(APPS), still_fails) == bad


def test_sixty_four_items():
    """finds one item among 64 in at most 6 restarts"""
    items = [f"item-{i:02d}" for i in range(1, 65)]
    for bad in ["item-01", "item-08", "item-18", "item-31", "item-32", "item-33", "item-64"]:
        still_fails, calls = restarts(bad, 6)
        assert find_culprit(items, still_fails) == bad


def test_uneven_halves():
    """keeps to the budget when the halves cannot be equal: 3 items in 2, 5 in 3, 100 in 7"""
    for n, budget in [(3, 2), (5, 3), (100, 7)]:
        items = [f"app-{i}" for i in range(n)]
        for bad in items:
            still_fails, calls = restarts(bad, budget)
            assert find_culprit(items, still_fails) == bad


def test_one_and_two_items():
    """needs no restart for one suspect and one restart for two"""
    still_fails, calls = restarts("only", 0)
    assert find_culprit(["only"], still_fails) == "only"
    for bad in ["first", "second"]:
        still_fails, calls = restarts(bad, 1)
        assert find_culprit(["first", "second"], still_fails) == bad


def test_keeps_the_list():
    """leaves the list of items it is given unchanged"""
    items = ["sync", "vpn", "chat", "pdf", "audio", "cloud", "print", "backup"]
    still_fails, calls = restarts("pdf", 3)
    find_culprit(items, still_fails)
    assert items == ["sync", "vpn", "chat", "pdf", "audio", "cloud", "print", "backup"]

Then work through the lesson’s self-check. Its questions follow your answers, beginning with the idea you are least sure of, and every choice carries its reason:

Interview questions

Warm-up (fresher): what is half-splitting, and why does it beat checking suspects one at a time?

Half-splitting finds one faulty item among many by testing half of them at a time. You turn on, or connect, half of the suspects and run a test whose answer is yes or no: if the fault appears, the culprit is in that half, and if not, it is in the other one. Each test rules out half of what is left, so n suspects need at most log₂ n tests, rounded up: 8 startup items need 3 restarts, 64 need 6 and 1,000 need 10. Checking one at a time needs up to n tests, 32.5 on average for 64 items. It works when there is a single culprit and the test gives the same answer every time, and it needs a written record of which half was on for each test.

Two harder questions start where this lesson stops: a fault that reaches many PCs at once, and a freeze that never happens while you watch. Plan an answer to each before you open its page.

Key takeaways

• Work every ticket through six steps: understand and restate the symptom, scope it, list likely and cheap causes, test one change at a time, fix and verify the whole task, then document.
• Scope before you guess: who has the fault, since when and what changed decide where to look, and a fault that hits many users is fixed at its source, not one PC at a time.
• Choose the strategy by the clue: top-down from an error message, bottom-up when nothing works, half-splitting for many alike suspects, and a known-good swap when a spare is at hand.
• Halving needs at most log₂ n tests, rounded up: 6 restarts for 64 startup items, against 18, 8 and 31 one at a time in the lesson’s three runs.
• Make one change per test, write each test and result down as you go, and undo what did not help, so another technician can repeat or continue your work.
• Try every step you can undo before one you cannot, and make a way back before a reset, a reimage or a registry edit.