Card sorting and tree testing

Twenty people sorted twelve cards from MealBox’s account area into piles of their own making, and the pair Pause plan and Cancel plan came out at 0.75: fifteen of the twenty kept them together. Five people did not, and the five are the interesting part, because a menu has to work for them too.

Two methods carry most of this work. A card sort asks people to group content and say what they would call each group, which is how a structure gets proposed; tree testing asks people to find things in a structure that has been written down, which is how it gets checked. This lesson runs both on invented data you can re-run: a similarity matrix and an average-linkage tree for the MealBox sort, then success, directness and first clicks for an eight-task tree test of the Ward Desk menu, ending with the one change each weak task asks for.

Three kinds of card sort, and what each one can tell you

MealBox has twelve things a customer can do to their account, and today they sit in one list of twelve links. A card sortcard sort: A study in which participants group content items written on cards, and often name each group, so that a structure can be proposed from how people think the content belongs together. puts each of those twelve on a card and asks people to group them. Three variants ask three different questions.

In an open sortopen card sort: A card sort in which participants are given the cards only: they make as many groups as they like and name each group in their own words. participants get the cards and nothing else: they make as many piles as they like and name each pile themselves. In a closed sortclosed card sort: A card sort in which the categories are given to participants already named, so the study reports how well those categories absorb the cards rather than what groups people would make. the categories come from you, already named, and participants place each card into one of them. A hybrid sortA card sort in which participants are given your named categories and may also create categories of their own for cards that fit nowhere. gives your categories and allows new ones, so a card that fits nowhere has somewhere to go.

The difference matters more than it looks, because only the open sort can tell you something you did not already think of. Nielsen Norman Group draws the line sharply: card sorting is a generative method, used to find possible groupings and the vocabulary that goes with them, and it is a poor way to evaluate a hierarchy you have already built. A closed sort reports how well your categories absorb the cards; it cannot report the category you forgot, because no participant can invent it.

Answer three questions about your own study in the table below and watch options drop out:

The table’s reason for ruling out an open sort when only hours are available is worth taking seriously: the slow part of an open sort is coding the pile names people wrote, and a coded-badly open sort is worse than a clean closed one.

Five participants are enough to test a design, not to build one

The usual advice for a usability test is to watch five people, because the fifth rarely shows you something new. Card sorting does not behave that way, and there is a measurement behind the claim. Tom Tullis and Larry Wood ran a sort with 168 participants at Fidelity Investments and then simulated smaller studies by drawing random subsets of that set, so the result of a 20-person study could be compared with the result of the full 168. Nielsen Norman Group reports the correlations:

Tullis and Wood recommended 20 to 30 participants; Nielsen’s own reading of the same numbers is 15 for most projects and 30 where a lot rides on the structure. Both readings beat five by a wide margin, and the reason is the kind of question being asked. A usability test evaluates something that exists, so one person struggling is already evidence of a problem. A sort generates something that does not exist yet, out of mental models and vocabulary that differ from person to person, and a stable picture of that variation needs a crowd.

Treat these figures as one careful study rather than a law: it was run at one company on one information space, and the analysis looked only at the numbers, not at what participants said while sorting. Nielsen Norman Group names both limits itself. The practical consequence is the useful part — spend the budget you save by stopping at 15 on a round of tree testing rather than on participants 16 to 30, because the second method finds a different class of problem.

The similarity matrix is the sort, and the cut is your guess

Everything quantitative in a card sort comes from one number per pair of cards: the share of participants who put those two cards in the same pile. All twenty together is 1.00, half of them is 0.50. Collect that number for every pair and you have a similarity matrixsimilarity matrix: A table holding one number per pair of cards in a sort: the share of participants who put those two cards in the same group, from 0 for none to 1 for all of them., which is the sort in a form you can compute with.

The program below builds the matrix for the twelve MealBox cards from twenty synthetic sorts, then groups the cards. Press Run to see the first five rows, five named pairs and three readings of the tree; press Edit and move a card between piles in UNSETTLED to watch the groups move:

"""An open card sort of the MealBox account area, analysed with the standard library.

Twenty synthetic participants sort twelve cards. Each one starts from a latent
grouping and then moves the cards this lesson calls unsettled; some keep one
pile for two of them. Every pile, card and number below is invented.
"""

import random

CARDS = [
    "Pause plan", "Cancel plan", "Skip one day",
    "Change address", "Change delivery window",
    "Update payment card", "Download invoices", "Apply a coupon",
    "Change meal preferences", "Rate a dish",
    "Report a missing box", "Contact support",
]
LATENT = ["plan", "plan", "plan", "delivery", "delivery", "money",
          "money", "money", "food", "food", "help", "help"]
UNSETTLED = {"Cancel plan": (0.25, ["help"]), "Skip one day": (0.40, ["delivery"]),
             "Apply a coupon": (0.45, ["plan", "food"]),
             "Download invoices": (0.40, ["help"]), "Rate a dish": (0.35, ["help"])}


def sorts(seed, people=20):
    """One pile name per card, for each participant."""
    out = []
    rng = random.Random(seed)
    for _ in range(people):
        piles = dict(zip(CARDS, LATENT))
        for card, (chance, elsewhere) in UNSETTLED.items():
            if rng.random() < chance:
                piles[card] = rng.choice(elsewhere)
        if rng.random() < 0.30:  # this participant keeps one pile for both
            piles = {c: ("plan" if g == "delivery" else g) for c, g in piles.items()}
        out.append(piles)
    return out


def similarity(piles):
    """The share of participants who put each pair of cards in one pile."""
    return [[sum(p[a] == p[b] for p in piles) / len(piles) for b in CARDS] for a in CARDS]


def average_linkage(sim):
    """Join the two closest groups until one is left; distance = 1 - similarity."""
    groups, merges = [[i] for i in range(len(CARDS))], []
    while len(groups) > 1:
        height, x, y = min(((sum(1 - sim[i][j] for i in a for j in b) / (len(a) * len(b)), p, q)
                            for p, a in enumerate(groups) for q, b in enumerate(groups) if p < q),
                           key=lambda t: (round(t[0], 9), t[1], t[2]))
        joined = sorted(groups[x] + groups[y])
        groups = [g for k, g in enumerate(groups) if k not in (x, y)] + [joined]
        merges.append((height, joined))
    return merges


def cut(merges, height):
    """The groups left when every join up to this height is kept."""
    groups = [[i] for i in range(len(CARDS))]
    for at, joined in merges:
        if at <= height + 1e-9:
            groups = [g for g in groups if not set(g) & set(joined)] + [joined]
    return sorted(groups)


def report(seed=194):
    sim = similarity(sorts(seed))
    print("Cards: " + ", ".join(f"C{i + 1} {c}" for i, c in enumerate(CARDS)))
    print("\nSimilarity matrix, first five rows (share of 20 participants):")
    print("      " + " ".join(f" C{i + 1:<2}" for i in range(len(CARDS))))
    for i in range(5):
        print(f"C{i + 1:<4} " + " ".join(f"{sim[i][j]:.2f}" for j in range(len(CARDS))))
    print()
    for a, b in [("Pause plan", "Cancel plan"), ("Change address", "Change delivery window"),
                 ("Apply a coupon", "Update payment card"), ("Apply a coupon", "Change meal preferences"),
                 ("Download invoices", "Contact support")]:
        print(f"{a} + {b}: {sim[CARDS.index(a)][CARDS.index(b)]:.2f}")
    merges = average_linkage(sim)
    print("\nJoin heights: " + ", ".join(f"{h:.2f}" for h, _ in merges))
    for height in (0.00, 0.50, 0.90):
        groups = cut(merges, height)
        print(f"\nCut at distance {height:.2f} ({1 - height:.2f} similarity): {len(groups)} groups")
        for g in groups:
            print("  " + " | ".join(CARDS[i] for i in g))
    return len(merges)


_ = report()

Cards: C1 Pause plan, C2 Cancel plan, C3 Skip one day, C4 Change address, C5 Change delivery window, C6 Update payment card, C7 Download invoices, C8 Apply a coupon, C9 Change meal preferences, C10 Rate a dish, C11 Report a missing box, C12 Contact support

Similarity matrix, first five rows (share of 20 participants):
       C1   C2   C3   C4   C5   C6   C7   C8   C9   C10  C11  C12
C1    1.00 0.75 0.50 0.20 0.20 0.00 0.00 0.20 0.00 0.00 0.00 0.00
C2    0.75 1.00 0.40 0.15 0.15 0.00 0.05 0.20 0.00 0.05 0.25 0.25
C3    0.50 0.40 1.00 0.70 0.70 0.00 0.00 0.15 0.00 0.00 0.00 0.00
C4    0.20 0.15 0.70 1.00 1.00 0.00 0.00 0.05 0.00 0.00 0.00 0.00
C5    0.20 0.15 0.70 1.00 1.00 0.00 0.00 0.05 0.00 0.00 0.00 0.00

Pause plan + Cancel plan: 0.75
Change address + Change delivery window: 1.00
Apply a coupon + Update payment card: 0.60
Apply a coupon + Change meal preferences: 0.20
Download invoices + Contact support: 0.40

Join heights: 0.00, 0.00, 0.25, 0.30, 0.30, 0.40, 0.50, 0.73, 0.85, 0.89, 0.96

Cut at distance 0.00 (1.00 similarity): 10 groups
  Pause plan
  Cancel plan
  Skip one day
  Change address | Change delivery window
  Update payment card
  Download invoices
  Apply a coupon
  Change meal preferences
  Rate a dish
  Report a missing box | Contact support

Cut at distance 0.50 (0.50 similarity): 5 groups
  Pause plan | Cancel plan
  Skip one day | Change address | Change delivery window
  Update payment card | Download invoices | Apply a coupon
  Change meal preferences | Rate a dish
  Report a missing box | Contact support

Cut at distance 0.90 (0.10 similarity): 2 groups
  Pause plan | Cancel plan | Skip one day | Change address | Change delivery window
  Update payment card | Download invoices | Apply a coupon | Change meal preferences | Rate a dish | Report a missing box | Contact support

Four pairs from that output say most of what the sort found. Change address with Change delivery window is 1.00, so nobody disagreed. Pause plan with Cancel plan is 0.75. Apply a coupon sits at 0.60 with Update payment card and 0.20 with Change meal preferences, so it is a money card for most people but not for all. Download invoices with Contact support is 0.40, which is the sort telling you that invoices are a help topic for two people in five.

Hierarchical clusteringhierarchical clustering: Building a tree of groups by repeatedly joining the two closest groups; average linkage measures closeness as the mean distance over every pair of their members. turns those numbers into a tree. Average linkage starts with each card alone, repeatedly joins the two groups whose average distance is smallest, and records the height at which each join happened; distance here is simply 1 minus similarity. Standard-library Python does all of that for twelve cards, with no package to install, and the heights it recorded are the output’s Join heights line. The tree it draws is a dendrogramdendrogram: The tree hierarchical clustering produces, with each join drawn at the distance at which it happened, so cutting it at a chosen height gives a set of groups., and a dendrogram on its own proposes nothing: you have to cut it at a height, and every height gives a different answer.

Start from the twelve leaves, step down the three cuts the program reports and watch the group count fall:

The five-group cut is the one worth writing into a menu, and notice what it claims: Skip one day belongs with the two delivery cards rather than with Pause plan and Cancel plan. In the matrix, Skip one day sits at 0.70 with Change address and 0.50 with Pause plan, so that claim rests on four participants. The program has never seen a label, a task or a screen.

A tree test checks findability before anything is designed

A tree testtree test: A study that evaluates a category structure by asking participants to find where given things would live in it, shown as bare expanding lists with no visual design or page content. is the cheapest way to check that hypothesis. Participants see the categories as a bare set of expanding lists, with no visual design, no page content and no search box, and they are given tasks: find where you would look for this. They click down, can climb back out, and finish by choosing one item. Tree testing therefore runs off a spreadsheet. Nothing has to be designed or written first, so a tree can be tested the day after a card sort, and a rejected tree costs an afternoon rather than a sprint.

Four numbers come out of it per task. Task successtask success: In a tree test, the share of participants who finished a task on an item that answers it; reported per task rather than as one figure for the study. is the share of participants who ended on an item that answers the task. Directnessdirectness: In a tree test, the share of participants who reached their answer without climbing back out of a branch they had opened, read as a sign of how confident the path felt. is the share who never climbed back out of a branch they had opened, which Nielsen Norman Group describes as a proxy for how confident people were in the path they took. The first clickfirst click: The branch or link a participant opens first in a task, counted across participants to show where a structure sends people before anything else happens. is the branch each participant opened first, counted across participants, and it tells you where the structure sent them before anything else happened. Time on task is the fourth, and it needs a tool’s clock rather than a click log.

The program below scores an eight-task tree test of the Ward Desk menu from a seeded log of forty participants, and prints the two weakest tasks with their first-click split. Press Run, then press Edit and move Birth certificate’s weight towards Pay and apply in TASKS to see a task recover:

"""A tree test of the Ward Desk menu: eight tasks, forty synthetic participants.

The log is drawn from a seeded model of where people click first and how often
they climb back out of a branch. Success is the share who ended on the right
leaf; directness is the share who never left a branch they had opened. Every
category, task and number below is invented.
"""

import random
from collections import Counter

TREE = {
    "Report a problem": ["Street light out", "Pothole", "Rubbish not collected", "Water leak"],
    "Pay and apply": ["Property tax", "Water bill", "Trade licence", "Birth certificate"],
    "Your ward": ["Ward officer", "Ward meetings", "Ward boundaries"],
    "About the council": ["Departments", "Budget", "Tenders"],
    "Help": ["Contact us", "Track a complaint", "Office hours"],
}
TOP_OF = {leaf: top for top, leaves in TREE.items() for leaf in leaves}
# task: the leaf that answers it, and how often each branch is opened first
TASKS = {
    "street light out": ("Street light out", {"Report a problem": 36, "Help": 3, "Your ward": 1}),
    "pay the property tax": ("Property tax", {"Pay and apply": 33, "About the council": 4, "Help": 3}),
    "chase a complaint": ("Track a complaint", {"Report a problem": 25, "Help": 9, "Your ward": 6}),
    "get a birth certificate": ("Birth certificate", {"About the council": 17, "Help": 12,
                                                      "Pay and apply": 8, "Your ward": 3}),
    "name the ward officer": ("Ward officer", {"Your ward": 31, "About the council": 6, "Help": 3}),
    "rubbish not collected": ("Rubbish not collected", {"Report a problem": 34, "Help": 4, "Your ward": 2}),
    "last year's spending": ("Budget", {"About the council": 28, "Your ward": 7, "Help": 5}),
    "office opening time": ("Office hours", {"Help": 30, "About the council": 7, "Your ward": 3}),
}


def branch(rng, weights):
    tops = list(weights)
    return rng.choices(tops, weights=[weights[t] for t in tops])[0]


def session(rng, leaf, weights):
    """One participant: the branches they opened, and the leaf they settled on."""
    path = [branch(rng, weights)]
    while path[-1] != TOP_OF[leaf] and len(path) < 3 and rng.random() < 0.45:
        path.append(TOP_OF[leaf] if rng.random() < 0.55 else branch(rng, weights))
    inside = TREE[path[-1]]
    chosen = leaf if path[-1] == TOP_OF[leaf] and rng.random() < 0.92 else rng.choice(inside)
    return path, chosen


def log(seed=31, people=40):
    rng = random.Random(seed)
    rows = []
    for task, (leaf, weights) in TASKS.items():
        for _ in range(people):
            path, chosen = session(rng, leaf, weights)
            rows.append({"task": task, "path": path, "chose": chosen, "right": leaf})
    return rows


def score(rows):
    out = {}
    for task in TASKS:
        mine = [r for r in rows if r["task"] == task]
        out[task] = {
            "success": sum(r["chose"] == r["right"] for r in mine) / len(mine),
            "directness": sum(len(r["path"]) == 1 for r in mine) / len(mine),
            "first": Counter(r["path"][0] for r in mine),
        }
    return out


def report():
    rows = log()
    scores = score(rows)
    print(f"{len(rows)} task attempts, {len(rows) // len(TASKS)} participants, {len(TASKS)} tasks")
    print(f"\n{'Task':<24}{'Success':>9}{'Direct':>8}  First branch most often")
    for task, s in scores.items():
        top, hits = s["first"].most_common(1)[0]
        print(f"{task:<24}{s['success']:>8.0%}{s['directness']:>8.0%}  {top} ({hits} of 40)")
    weakest = sorted(scores, key=lambda t: (scores[t]["success"], scores[t]["directness"]))[:2]
    print("\nThe two weakest tasks:")
    for task in weakest:
        s = scores[task]
        print(f"\n  {task}: answer {TASKS[task][0]!r} under {TOP_OF[TASKS[task][0]]!r}")
        print(f"  success {s['success']:.0%}, directness {s['directness']:.0%}; first branch:")
        for top, hits in s["first"].most_common():
            print(f"    {top:<20}{hits:>3} of 40 ({hits / 40:.0%})")
    return weakest


_ = report()

320 task attempts, 40 participants, 8 tasks

Task                      Success  Direct  First branch most often
street light out             88%    100%  Report a problem (36 of 40)
pay the property tax         95%     80%  Pay and apply (32 of 40)
chase a complaint            48%     70%  Report a problem (23 of 40)
get a birth certificate      40%     68%  About the council (17 of 40)
name the ward officer        85%     95%  Your ward (33 of 40)
rubbish not collected        90%    100%  Report a problem (38 of 40)
last year's spending         75%     80%  About the council (25 of 40)
office opening time          80%     88%  Help (30 of 40)

The two weakest tasks:

  get a birth certificate: answer 'Birth certificate' under 'Pay and apply'
  success 40%, directness 68%; first branch:
    About the council    17 of 40 (42%)
    Help                 14 of 40 (35%)
    Pay and apply         8 of 40 (20%)
    Your ward             1 of 40 (2%)

  chase a complaint: answer 'Track a complaint' under 'Help'
  success 48%, directness 70%; first branch:
    Report a problem     23 of 40 (57%)
    Help                 12 of 40 (30%)
    Your ward             5 of 40 (12%)

Six of the eight tasks are healthy: reporting a street light succeeded for 88 % of participants with every one of them going straight there, and paying the property tax reached 95 %. Two are not, and the output names them.

One more limit is worth knowing before you read any tool’s report: most tree-testing software accepts only a bottom-level item as a correct answer, never a category. Real sites often give a category its own landing page, and a task answered by that page cannot be marked right, so a sensible tree can score badly for a reason that has nothing to do with people.

Two weak tasks, and the change each one asks for

Success alone tells you that a task failed. Success read against directness tells you what kind of failure it was, and both weak Ward Desk tasks come out the same way on that reading:

Directness well above success is the clearer of the two diagnoses: most people did not hunt, so the structure is not so much confusing them as misleading them. The first clicks say where. For the birth certificate, 17 of 40 opened About the council and 14 opened Help, while only 8 opened Pay and apply, where the item actually sits. For chasing a complaint, 23 of 40 opened Report a problem, which is where you report a new one, and 12 opened Help, where the tracker lives.

The first clicks are where the two tasks part company, and they point at two different fixes. A card most people look for somewhere else is a placement problem: a certificate is something you apply for, so either the item moves to where people look, or the branch that attracts them gets the item too, which is what a tree test is for — the report tells you whether anyone would use the second home before you build it. A branch that people read as covering two jobs is a labelling problem, and the words come from people rather than from the organisation. The GOV.UK Service Manual makes this concrete for service names: use the words users use, describe a task rather than a technology, and prefer a verb. Renaming Help to Track a complaint or anything else is itself a change to tree-test, and the Service Manual names tree testing as a way to check whether a new name lets people navigate to the service at all.

Two rules keep this from becoming guesswork. Change one thing per round, because a rename and a move in the same test leave you unable to say which one moved the number. And keep the tasks identical between rounds, so the before-and-after numbers compare.

Structure is also an accessibility matter: the W3C’s Web Accessibility Initiative asks for navigation that is named, styled and positioned consistently across pages, for more than one route to content such as a search or a sitemap, and for orientation cues such as breadcrumbs. A tree test checks the first of those directly, because it strips everything else away. The reason any of it works at all is older than the method: Peter Pirolli and Stuart Card’s information-foraging account describes people judging the cues a link offers before they spend a click on it, which is exactly the judgement a first click records.

Practise: read a sort and a tree test for MealBox

The exercise hands you a second pair of invented datasets, both from MealBox’s account area: a similarity table for twelve cards from twenty-four participants, stored one way round as a tool exports it, and a log of six tasks run by twenty people against the account menu. Work out five pair similarities, the two tasks with the lowest success and the directness of each:

"""Read a MealBox card sort and tree test, then fill in the three answers at the end.

Twenty-four synthetic participants sorted twelve cards of MealBox's account
area; twenty more ran six tasks against the proposed account menu. All data is
invented for this exercise.

SIMILARITY holds each pair once, in the order the cards appear in CARDS, the
way a tool exports it: look a pair up in both orders. A session string is the
branches the participant opened, then the leaf they settled on, so
"Deliveries>Billing>Refunds" means they opened Deliveries, climbed back out,
opened Billing and chose Refunds.
"""

CARDS = [
    "Gift a box", "Refer a friend", "Change password", "Delete account",
    "Export my data", "Notification settings", "Add a second address",
    "Switch to weekly", "See past boxes", "Reorder a past box",
    "Claim a refund", "Report a late box",
]

SIMILARITY = {
    'Gift a box': {
        'Refer a friend': 1.00, 'Change password': 0.00, 'Delete account': 0.00, 'Export my data': 0.00,
        'Notification settings': 0.00, 'Add a second address': 0.00, 'Switch to weekly': 0.00,
        'See past boxes': 0.00, 'Reorder a past box': 0.00, 'Claim a refund': 0.00, 'Report a late box': 0.00
    },
    'Refer a friend': {
        'Change password': 0.00, 'Delete account': 0.00, 'Export my data': 0.00,
        'Notification settings': 0.00, 'Add a second address': 0.00, 'Switch to weekly': 0.00,
        'See past boxes': 0.00, 'Reorder a past box': 0.00, 'Claim a refund': 0.00, 'Report a late box': 0.00
    },
    'Change password': {
        'Delete account': 0.75, 'Export my data': 0.62, 'Notification settings': 1.00,
        'Add a second address': 0.33, 'Switch to weekly': 0.00, 'See past boxes': 0.00,
        'Reorder a past box': 0.00, 'Claim a refund': 0.00, 'Report a late box': 0.00
    },
    'Delete account': {
        'Export my data': 0.54, 'Notification settings': 0.75, 'Add a second address': 0.25,
        'Switch to weekly': 0.25, 'See past boxes': 0.00, 'Reorder a past box': 0.17, 'Claim a refund': 0.00,
        'Report a late box': 0.00
    },
    'Export my data': {
        'Notification settings': 0.62, 'Add a second address': 0.21, 'Switch to weekly': 0.00,
        'See past boxes': 0.38, 'Reorder a past box': 0.17, 'Claim a refund': 0.00, 'Report a late box': 0.00
    },
    'Notification settings': {
        'Add a second address': 0.33, 'Switch to weekly': 0.00, 'See past boxes': 0.00,
        'Reorder a past box': 0.00, 'Claim a refund': 0.00, 'Report a late box': 0.00
    },
    'Add a second address': {
        'Switch to weekly': 0.00, 'See past boxes': 0.00, 'Reorder a past box': 0.00, 'Claim a refund': 0.33,
        'Report a late box': 0.67
    },
    'Switch to weekly': {
        'See past boxes': 0.00, 'Reorder a past box': 0.67, 'Claim a refund': 0.00, 'Report a late box': 0.00
    },
    'See past boxes': {
        'Reorder a past box': 0.33, 'Claim a refund': 0.00, 'Report a late box': 0.00
    },
    'Reorder a past box': {
        'Claim a refund': 0.00, 'Report a late box': 0.00
    },
    'Claim a refund': {
        'Report a late box': 0.54
    },
    'Report a late box': {},
}

TREE_TEST = [
    {
        "task": 'switch to a weekly box',
        "answer": 'Switch to weekly',
        "sessions": [
            'Plan>Switch to weekly',
            'Plan>Cancel',
            'Plan>Switch to weekly',
            'Plan>Switch to weekly',
            'Plan>Switch to weekly',
            'Plan>Switch to weekly',
            'Account>Plan>Switch to weekly',
            'Plan>Switch to weekly',
            'Plan>Switch to weekly',
            'Plan>Switch to weekly',
            'Plan>Switch to weekly',
            'Plan>Switch to weekly',
            'Deliveries>Deliveries>Plan>Switch to weekly',
            'Plan>Switch to weekly',
            'Plan>Switch to weekly',
            'Deliveries>Plan>Switch to weekly',
            'Plan>Pause',
            'Plan>Switch to weekly',
            'Plan>Switch to weekly',
            'Plan>Switch to weekly',
        ],
    },
    {
        "task": 'get money back for a box that never came',
        "answer": 'Refunds',
        "sessions": [
            'Deliveries>Delivery window',
            'Billing>Refunds',
            'Deliveries>Billing>Refunds',
            'Deliveries>Billing>Refunds',
            'Deliveries>Delivery window',
            'Deliveries>Deliveries>Delivery window',
            'Deliveries>Billing>Refunds',
            'Deliveries>Delivery window',
            'Account>Deliveries>Account>Delete account',
            'Deliveries>Addresses',
            'Deliveries>Billing>Refunds',
            'Deliveries>Delivery window',
            'Deliveries>Deliveries>Billing>Refunds',
            'Billing>Refunds',
            'Deliveries>Deliveries>Addresses',
            'Deliveries>Addresses',
            'Billing>Refunds',
            'Deliveries>Billing>Refunds',
            'Deliveries>Late or missing box',
            'Account>Export data',
        ],
    },
    {
        "task": 'stop the emails',
        "answer": 'Notifications',
        "sessions": [
            'Deliveries>Delivery window',
            'Account>Notifications',
            'Account>Notifications',
            'Account>Notifications',
            'Account>Notifications',
            'Account>Notifications',
            'Account>Notifications',
            'Plan>Plan>Cancel',
            'Account>Notifications',
            'Account>Notifications',
            'Plan>Account>Delete account',
            'Account>Notifications',
            'Account>Notifications',
            'Account>Notifications',
            'Account>Notifications',
            'Account>Notifications',
            'Account>Notifications',
            'Account>Notifications',
            'Account>Notifications',
            'Plan>Pause',
        ],
    },
    {
        "task": "download last month's invoice",
        "answer": 'Invoices',
        "sessions": [
            'Plan>Cancel',
            'Plan>Billing>Invoices',
            'Billing>Invoices',
            'Billing>Invoices',
            'Billing>Refunds',
            'Billing>Invoices',
            'Billing>Invoices',
            'Billing>Invoices',
            'Billing>Invoices',
            'Billing>Invoices',
            'Billing>Invoices',
            'Billing>Invoices',
            'Billing>Invoices',
            'Billing>Invoices',
            'Billing>Invoices',
            'Billing>Invoices',
            'Billing>Invoices',
            'Billing>Invoices',
            'Billing>Invoices',
            'Billing>Invoices',
        ],
    },
    {
        "task": 'give a box to a friend as a present',
        "answer": 'Gift a box',
        "sessions": [
            'Plan>Rewards>Gift a box',
            'Plan>Plan>Pause',
            'Account>Rewards>Gift a box',
            'Plan>Cancel',
            'Plan>Deliveries>Addresses',
            'Deliveries>Plan>Deliveries>Addresses',
            'Plan>Cancel',
            'Plan>Rewards>Gift a box',
            'Account>Rewards>Gift a box',
            'Plan>Pause',
            'Plan>Plan>Switch to weekly',
            'Plan>Plan>Rewards>Gift a box',
            'Deliveries>Rewards>Gift a box',
            'Plan>Plan>Pause',
            'Deliveries>Late or missing box',
            'Rewards>Gift a box',
            'Plan>Deliveries>Addresses',
            'Plan>Deliveries>Rewards>Gift a box',
            'Rewards>Gift a box',
            'Deliveries>Plan>Cancel',
        ],
    },
    {
        "task": 'change the day the box arrives',
        "answer": 'Delivery window',
        "sessions": [
            'Account>Export data',
            'Deliveries>Delivery window',
            'Deliveries>Delivery window',
            'Deliveries>Delivery window',
            'Deliveries>Delivery window',
            'Deliveries>Delivery window',
            'Deliveries>Delivery window',
            'Deliveries>Delivery window',
            'Deliveries>Delivery window',
            'Account>Export data',
            'Deliveries>Delivery window',
            'Deliveries>Delivery window',
            'Plan>Cancel',
            'Plan>Deliveries>Delivery window',
            'Deliveries>Delivery window',
            'Deliveries>Delivery window',
            'Deliveries>Delivery window',
            'Deliveries>Delivery window',
            'Account>Export data',
            'Deliveries>Delivery window',
        ],
    },
]

PAIRS = [
    ("Export my data", "Delete account"),
    ("Add a second address", "Report a late box"),
    ("Switch to weekly", "Reorder a past box"),
    ("Export my data", "See past boxes"),
    ("Notification settings", "Add a second address"),
]

# ---- your answers ----------------------------------------------------------

# 1. For each pair in PAIRS, the share of the 24 participants who put the two
#    cards in one pile, rounded to two decimals: {("card a", "card b"): 0.42}.
PAIR_SIMILARITY = {}

# 2. The two tasks of TREE_TEST with the lowest success, as a list of their
#    task names in any order.
WORST_TASKS = []

# 3. For each of those two tasks, the share of participants who never left the
#    branch they opened first: {"task name": 0.42}.
DIRECTNESS = {}

from analyse import CARDS, DIRECTNESS, PAIR_SIMILARITY, PAIRS, SIMILARITY, TREE_TEST, WORST_TASKS

TOLERANCE = 0.01


def stored(a, b):
    """The similarity of a pair, whichever way round it was asked for."""
    if b in SIMILARITY.get(a, {}):
        return SIMILARITY[a][b]
    return SIMILARITY[b][a]


def success(task):
    hits = sum(s.split(">")[-1] == task["answer"] for s in task["sessions"])
    return hits / len(task["sessions"])


def directness(task):
    straight = sum(len(s.split(">")) == 2 for s in task["sessions"])
    return straight / len(task["sessions"])


def test_every_pair_is_answered_with_its_similarity():
    """the five pairs of PAIRS carry the share of participants who grouped them"""
    assert set(PAIR_SIMILARITY) == set(PAIRS), "answer every pair of PAIRS, and only those pairs"
    for pair in PAIRS:
        for card in pair:
            assert card in CARDS, f"{card!r} is not one of the twelve cards"
        got, want = PAIR_SIMILARITY[pair], stored(*pair)
        assert isinstance(got, float), f"{pair}: give a share such as 0.42, not {got!r}"
        assert abs(got - want) <= TOLERANCE, f"{pair}: {got} is not within 0.01 of the table"


def test_the_two_weakest_tasks_are_named():
    """WORST_TASKS holds the two task names with the lowest success"""
    names = [t["task"] for t in TREE_TEST]
    assert len(WORST_TASKS) == 2, "name exactly two tasks"
    for name in WORST_TASKS:
        assert name in names, f"{name!r} is not one of the six tasks"
    ranked = sorted(TREE_TEST, key=success)
    assert set(WORST_TASKS) == {ranked[0]["task"], ranked[1]["task"]}, (
        "these are not the two tasks with the lowest share of participants on the right leaf"
    )


def test_directness_is_given_for_those_two_tasks():
    """DIRECTNESS holds the share who never left the branch they opened first"""
    assert set(DIRECTNESS) == set(WORST_TASKS), "give the directness of the two tasks you named"
    by_name = {t["task"]: t for t in TREE_TEST}
    for name, got in DIRECTNESS.items():
        want = directness(by_name[name])
        assert isinstance(got, float), f"{name!r}: give a share such as 0.42, not {got!r}"
        assert abs(got - want) <= TOLERANCE, f"{name!r}: {got} is not within 0.01 of the log"


def test_success_and_directness_disagree_on_one_task():
    """the weaker pair differ: one task was hunted for, the other walked into confidently"""
    picked = sorted(DIRECTNESS.values())
    assert len(picked) == 2 and picked[1] - picked[0] >= 0.10, (
        "on this log the two weakest tasks differ in directness by 0.10 or more; check both shares"
    )

def stored(a, b):
    if b in SIMILARITY.get(a, {}):
        return SIMILARITY[a][b]
    return SIMILARITY[b][a]

Once its tests pass, the self-check below starts with whichever idea you have answered least well:

Interview questions

Warm-up (fresher to 2–5 years): when would you reach for an open sort, and when for a closed one?

An open sort when you do not yet have a structure, or when you need the words people use for these groups, because only an open sort collects pile names in their own vocabulary and only an open sort can surface a category you never thought of. A closed sort when a structure exists, named end to end, and the question is narrower: do cards land where we put them, and which cards do not. Two practical notes make the answer stronger. A closed sort is much faster to analyse, since the counts fall straight out, while an open sort needs the pile names people wrote coded first, and that coding is the work. And a closed sort leads people, because every card has to go somewhere, so a category that should not exist still fills up. A hybrid sits between them: your categories, plus room for new ones, which is a reasonable default when a structure exists but you suspect a gap in it.

The two questions below go further: one on sample size when a stakeholder quotes the wrong rule of thumb, the other on a tree test with two tasks that failed. Sketch the first two sentences of your own answer to each, then open the question to compare it with the first step given there:

Key takeaways

• An open sort generates groupings and vocabulary you did not have, a closed sort only reports how well your own categories absorb the cards, and a hybrid sort trades some of the first for much less coding work.
• Five participants settle a usability test but not a sort: the correlation with a full result is 0.75 at five participants, 0.90 at fifteen and 0.95 at thirty, so fifteen is the usual stopping point.
• The similarity matrix holds one number per pair of cards, the share of participants who grouped them, and it is the only quantitative thing a sort produces.
• Average-linkage clustering turns that matrix into a tree, and every height you cut it at gives a different structure, so a cut is a hypothesis to test rather than an answer.
• A tree test scores that hypothesis on bare categories with no visual design, and success read together with directness and the first clicks tells you whether to move an item, rename a branch, or list the item twice.