Framing problems and choosing what to solve

MealBox’s quarterly workshop ended with 12 ideas on a board, and the CEO wants 3 of them shipped before the quarter closes. Prioritisation now has to turn into a conversation about numbers nobody has measured yet, and about the risk of spending a quarter on the wrong three. Four tools help, in the order a team usually needs them: write the problem worth solving, map the beliefs inside it, score what is left, and name the signals that say whether a shipped answer worked.

Six invented MealBox opportunities thread through: a pause-flow, a served-area check, a skip-day control, a meal swap, a referral and a nutrition label. You will write problem statements from a few insights, place assumptions on an importance-by-evidence grid, score the opportunities with RICE and watch one halved estimate reorder the top of the list, and finish with a HEART worksheet that turns a goal into measurable signals.

A problem statement names three things, in one sentence

A problem statementproblem statement: A one-sentence summary of a problem to solve, naming the user, their need and the evidence that the need is real; written before any solution is chosen. is a short sentence with three parts: the user, their need and the evidence behind it. Each of the three is a test an idea has to pass before any quarter is spent on it, so a statement missing one of them is unfinished.

Take MealBox’s pause-flow. “People cannot pause a subscription” is a complaint, not a statement. “New subscribers who travel in week one need a way to skip a delivery, so that a short trip does not cost them a plan (eight of eight cancellers interviewed mentioned a trip)” is the finished shape. The user is named (new subscribers who travel), the need is named (skip a delivery), and the evidence is named (eight of eight cancellers interviewed). A reader can disagree about any one of the three, and that is exactly what makes the sentence useful.

A how might wehow might we: A short, open question in the form 'How might we...' that frames a design task without naming a particular solution, so several answers remain possible. question opens the problem into answers without smuggling one of them in. Its three words carry the frame: “how” says we do not know yet, “might” says we are not committing, “we” says a team will go after it. “How might we let subscribers skip a week they will be away?” leaves room for ten answers; “how might we add a Pause button to the plan screen?” leaves room for one, and the button is already chosen before any research is read.

Three insights from invented interviews, with the problem statement each one supports and a HMW that still has room:

Map your assumptions before you score anything

Every problem statement rests on beliefs the team has not tested yet. Assumption mappingassumption mapping: A grid that places beliefs by how important they are to a plan and how strong the evidence for them is, used to decide which ones to test first., a common workshop technique, places those beliefs on two axes: how important the belief is to the plan (vertical) and how strong the evidence for it is (horizontal). The four quadrants decide what the team does next with each belief.

Top-left (high importance, weak evidence) is the quadrant to test first, because a plan rests on it and no measurement supports it yet. Top-right (high importance, strong evidence) is known risk: build on it, with a check in beta that the pattern still holds. Bottom-right (low importance, strong evidence) is given: no test earns its cost. Bottom-left (low importance, weak evidence) is ignore-for-now: tests go to top-left first, and come back only when a different plan surfaces.

Step through four invented MealBox beliefs behind the pause-flow plan, each landing in the quadrant its importance and evidence put it:

The payoff of the grid is in the top-left square. On this plan the top-left belief is “a visible pause control reaches the subscribers who would otherwise cancel”, and it decides whether the whole quarter’s work is worth doing. A five-user prototype test of a pause control on the MealBox plan page settles the question in two days; without the test the team could sketch, build and ship a flow that nobody meets at the moment they would have left.

The GOV.UK Service Manual asks for the same discipline when it tells teams that an opinion or suggestion which does not come from users is an assumption, one that has “to be proven by doing research” (Learning about users and their needs, Service Manual). The grid is one way to pick which to test first.

Prioritise with impact-effort, RICE and Kano, honestly

Once the top-left beliefs are tested, several opportunities are still competing for the quarter. Four tools help a team compare them, each answering a different question. The table below walks the four against the questions a team usually has:

An impact-effort grid works for a workshop: two estimates per item, in an afternoon, and the top-right corner wins. It is cheap and honest about being cheap. RICERICE score: Intercom's opportunity score for prioritisation: reach x impact x confidence / effort, with each input written as an estimate. (reach × impact × confidence / effort), the Intercom formula, pays for its extra inputs by comparing opportunities on one score, which is what a quarter-ahead decision usually asks for. The six MealBox opportunities below are scored on invented reach, impact, confidence and effort estimates, so that the ranks and the shift when one estimate is halved come from the formula alone:

"""
MealBox opportunities: RICE scores and a HEART worksheet.

Six invented MealBox opportunities from a workshop are scored with RICE
(reach x impact x confidence / effort), the Intercom formula. One reach
estimate is then halved to show how a small change in a number nobody
has measured yet reorders the top of the list. A HEART worksheet
(Rodden, Hutchinson and Fu, CHI 2010) for the top opportunity closes
the program, by turning one goal into signals and measurable metrics.

Numbers invented for this lesson.
"""


def rice(reach: float, impact: float, confidence: float, effort: float) -> float:
    """RICE = Reach x Impact x Confidence / Effort (Intercom)."""
    return reach * impact * confidence / effort


opportunities = {
    "pause-flow": {
        "label": "Pause a subscription for a week",
        "reach": 4000, "impact": 2.0, "confidence": 0.8, "effort": 2.0,
    },
    "served-area-check": {
        "label": "Check served area before plan choice",
        "reach": 5000, "impact": 1.0, "confidence": 0.5, "effort": 1.0,
    },
    "skip-day": {
        "label": "Skip one delivery day",
        "reach": 3000, "impact": 1.0, "confidence": 0.7, "effort": 1.0,
    },
    "meal-swap": {
        "label": "Swap one meal for an alternative",
        "reach": 2500, "impact": 0.5, "confidence": 0.6, "effort": 0.5,
    },
    "nutrition-info": {
        "label": "Show nutrition info beside each meal",
        "reach": 3000, "impact": 0.5, "confidence": 0.3, "effort": 1.0,
    },
    "referral": {
        "label": "Refer a friend, both get a week off",
        "reach": 1500, "impact": 1.0, "confidence": 0.4, "effort": 1.5,
    },
}


def rank(opps: dict) -> list:
    scored = [
        (k, round(rice(v["reach"], v["impact"], v["confidence"], v["effort"])))
        for k, v in opps.items()
    ]
    return sorted(scored, key=lambda p: p[1], reverse=True)


def print_table(name: str, order: list, opps: dict) -> None:
    print(f"=== {name} ===")
    print(f"{'rank':<6}{'opportunity':<20}{'RICE':>6}  reach  impact  confidence  effort")
    for i, (k, score) in enumerate(order, start=1):
        o = opps[k]
        print(
            f"{i:<6}{k:<20}{score:>6}  {o['reach']:>5}  {o['impact']:>6}"
            f"  {o['confidence']:>10}  {o['effort']:>6}"
        )
    print()


def main() -> None:
    print("MealBox opportunities: RICE scores\n")
    original = rank(opportunities)
    print_table("Original estimates", original, opportunities)

    print("Sensitivity: halve one reach estimate")
    print("  pause-flow reach 4000 -> 2000 (what if only half the subscribers see it?)\n")
    adjusted = {k: dict(v) for k, v in opportunities.items()}
    adjusted["pause-flow"]["reach"] = 2000
    sensitised = rank(adjusted)
    print_table("Adjusted estimates", sensitised, adjusted)

    before = {k: i + 1 for i, (k, _) in enumerate(original)}
    after = {k: i + 1 for i, (k, _) in enumerate(sensitised)}
    moved = [k for k in before if before[k] != after[k]]
    print("Rank changes after halving pause-flow reach:")
    for k in moved:
        print(f"  {k}: rank {before[k]} -> {after[k]}")
    print()

    print("HEART worksheet for the top opportunity (pause-flow)\n")
    heart = [
        ("Retention", "subscribers keep their plan through busy weeks",
         "pauses taken instead of cancellations",
         "90-day retention of pausers vs non-pausers"),
        ("Engagement", "the pause flow fits how people travel",
         "pauses taken by active subscribers",
         "pause-use rate: pauses per subscription per quarter"),
        ("Task success", "a pause is reachable and clear",
         "people who find and complete a pause",
         "task success rate on the first-time pause task"),
    ]
    print(f"{'HEART':<14}{'goal':<48}{'signal':<40}metric")
    for cat, goal, signal, metric in heart:
        print(f"{cat:<14}{goal:<48}{signal:<40}{metric}")

    print("\nAccessibility and consumer-protection rules are constraints, not features to score.")


if __name__ == "__main__":
    _ = main()

MealBox opportunities: RICE scores

=== Original estimates ===
rank  opportunity           RICE  reach  impact  confidence  effort
1     pause-flow            3200   4000     2.0         0.8     2.0
2     served-area-check     2500   5000     1.0         0.5     1.0
3     skip-day              2100   3000     1.0         0.7     1.0
4     meal-swap             1500   2500     0.5         0.6     0.5
5     nutrition-info         450   3000     0.5         0.3     1.0
6     referral               400   1500     1.0         0.4     1.5

Sensitivity: halve one reach estimate
  pause-flow reach 4000 -> 2000 (what if only half the subscribers see it?)

=== Adjusted estimates ===
rank  opportunity           RICE  reach  impact  confidence  effort
1     served-area-check     2500   5000     1.0         0.5     1.0
2     skip-day              2100   3000     1.0         0.7     1.0
3     pause-flow            1600   2000     2.0         0.8     2.0
4     meal-swap             1500   2500     0.5         0.6     0.5
5     nutrition-info         450   3000     0.5         0.3     1.0
6     referral               400   1500     1.0         0.4     1.5

Rank changes after halving pause-flow reach:
  pause-flow: rank 1 -> 3
  served-area-check: rank 2 -> 1
  skip-day: rank 3 -> 2

HEART worksheet for the top opportunity (pause-flow)

HEART         goal                                            signal                                  metric
Retention     subscribers keep their plan through busy weeks  pauses taken instead of cancellations   90-day retention of pausers vs non-pausers
Engagement    the pause flow fits how people travel           pauses taken by active subscribers      pause-use rate: pauses per subscription per quarter
Task success  a pause is reachable and clear                  people who find and complete a pause    task success rate on the first-time pause task

Accessibility and consumer-protection rules are constraints, not features to score.

The original ranking puts pause-flow first with a RICE of 3,200, served-area-check second with 2,500, and skip-day third with 2,100. Halving the reach on pause-flow — “what if only half the subscribers actually see the control?” — drops its score to 1,600, served-area-check becomes first, and pause-flow falls to third. One estimate that nobody has measured yet reorders the top three. The point of running the sensitivity is not to pick the “right” number; it is to show which items are robust to a bad guess and which are not.

The Kano modelKano model: A way of sorting product qualities into must-be, one-dimensional and attractive, from Kano and colleagues (1984), to separate qualities a single score blurs. (Kano and colleagues, Journal of the Japanese Society for Quality Control, 1984) asks a different question: which qualities users take for granted, so that missing them causes dissatisfaction while having them earns no credit (must-be qualities), which raise satisfaction in step with how well they are delivered (one-dimensional qualities), and which delight users beyond what they would ask for (attractive qualities). A served-area check is a must-be quality on a lunch service: the service either delivers to the address or it does not, and no amount of charm makes up for it. A referral programme is attractive at best. Kano separates qualities a RICE score blurs.

One rule beats all four tools: accessibility and the law are constraints, not features to score. A design that fails WCAG 2.2 or India’s dark-pattern rules is not shipped, whatever the RICE board says, so these rules go above the prioritisation table rather than inside it.

Set goals, signals and metrics with HEART

Once a change ships, the question becomes whether it moved the right numbers. The HEART frameworkHEART framework: Rodden, Hutchinson and Fu's five user-experience categories (Happiness, Engagement, Adoption, Retention, Task success), paired with Goals-Signals-Metrics to measure a design change. (Rodden, Hutchinson and Fu, CHI 2010) names five categories a design can be measured in: Happiness, Engagement, Adoption, Retention and Task success. The authors pair the categories with a discipline they call Goals-Signals-Metrics: start from the goal a design change is for, then name the user behaviour that would be a signal of progress, and only then pick the metric that counts it.

The script above printed a HEART worksheet for the pause-flow opportunity, in three rows. The team chose Retention, Engagement and Task success as the categories that fit the change:

Reading the worksheet left to right stops two common mistakes. A team that writes a metric first (“pause-use rate”) rarely stops to ask what the number is a signal of, or which goal it serves, so the number drifts away from the decision it was for. A team that writes a goal first (“subscribers keep their plan”) sometimes names a metric that is a signal of a different goal, so the measurement cannot settle anything. Walking from goal to signal to metric keeps the chain intact.

Not every HEART row is worth collecting. The CHI paper’s own advice is to skip categories that are not meaningful to the change at hand, which is why Happiness and Adoption are missing from this worksheet. A pause-flow does not move survey happiness much, and nobody adopts “pause” in the way they adopt a new product. A row that would be guessed, or a metric nobody can collect, is cut; three rows that measure real behaviour are better than five that paper over nothing.

Practise

Five invented MealBox insights, two problem statements, six HMW questions and a HEART table for one goal: write them in the exercise and watch the sample tests check the shape and the rules.

"""
Prioritisation workshop for MealBox: write problem statements, HMW questions
and a HEART table from five insights.

The insights come from eight invented MealBox interviews. Fill in
`problem_statements`, `hmw_questions` and `heart_rows` below.
"""

insights = [
    {"id": "i1",
     "text": "Travel in the first week leaves new subscribers with meals they cannot eat.",
     "participants": ["P2", "P5"]},
    {"id": "i2",
     "text": "Prices are easy to find; delivery windows are not.",
     "participants": ["P1", "P3"]},
    {"id": "i3",
     "text": "Pausing a week is buried; cancelling is one tap.",
     "participants": ["P2", "P6", "P7"]},
    {"id": "i4",
     "text": "The next-delivery date is missing from reminders.",
     "participants": ["P4", "P6"]},
    {"id": "i5",
     "text": "Elderly subscribers read menu photos before any text.",
     "participants": ["P8"]},
]

# TODO: Write 2 problem statements from the insights. Each statement names
# the user, the need and the evidence (an insight id such as "i1").
problem_statements: list[str] = [
    # "New subscribers who travel in week one need a way to skip a delivery "
    # "without cancelling the plan (i1).",
]

# TODO: Write 6 "How might we" questions that open the solution space.
# Each must start with "How might we " and must not contain any of these
# words as whole words: button, screen, popup, email, notification.
hmw_questions: list[str] = [
    # "How might we let new subscribers skip a delivery before travelling?",
]

# TODO: Fill a HEART table for ONE goal of your choice from the insights.
# At least 3 rows, each a dict with keys: goal, signal, metric.
heart_rows: list[dict[str, str]] = [
    # {"goal": "subscribers keep their plan through busy weeks",
    #  "signal": "pauses taken instead of cancellations",
    #  "metric": "pause-use rate per subscription per quarter"},
]

"""Sample tests for the hmw-and-heart exercise."""

import re

from solution import problem_statements, hmw_questions, heart_rows


USER_KEYWORDS = {
    "subscriber", "subscribers", "customer", "customers",
    "user", "users", "parent", "parents", "elderly", "member",
    "members", "resident", "residents", "person", "people",
}
NEED_KEYWORDS = {
    "need", "needs", "want", "wants", "require", "requires",
    "cannot", "struggle", "struggles", "must",
}
EVIDENCE_PATTERN = re.compile(r"\bi\d+\b|\bevidence\b|\binterview", re.I)
HMW_PATTERN = re.compile(r"^How might we ")
BANNED_HMW = ("button", "screen", "popup", "email", "notification")


def _words(text: str) -> list[str]:
    return re.findall(r"[a-zA-Z]+", text.lower())


def test_two_problem_statements():
    assert len(problem_statements) == 2, (
        f"Write exactly 2 problem statements, got {len(problem_statements)}"
    )


def test_problem_statements_name_a_user():
    for s in problem_statements:
        assert any(w in USER_KEYWORDS for w in _words(s)), (
            f"'{s}' names no user "
            f"(try subscriber, user, parent, elderly, ...)"
        )


def test_problem_statements_name_a_need():
    for s in problem_statements:
        assert any(w in NEED_KEYWORDS for w in _words(s)), (
            f"'{s}' names no need "
            f"(try need, want, cannot, struggle, must, ...)"
        )


def test_problem_statements_cite_evidence():
    for s in problem_statements:
        assert EVIDENCE_PATTERN.search(s), (
            f"'{s}' cites no evidence "
            f"(name an insight id like i1, or write 'evidence' or 'interview')"
        )


def test_six_hmw_questions():
    assert len(hmw_questions) == 6, (
        f"Write exactly 6 HMW questions, got {len(hmw_questions)}"
    )


def test_hmw_prefix():
    for q in hmw_questions:
        assert HMW_PATTERN.match(q), (
            f"'{q}' does not start with 'How might we '"
        )


def test_hmw_no_solution_words():
    for q in hmw_questions:
        lower = q.lower()
        hit = [w for w in BANNED_HMW if re.search(rf"\b{w}\b", lower)]
        assert not hit, (
            f"'{q}' contains banned solution words: {hit}"
        )


def test_heart_rows_count():
    assert len(heart_rows) >= 3, (
        f"Write at least 3 HEART rows, got {len(heart_rows)}"
    )


def test_heart_row_shape():
    required = {"goal", "signal", "metric"}
    for row in heart_rows:
        missing = required - set(row.keys())
        assert not missing, f"Row {row} is missing keys: {missing}"
        for k in required:
            assert row[k].strip(), f"Row {row}: {k} is empty"

Then take the lesson’s questions. They start on whichever concept you have answered least well so far, and each wrong choice says the slip behind it:

Interview questions

Warm-up (fresher to 2–5 years): when is a “How might we” question really a solution in disguise?

When the question names a thing to build. “How might we add a Pause button to the plan screen?” already picks a particular button on a particular screen, and the ten different sketches a workshop might produce in reply are reduced to one that already has a drawing. The telltale words are the ones that name a thing: button, screen, popup, email, notification, modal. Keep the question at the level of what the subscriber needs to do: “How might we let subscribers skip a week they will be away?” leaves room for a toggle in the reminder, a dialog on the web, a phone-agent script and the three other answers nobody has sketched yet. If a word in the question is already what the team would build, rewrite the question until it could still have ten answers.

Two heavier questions sit under the warm-up. Draft the opening of each in a sentence or two of your own before you tap through to the first step:

Key takeaways

• A problem statement is one sentence with three parts — a user, a need and the evidence behind it — and a reader must be able to disagree about each one.
• “How might we” questions open a problem into answers without naming what the team will build, so solution words like button and screen have no place in them.
• Assumption mapping places beliefs by importance and evidence, and the top-left square (important, weakly evidenced) is where the quarter’s first test belongs.
• Impact-effort, RICE, Kano and HEART each answer a different question, and RICE’s crisp score hides four estimates that must be labelled estimates and recorded with who guessed them.
• HEART ties a design change to its measurement by walking from goal to signal to metric, and skips the categories that are not meaningful to the change at hand.
• Accessibility and consumer-protection rules are constraints that sit above the prioritisation table, not features the team scores against the others.