Planning UX research: questions and methods

Half of new MealBox subscribers cancel inside their first week, and nobody around the table can say why. The product lead wants a plan by Friday, the engineering lead has already drawn a sign-up form, and the design critique next week expects a decision about which part of the week-view is redesigned first. The question in front of you is not which tool to run; it is which questions, asked of which people, with which method, in a week.

The invented numbers above are this lesson’s own; the way of working is from the GOV.UK Service Manual and the human-centred-design standard. One round of research walks from a vague brief through a short list of questions, a chosen method for each, a careful recruit, the sessions themselves, analysis the same day, and finally a design decision that changes.

A research question earns its place when a decision is waiting

The slowest round of research is the one started without research questions. A research question is not a question the researcher asks a participant; it is what the team has to learn to decide what to do next. GOV.UK’s Service Manual, in its page on capturing research questions, puts the test as simply as it can be put: these are things the team needs to learn, not questions you will ask users. Broad, open starters work best, beginning with “who”, “what”, “when”, “where” or “why”, because they leave space for an answer the writer did not already have in mind.

A useful second test is harder, and it keeps most lists short. For every question on your sticky note, name the decision its answer would change. If you cannot, drop the question — or rewrite it until a decision rides on it. One small cancellation project turns the test into four stages, each one producing the input the next one needs:

Three decisions were in the air at the start of that project: whether to redesign the week-view or the plan page first, which friction to remove in the pause flow, and whether to publish the kitchen cut-off time in the app. The questions were written against each of them, out loud, and the ones that could not be matched to a decision were dropped. “Would subscribers use a skip-a-day button?” is the question that always wants to join a plan like this one; it is a feature test in disguise, and feature tests do not predict behaviour.

Methods split by what they reveal, and what they do not

A method map is older than its sticky notes. Christian Rohrer’s 2014 article at the Nielsen Norman Group, “When to Use Which User-Experience Research Methods”, draws it as three axes: attitudinal against behavioural, qualitative against quantitative, and natural against scripted context. The axes matter more than the exact position of any method on them: they say what a method can see.

• Attitudinal methods (interviews, surveys) ask people about their experience, so they reveal reasons and reactions as people describe them. They are weak at behaviour nobody noticed, which is where the hardest usability problems live.
• Behavioural methods (contextual inquiry, usability tests, analytics) watch or record what people did. They are strong at the parts of the task that happened, and silent on why.
• Qualitative methods (interviews, contextual inquiry, usability tests with think-aloud) give rich detail about a few people. Their samples are too small to say how common anything is.
• Quantitative methods (surveys, analytics, logs) count behaviour or attitudes across many people, which lets rates be reported with a margin. They answer “how many” and leave “why” to the follow-up.
• Natural context (contextual inquiry, diary studies, analytics) keeps the task in its real place; scripted context (lab usability tests, surveys) lets the researcher compare like with like.

The practical use of the map is to pick two methods that answer different questions together. For the MealBox round, six interviews explain why subscribers cancel, a usability test on the pause flow finds the friction on a 320 px phone, and a short item at the end of the survey counts how many subscribers knew the kitchen cut-off. The table below lays out what five methods will and will not show on this problem:

Discovery widens, and evaluative tests, and they ask different questions

A round of research is one of two shapes, and the shape decides which methods are allowed in the room. Discovery widens the picture of a problem: who the users are, what they are trying to do, how they do it today and where the trouble is. The GOV.UK Service Manual’s page on user research in discovery is clear that it comes before design decisions, not after: “You must do this before you start planning, designing or building your service.” A split test, a usability test and a satisfaction survey all answer the wrong question in discovery, because they measure a design that nobody has argued through.

Evaluative research asks whether a design works. It takes a candidate — a sketch, a prototype, a live screen — and tests it against named users, named goals and a context of use, in the way the first module’s lesson on usability set out. A usability test on the pause flow is evaluative; so is a satisfaction survey about the week-view, or an A/B test of two labels. The result is the number or the quote that moves a design one way or the other.

Teams get into trouble when they mix the two inside one round. Running a usability test “to see if people like the plan page” when nobody has described why they cancel is work on the wrong artefact: the plan page may not be the problem at all, and the test will mostly report what people say about something that is beside the point. Keep the shapes separate, run them in that order, and only recruit evaluative participants once discovery has written a problem statement anyone can argue with.

Desk research before anyone is recruited

Recruiting participants is the single most expensive step of a round. It costs calendars, incentives, waiting time and goodwill, and most teams can run it only every few weeks. Everything that can be learned without recruiting belongs before the recruit. The Service Manual calls this “examining existing data”; the shorter name in a working team is desk research, and it is the step most teams skip.

Four kinds of existing data are usually in reach, and the first three cost nothing new:

• Support logs and tickets. The people who use the service have already written to say what went wrong; categorise six months of it, and the first draft of the research questions tends to be already there.
• Analytics and search logs. The product funnel shows where people fall out of a flow, and the on-site search log shows what people typed that returned nothing — a cheap list of missing labels and missing features.
• Earlier research. The organisation almost always has old interviews, usability-test reports and personas; most of them answer at least one question in the new brief, if they are read instead of forgotten.
• Competitive analysis. A task-based scan compares the same job on two or three other services, counting the steps a person has to take to pause a subscription or to report a problem. The output is a numbered flow, not a gallery of screenshots: copying a competitor’s visual design copies its mistakes, as the Service Manual notes.

Desk research pays twice. It sharpens the research questions, so the recruit answers more of them, and it gives the team an evidence base before anyone is interviewed, so interviews start from “here is what we know, what else is going on?” rather than from scratch.

A small sample finds problems but does not count them

Qualitative samples are smaller than most newcomers expect, and they have to be: the point is depth, not reach. Five to eight participants is a common size for one round, with another round after any big design change. That number is not a shortcut; it comes from a trade-off between the time a session takes to analyse and the themes a session adds. By the fourth or fifth session, a researcher is usually hearing the same patterns, which is a good sign that another interview on the same question will add little.

What a small sample cannot do is say how common any of those patterns is. A finding like “five of six new subscribers missed the kitchen cut-off time” names a problem worth designing for; it does not say that 83 % of all subscribers miss the cut-off time. The rate needs a method that can count a representative sample, which the lesson on surveys and behavioural data takes up. Reporting both honestly in the same paragraph is the discipline that keeps a team from either understating a real problem or overstating its size.

A better run of this program checks a plan with four rules in mind: every question moves a decision, every method covers at least one question, participants are described by behaviour, and an access need is named with the extra time it adds. The program below runs those checks on a draft MealBox plan:

"""A one-page MealBox research plan, checked against four rules.

Every number and quote below is invented for this lesson. The plan is for
MealBox (mealbox.example), a fictional lunch-subscription service in Pune,
and the problem is "why do new subscribers cancel in the first week?".

The four rules the plan is checked against come from GOV.UK's Service
Manual pages on capturing research questions and planning a round of
research: a question has to change a decision if it is answered, a method
has to answer at least one of the questions, the participants have to be
described by what they have done rather than only by age or job, and the
plan has to name an access need and the time it adds.
"""

PLAN = {
    "problem": "Half of new MealBox subscribers cancel inside their first week",
    "questions": [
        {"id": "Q1", "text": "What do new subscribers notice between the first delivery and the first cancel?",
         "informs": ["D1", "D2"]},
        {"id": "Q2", "text": "Which parts of the order flow do people find hardest on a 320 px phone?",
         "informs": ["D2"]},
        {"id": "Q3", "text": "How does the kitchen cut-off time show up in people's week?",
         "informs": ["D3"]},
        {"id": "Q4", "text": "Would subscribers use a skip-a-day button?",
         "informs": []},  # solution in disguise: tests a feature, not a decision
    ],
    "decisions": [
        {"id": "D1", "about": "Whether to redesign the week-view or the plan page first"},
        {"id": "D2", "about": "Which friction to remove in the pause flow"},
        {"id": "D3", "about": "Whether to publish the kitchen cut-off time in the app"},
    ],
    "methods": [
        {"id": "M1", "name": "Six 60-minute interviews with new subscribers",
         "covers": ["Q1", "Q3"], "kind": "qualitative"},
        {"id": "M2", "name": "Moderated usability test on the pause flow, eight people",
         "covers": ["Q2"], "kind": "qualitative"},
        {"id": "M3", "name": "Zero-result search log review for the last 30 days",
         "covers": ["Q1"], "kind": "behavioural"},
        {"id": "M4", "name": "Short survey of subscribers who cancelled",
         "covers": [], "kind": "quantitative"},  # covers no question
    ],
    "participants": "Six people who signed up this month and either kept the plan or cancelled",
    "access_need": None,  # a missing field
    "timeline_days": 10,
}


def check_plan(plan):
    """Return (findings, is_ready). Each finding is (rule, item id, note)."""
    findings = []
    informed = {d["id"] for q in plan["questions"] for d in plan["decisions"] if d["id"] in q["informs"]}
    for q in plan["questions"]:
        if not q["informs"]:
            findings.append(("decision", q["id"], "no decision is waiting on this question"))
        if q["text"].lower().startswith("would "):
            findings.append(("hypothetical", q["id"], "'would you use' tests a feature, not a decision"))
    orphan = {d["id"] for d in plan["decisions"]} - informed
    for d_id in sorted(orphan):
        findings.append(("decision", d_id, "no question would move this decision"))
    for m in plan["methods"]:
        if not m["covers"]:
            findings.append(("method", m["id"], "answers none of the research questions"))
    if not plan.get("participants") or "and either" not in plan["participants"]:
        pass  # placeholder: a real check would read for behaviour, not demographics
    if not plan.get("access_need"):
        findings.append(("participants", "plan", "no access need is named, so the recruit will leave people out"))
    if plan.get("timeline_days", 0) < 7:
        findings.append(("timeline", "plan", "under a week leaves no time for analysis"))
    return findings, not findings


findings, ready = check_plan(PLAN)

print(f"MealBox research plan: {len(PLAN['questions'])} questions, {len(PLAN['methods'])} methods, "
      f"{PLAN['timeline_days']} days")
print()
if ready:
    print("The plan passes all four checks and is ready to run.")
else:
    print(f"The plan has {len(findings)} findings to fix before it is ready:")
    for rule, item, note in findings:
        print(f"  [{rule:<12}] {item}: {note}")
print()
print("What a fix looks like")
print("  Q4 'Would subscribers use a skip-a-day button?' tests a solution; rewrite as")
print("     'How do subscribers notice and manage the days they do not want a delivery?'")
print("  M4 adds a decision link: 'cancellation survey informs D1 (what to redesign first)'.")
print("  Access need: 'at least one participant uses a screen reader; set aside 30 extra minutes'.")

MealBox research plan: 4 questions, 4 methods, 10 days

The plan has 4 findings to fix before it is ready:
  [decision    ] Q4: no decision is waiting on this question
  [hypothetical] Q4: 'would you use' tests a feature, not a decision
  [method      ] M4: answers none of the research questions
  [participants] plan: no access need is named, so the recruit will leave people out

What a fix looks like
  Q4 'Would subscribers use a skip-a-day button?' tests a solution; rewrite as
     'How do subscribers notice and manage the days they do not want a delivery?'
  M4 adds a decision link: 'cancellation survey informs D1 (what to redesign first)'.
  Access need: 'at least one participant uses a screen reader; set aside 30 extra minutes'.

The draft fails all four of its own checks, which is the common state of a plan the night before it is due. Press Run, then open the plan dictionary at the top and fix one field at a time: a hypothetical question becomes a question about a past delivery, a decision without a question earns one, a method that answers nothing gains a question or is cut, and the access-need line gains a sentence. Rerun each time and the findings list shortens.

A one-page plan fits on one page, and ends with ethics

The plan is where the questions, the methods and the recruit sit on one piece of paper the whole team can argue with. The Service Manual’s page on planning a round of research sets out the shape, which is nearly the same across rounds: objectives, participants, session times, recording and note-taking, the prototype if there is one, time for analysis. Keep it on one page: a plan nobody reads is the artefact that earns its keep least often of all.

Two parts of that page carry weight far beyond their size. The participants line decides who the research is for and who it leaves out; the Service Manual’s page on discovery is firm that research must include disabled people and people with low digital skills, and most ethics problems that appear later can be traced to a screener that did not plan for them. The ethics line decides what happens to recordings, consent and incentives — covered in full in the next lesson of this module — and should name a retention period and the moment the data is deleted. GOV.UK’s page on managing user research data suggests two or three years as a reasonable retention period, and sooner if the data is no longer needed.

The lessons after this one go deeper into each part of this plan: recruiting and consent for the people, interviews and field research for the sessions, and surveys and behavioural data for the parts that need to count.

Practise

The best test of a plan is to write one. Fill in the four gaps in a MealBox plan so that it would survive the four rules the lesson sets, and read the test file to see what each rule looks like in code:

python -m pytest test_plan.py

"""Starter: fill in the four TODOs so plan matches the four rules in prompt.md."""

PLAN = {
    "problem": "Half of new MealBox subscribers cancel inside their first week",
    "questions": [
        {"id": "Q1",
         "text": "What do new subscribers notice between the first delivery and the first cancel?",
         "informs": ["D1", "D2"]},
        {"id": "Q2",
         "text": "Which parts of the pause flow do people find hardest on a 320 px phone?",
         "informs": ["D2"]},
        {"id": "Q3",
         "text": "Would you use a skip-a-day button?",   # TODO 1: rewrite this as a question about something people have already done, informing D3
         "informs": []},
        {"id": "Q4",
         "text": "How many subscribers who cancelled did not know the kitchen cut-off time?",
         "informs": []},                                 # TODO 2: name the decision this question moves (one of D1 / D2 / D3)
    ],
    "decisions": [
        {"id": "D1", "about": "Whether to redesign the week-view or the plan page first"},
        {"id": "D2", "about": "Which friction to remove in the pause flow"},
        {"id": "D3", "about": "Whether to publish the kitchen cut-off time in the app"},
    ],
    "methods": [
        {"id": "M1", "name": "Six 60-minute interviews with new subscribers",
         "covers": ["Q1", "Q3"], "kind": "qualitative"},
        {"id": "M2", "name": "Moderated usability test on the pause flow, eight people",
         "covers": ["Q2"], "kind": "qualitative"},
        {"id": "M3", "name": "Short survey of subscribers who cancelled",
         "covers": [], "kind": "quantitative"},          # TODO 3: list the question ids this survey covers (make the survey count the cut-off awareness)
    ],
    "participants": "People aged 25-40 who live in Pune",  # TODO 4: rewrite as a behavioural screen (include "signed up" and either "kept" or "cancelled")
    "access_need": "",                                     # TODO 5: name an assistive technology and the extra time
    "timeline_days": 10,
}

from plan import PLAN


def test_every_question_informs_a_decision():
    """Every question lists at least one decision id, and no question starts with 'Would you'."""
    decision_ids = {d["id"] for d in PLAN["decisions"]}
    for q in PLAN["questions"]:
        assert q["informs"], f"{q['id']}: no decision is waiting on this question"
        for d_id in q["informs"]:
            assert d_id in decision_ids, f"{q['id']}: {d_id!r} is not one of the decisions"
        assert not q["text"].lower().startswith("would "), f"{q['id']}: 'would you' tests a feature, not a decision"


def test_every_decision_is_informed_by_a_question():
    """No decision in the plan lacks a question that would move it."""
    informed = {d_id for q in PLAN["questions"] for d_id in q["informs"]}
    for d in PLAN["decisions"]:
        assert d["id"] in informed, f"{d['id']}: no question would move this decision"


def test_every_method_covers_a_question_and_vice_versa():
    """Each method answers at least one question, and each question has a method."""
    all_q = {q["id"] for q in PLAN["questions"]}
    covered = set()
    for m in PLAN["methods"]:
        assert m["covers"], f"{m['id']}: this method answers none of the research questions"
        for q_id in m["covers"]:
            assert q_id in all_q, f"{m['id']}: {q_id!r} is not one of the questions"
            covered.add(q_id)
    missing = all_q - covered
    assert not missing, f"no method covers: {sorted(missing)}"


def test_participants_screen_by_behaviour():
    """The participants string names behaviour (signed up, kept or cancelled), not only an age range."""
    text = PLAN["participants"].lower()
    assert "signed up" in text, "name the behaviour that qualifies, not only an age range"
    assert ("kept" in text) or ("cancelled" in text), "name whether participants kept the plan or cancelled it"


def test_access_need_is_named_with_extra_time():
    """access_need names a technology or support need and the time it adds."""
    text = PLAN["access_need"]
    assert isinstance(text, str) and len(text) >= 20, "name an access need in a short sentence"
    assert any(word in text.lower() for word in ("screen reader", "braille", "sign language", "voice over",
                                                  "voiceover", "talkback", "magnifier")), \
        "name an assistive technology or support (for example 'screen reader')"
    assert any(word in text.lower() for word in ("minutes", "extra time", "additional time")), \
        "name the extra time this adds to the session"

The self-check below mixes recall with two judgements about sizing a round:

Interview questions

Warm-up (fresher to 2–5 years): what makes a research question strong?

A strong research question is open, specific, and tied to a decision. Open means it starts with what, when, how or who, and leaves room for an answer the writer did not have in mind; “would you use…?” and other yes/no shapes close the enquiry before it starts. Specific means it names the people, the task and the moment: “what do new subscribers notice between the first delivery and the first cancel?” rather than “why do people cancel?”. Tied to a decision means its answer would change a design choice already on the table: which part of the week-view is redesigned first, or whether to publish a kitchen cut-off. Questions that cannot survive that test are either curiosity or features in disguise, and the plan should drop them.

Two harder questions follow: one takes a brief that carries a number and no problem, the other reads a six-person qualitative sample. Sketch your own opening, then set it against the step published with each:

Key takeaways

• A research question earns its place when a decision is waiting on its answer, so write the question beside the decision and drop any question nothing rides on.
• Methods split by what they reveal — attitudes against behaviour, “why” against “how many”, natural against scripted — so pick two that answer different questions together, not two of the same kind.
• Discovery widens a problem before any design exists; evaluative work tests a design that does; running them together in one round is how teams build the wrong thing well.
• Desk research with support logs, analytics, old studies and a task-based competitive scan comes before anybody is recruited, and it sharpens every question that goes into the round.
• A one-page plan fits on one page because its ethics line, its access-need line and its retention sentence are the last things to be cut, not the first.