Surveys and behavioural data for UX

A product lead at MealBox said last week that four thousand people had taken the in-service survey, so the results had to be representative. The margin of error at four thousand responses is small; what the number cannot do is say anything about the people who left the service two months ago, or who never signed up at all. Survey mathematics is unforgiving on one line: a confident margin counts only when the sample has a real shot at the whole population.

This lesson is the counting half of a round of research. It writes the questions people can answer honestly, reads the funnel beside the interviews rather than instead of them, and ends at a short decision supported by three sources that each answer a different piece of it.

One idea per question, with balanced words

Pew Research Center’s “Writing Survey Questions” guide and AAPOR’s “Best Practices for Survey Research” agree on a small list that handles most of the slips in a draft questionnaire. Each line below reads as a rule, and failing it has a predictable effect on the data.

• One idea per question. A question that mentions two different things — “how often do you order delivery and cook at home?” — is a double-barrelled question; the answer cannot say which half it is responding to. Split it into two.
• Balanced wording. Loaded words signal the answer. “Generous plans” tips one way, “forced autopay” the other; the fix is a neutral alternative on each side, with the same kind of word for both halves.
• An honest “don’t know”. When “don’t know” is a real possibility, give respondents the option; the Service Manual, Pew and AAPOR each note that omitting it pushes answers to a position the respondent did not actually hold.
• Specific and concrete. “In the last 30 days, how many lunches did you order as delivery?” beats “how often do you order delivery?”; a specific window and a count turn an average into a number the respondent can actually give.
• Short, plain words. Team jargon (“the UX of the pause flow”) invites polite nodding and takes the answer with it. The replacement is the plain phrase (“the pause flow”).

Replace agree–disagree grids on belief statements with a forced choice between two reasonable alternatives. Pew calls this the forced-choice format and finds it less prone to acquiescence bias: less informed respondents tend to say “agree” to statements they would not have thought up on their own, which is one of the oldest effects in survey methodology. “Which of these is closer to your view? MealBox is a service I would use again. MealBox is a service I would not use again.” The pattern of answers tends to be more honest than any grid.

Scales, forced choices, and the agree–disagree trap

A scale records how strong an opinion is. The common five-point scale (“very bad, bad, neither, good, very good”) works when the thing being rated has an ordered quality and the labels are symmetrical. Pew’s guidance is pragmatic about where scales break: when the labels are unbalanced (“good, very good, excellent, truly amazing”), when the midpoint is missing, when the ends are vague, and when the question actually calls for a count rather than a feeling.

Three choices for most short UX surveys fit most plans:

• Specific past-behaviour counts (“In the last 30 days, how many lunches did you order as delivery?”) when a number is available.
• Forced choice between two statements when the question is about belief (“MealBox is a service I would use again”), which replaces the agree–disagree grid.
• Five-point rating with symmetrical labels when a single comparable score across the service is useful (“much faster, somewhat faster, about the same, somewhat slower, much slower than other delivery services”).

Mode matters too. The table below is the practical choice between four survey channels, scored on cost and speed, honesty on sensitive items, and representativeness:

Keep the chosen mode compatible with the respondent’s own device and context. The Service Manual’s point 3 about joined-up channels applies to surveys as well: a service that reaches people by phone and at a counter needs a survey that can reach them there too, or the response set only hears from the online half.

Margin grows with smaller samples, and with weaker sampling

Margin of error is the common way to say how precise a proportion from a survey is. The common planning formula at 95 % confidence, for a proportion p and sample size n, is

m = z × sqrt(p × (1 − p) / n), with z ≈ 1.96 and p = 0.5 for the widest case.

In numbers, which the example below prints:

"""Margin of error and funnel math for a MealBox survey.

Every number below is drawn for this lesson. The margin uses the normal
(Wald) approximation for p = 0.5 at 95 % confidence, which is the common
planning formula AAPOR describes on its margin-of-error page. The funnel
runs on synthetic MealBox events generated with a fixed seed so the drop
between steps is reproducible.
"""

from math import sqrt

Z95 = 1.959963984540054  # two-sided 95 % confidence multiplier


def margin(n, p=0.5):
    """Simple normal-approximation margin of error, in percentage points."""
    return 100 * Z95 * sqrt(p * (1 - p) / n)


def sample_needed(margin_percent, p=0.5):
    """n needed for the given margin, at 95 % confidence."""
    m = margin_percent / 100
    return int(round((Z95 ** 2 * p * (1 - p)) / (m ** 2)))


print("Margin of error at 95 %, for p = 0.5 (the widest it gets)")
print(f"  n = {'sample':>7}  margin")
for n in (50, 100, 200, 385, 500, 1000):
    print(f"  n = {n:>7}  ±{margin(n):>4.1f} points")

print()
print("Sample needed for a given margin, at 95 %, p = 0.5")
for m in (10, 5, 3, 2):
    print(f"  ±{m:>2} points: n = {sample_needed(m):>5}")

# ---------------------------------------------------------------------------
# A small MealBox funnel, generated with a fixed seed
# ---------------------------------------------------------------------------
import random

random.seed(2026)
VISITORS = 10_000
# Share of the previous step that advances (invented, deterministic)
STAGE_RATE = {
    "plan_viewed": 1.00,
    "plan_chosen": 0.42,
    "address_entered": 0.78,
    "payment_started": 0.72,
    "payment_success": 0.86,
}
counts = {}
run = VISITORS
for stage, rate in STAGE_RATE.items():
    # Draw a few per-user outcomes so the counts are not perfectly deterministic in shape
    run = sum(1 for _ in range(run) if random.random() < rate)
    counts[stage] = run

print()
print(f"MealBox funnel, from {VISITORS:,} invented visits")
print(f"  {'stage':<17} {'through':>8}  {'from prev':>10}")
previous = VISITORS
for stage, c in counts.items():
    rate = 100 * c / previous
    print(f"  {stage:<17} {c:>8,}  {rate:>8.1f} %")
    previous = c

print()
drop_count = counts["plan_chosen"] - counts["address_entered"]
drop_rate = 100 * drop_count / counts["plan_chosen"]
print(f"Biggest drop: {drop_count:,} people stopped between 'plan chosen' and 'address entered' "
      f"({drop_rate:.1f} % of those who chose a plan).")
print()
print("Three sentences that keep the counting honest")
print("  'n = 100 interviews is not n = 100 of our subscribers': a convenience sample has no margin.")
print("  'The funnel shows what happened, not why': interviews, not events, name the reasons.")
print("  'Plus or minus 5 points' needs n = 385 or more at 95 %, p = 0.5; smaller numbers get wider margins.")

Margin of error at 95 %, for p = 0.5 (the widest it gets)
  n =  sample  margin
  n =      50  ±13.9 points
  n =     100  ± 9.8 points
  n =     200  ± 6.9 points
  n =     385  ± 5.0 points
  n =     500  ± 4.4 points
  n =    1000  ± 3.1 points

Sample needed for a given margin, at 95 %, p = 0.5
  ±10 points: n =    96
  ± 5 points: n =   384
  ± 3 points: n =  1067
  ± 2 points: n =  2401

MealBox funnel, from 10,000 invented visits
  stage              through   from prev
  plan_viewed         10,000     100.0 %
  plan_chosen          4,270      42.7 %
  address_entered      3,372      79.0 %
  payment_started      2,384      70.7 %
  payment_success      2,080      87.2 %

Biggest drop: 898 people stopped between 'plan chosen' and 'address entered' (21.0 % of those who chose a plan).

Three sentences that keep the counting honest
  'n = 100 interviews is not n = 100 of our subscribers': a convenience sample has no margin.
  'The funnel shows what happened, not why': interviews, not events, name the reasons.
  'Plus or minus 5 points' needs n = 385 or more at 95 %, p = 0.5; smaller numbers get wider margins.

The useful readings off the table are: ±14 points at n = 50, ±10 at n = 100, ±5 at n = 385, ±4 at n = 500 and ±3 at n = 1,000. The common planning targets are ±5 points at 95 % (385 or more respondents) and ±3 points for a tighter claim (about 1,067 respondents). Halving a margin needs about four times the sample, because the standard error scales with 1 over the square root of n; the practical cost of a precise claim is not “a bit more survey” but a lot more. For a quick check against the formula, MySmartCoPilot’s own sample-size calculator and confidence-interval calculator carry the same numbers.

Representativeness is a separate claim. Margins assume a probability sample from a defined frame; most UX surveys are not that. A self-selected pop-up, an in-service prompt, a social-media post and an opt-in panel are all convenience samples, which say nothing about the people who did not take them, however many did. The sentence to use in a readout is “n = 300 opinions from the people in our app this week, which does not represent the people who left or never joined”; it is longer than “n = 300” and far more honest.

Behavioural data shows what happened, not why

Analytics are strong at scale and silent on reasons. A funnel from a plan page to a paid order counts every visit and shows which step people stop at; search logs record every term that returned nothing; event streams record every click. Three habits make the data trustworthy on its own and useful alongside interviews.

• Keep events small and named for the thing that happened. plan_viewed, plan_chosen, address_entered, payment_started, payment_success carries more weight than step_1_click. Match event names to the plain words users see.
• Keep personal data out of the event stream. Account ids are usually fine; names, addresses, phone numbers, email addresses and anything that could re-identify a person belong in a different store with its own access controls. The Service Manual’s rule for research data applies to product analytics too: collect only what you need, protect it and delete it on schedule.
• Read what did not happen as evidence. A zero-result search log is a short list of labels and features that people expected to find and did not. In many services, it is the cheapest source of research questions, with no recruitment at all.

The lesson’s example prints a short MealBox funnel on synthetic events with a fixed seed. The biggest drop sits between “plan chosen” and “address entered”, about one in five of those who got that far. That number names a problem to solve; it does not name a reason. The next method does.

Three sources feed one decision

Triangulation sits at the end of the module because it is where the methods meet. Three sources on one design decision tend to beat any one of them, because each answers a question the others cannot. The timeline below walks through a MealBox decision about the address step: a funnel finding, a round of interviews, and a short usability test, each on its own and then together:

The redesign of the address step was supported, in order, by: a drop of about 21 % in the funnel, three plausible reasons in interviews, and a ranking from eight usability tests that put the landmark field at the top. Without the funnel, the team would have argued about whether the problem was common. Without the interviews, the funnel would have said nothing about why. Without the usability test, the three reasons would have been treated as equal weight, which they were not. The one decision costs three methods, and that is cheaper than the one bad redesign would have been.

Where does the module end up? The three lessons before this one planned the research, recruited the people and ran the sessions; this one counted what counts can count. The next module, synthesis, takes the mixture of qualitative and quantitative findings through to the design decisions themselves — themes, personas, journeys and the one-sentence problem statement that pulls the whole round together.

Practise

Eight MealBox survey questions, each with a planted problem, and a sample-size calculation at the end: rewrite each question so a lint identical to the lesson’s rules passes, and set n_for_5_points to the right value:

python -m pytest test_survey.py

"""Starter: rewrite the eight survey questions and compute the sample size."""

SURVEY = [
    # Each item: (question text, options list; empty list means open-ended)
    (
        "Would you say the MealBox pause flow is clear and easy?",
        ["Yes", "No"],
    ),
    (
        "Do you agree that MealBox's delivery is amazing and the plans are generous?",
        ["Strongly agree", "Agree", "Neither", "Disagree", "Strongly disagree"],
    ),
    (
        "How often do you order lunch delivery and do you cook at home?",
        ["Never", "Rarely", "Sometimes", "Often", "Always"],
    ),
    (
        "What is the kitchen cut-off time at your location?",
        ["09:30", "10:00", "10:30"],
    ),
    (
        "The pause button is in the top-right of the menu. Did you find it intuitive?",
        ["Yes", "No"],
    ),
    (
        "Our customers love MealBox. Which best describes you?",
        ["I love MealBox", "I like MealBox", "I am a customer"],
    ),
    (
        "How did you find the sign-up funnel?",
        ["Great", "OK", "Not great"],
    ),
    (
        "Would you prefer forced autopay or a flexible pay-per-week plan?",
        ["Forced autopay", "Flexible pay-per-week"],
    ),
]

n_for_5_points = 0  # TODO: compute the n needed for a 5-point margin at 95 % confidence, p = 0.5

from survey import SURVEY, n_for_5_points

LEADING = ["would you say", "do you agree", "clear and easy", "intuitive", "amazing", "the best",
           "great", "love mealbox"]
LOADED = ["generous", "forced", "amazing", "love "]
JARGON = ["ux", "ui", "cta", "mvp", "funnel", "nps", "sprint", "sop"]


def is_double_barrelled(text):
    for conj in (" and ", " or "):
        parts = text.lower().split(conj)
        if len(parts) >= 2 and sum(
            1 for p in parts if p.strip().startswith(("do you", "did you", "how often", "how much", "would you"))
        ) >= 2:
            return True
    return False


def check_question(text, options):
    text_low = text.lower()
    problems = []
    if any(phrase in text_low for phrase in LEADING):
        problems.append("leading")
    if is_double_barrelled(text):
        problems.append("double-barrelled")
    if any(word in text_low for word in LOADED):
        problems.append("loaded wording")
    for term in JARGON:
        if any(text_low[i:i + len(term) + 1] == f" {term}" for i in range(len(text_low) - len(term))):
            problems.append(f"jargon: {term}")
            break
    # Agree-disagree grid on a belief statement
    opts_low = [o.lower() for o in options]
    is_grid = any("agree" in o for o in opts_low) and any("disagree" in o for o in opts_low)
    starts_with_belief = text_low.startswith(("do you agree", "mealbox", "our customers"))
    if is_grid and starts_with_belief:
        problems.append("agree-disagree grid")
    # Factual question with no 'don't know'
    is_factual = any(ch.isdigit() for ch in text) or "cut-off" in text_low or "price" in text_low
    has_dontknow = any(("don" in o and "know" in o) or ("do not" in o and "know" in o) for o in opts_low)
    if is_factual and options and not has_dontknow:
        problems.append("no don't know option")
    return problems


def test_every_question_passes():
    for text, options in SURVEY:
        problems = check_question(text, options)
        assert not problems, f"{text!r}: {problems}"


def test_number_of_questions_unchanged():
    assert len(SURVEY) == 8, "rewrite the eight questions, do not add or remove any"


def test_sample_size_for_five_points():
    assert n_for_5_points == 385, "n for ±5 points at 95 % with p = 0.5 is 385 (round up from 384.16)"

The self-check below mixes question wording with one margin calculation and one triangulation judgement:

Interview questions

Warm-up (fresher to 2–5 years): what does margin of error measure, and what does it not?

Margin of error measures how precise a proportion from a probability sample is at a given confidence level. At 95 % with p = 0.5, the common planning formula gives about ±5 points at n = 385, ±4 at n = 500 and ±3 at n = 1,000. Halving the margin needs the sample to grow by about four times, which is the practical cost of a tighter claim. What it does not measure is representativeness: a margin assumes the respondents are a probability sample from a defined frame, and a convenience sample (a pop-up, an in-service prompt, a social-media post) cannot carry a margin honestly even when the count is large. Keep the two claims apart in a readout: “n = responses with margin M” names precision; the sentence beside it says what the sample does and does not stand in for.

Two harder questions follow: one reads a satisfaction score beside a funnel drop, the other answers a request to publish the raw response CSV openly. Plan your own answer before you read the step published with each:

Key takeaways

• One idea per question, balanced words and an honest “don’t know” option carry most of the quality in a short survey; replace agree–disagree grids on belief statements with forced-choice alternatives.
• Margin of error at 95 % with p = 0.5 is about ±5 points at n = 385 and ±3 at n = 1,000; halving the margin needs about four times the sample.
• Margin assumes a probability sample; a convenience sample cannot carry a margin honestly, however many respondents it has.
• Behavioural data counts what happened; funnels, events and zero-result searches do not say why, so they need a method that can report reasons.
• Three sources on one decision — a funnel finding, interview reasons and a usability-test ranking — line up design work that any one of them, on its own, would leave open.