Product Manager Interviews Module 3 – Metrics, analytics and experiments
Reading A/B test results
Read an A/B test result as a product manager: the lift, its interval and p-value, the split check, false wins from peeking, and the call to ship or not.
What you will learn
- Read a result's lift, confidence interval and p-value in plain words
- Check a test for sample ratio mismatch before trusting it
- Avoid peeking, too many metrics and segments chosen after the fact
- Decide to ship, iterate or stop from a result and its guardrails
Before you start
What you’ll learn · Before you start
What you will learn
- Read a result's lift, confidence interval and p-value in plain words
- Check a test for sample ratio mismatch before trusting it
- Avoid peeking, too many metrics and segments chosen after the fact
- Decide to ship, iterate or stop from a result and its guardrails
Before you start
On this page
In a test, 1,000 of 10,000 control users ordered and 1,090 of 10,000 treatment users did, and the team asks whether to ship. Reading A/B test results turns those four numbers into a call: how big the lift is, how sure the data are, and whether the test itself can be trusted. Most wrong calls skip a step, such as a split nobody checked, a test stopped on its best day, or a lift too small to be worth its cost.
Below, you read a lift, its interval and its p-value in plain words, check the split, see with a simulation how peeking makes false wins, and decide.
Read the lift, its interval and its p-value together
The lift is the treatment’s rate minus the control’s: 10.9 % − 10.0 % = +0.90 points, 9.0 % in relative terms. The p-value: The probability of a difference at least as large as the one observed if the change did nothing and the test's assumptions hold. It is not the probability that the change works. says how surprising that would be if the change did nothing. The 95 % confidence interval: A range of effect sizes that fit the data, such as +0.05 to +1.75 points for a lift. A 95 % interval comes from a method that covers the true effect in 95 % of tests run the same way. gives the range of lifts that fit the data. Run the readout:
# Reading one A/B test: the lift, a two-sided z-test for two proportions (pooled), the 95 % interval of the
# difference (unpooled) and a chi-square check of the split against the 50/50 that was configured.
from math import sqrt
from statistics import NormalDist
N01 = NormalDist()
def readout(conv_a, n_a, conv_b, n_b):
p_a, p_b = conv_a / n_a, conv_b / n_b
pooled = (conv_a + conv_b) / (n_a + n_b)
z = (p_b - p_a) / sqrt(pooled * (1 - pooled) * (1 / n_a + 1 / n_b))
p_value = 2 * (1 - N01.cdf(abs(z)))
se = sqrt(p_a * (1 - p_a) / n_a + p_b * (1 - p_b) / n_b)
margin = N01.inv_cdf(0.975) * se
return p_a, p_b, z, p_value, p_b - p_a - margin, p_b - p_a + margin
def split_check(n_a, n_b, share_a=0.5):
total = n_a + n_b
expected_a, expected_b = total * share_a, total * (1 - share_a)
chi2 = (n_a - expected_a) ** 2 / expected_a + (n_b - expected_b) ** 2 / expected_b
return chi2, 2 * (1 - N01.cdf(sqrt(chi2))) # one degree of freedom: the chi-square is z squared
p_a, p_b, z, p, low, high = readout(1_000, 10_000, 1_090, 10_000)
print(f"control {p_a:.1%}, treatment {p_b:.1%}: lift {(p_b - p_a) * 100:+.2f} points ({(p_b - p_a) / p_a:+.1%} relative)")
print(f"z = {z:.2f}, two-sided p = {p:.4f}")
print(f"95 % interval of the lift: {low * 100:+.2f} to {high * 100:+.2f} points")
chi2, p_split = split_check(10_000, 10_000)
print(f"split 10,000 against 10,000: chi-square {chi2:.2f}, p = {p_split:.2f}")
chi2, p_split = split_check(821_588, 815_482)
print(f"split 821,588 against 815,482: chi-square {chi2:.2f}, p = {p_split:.1e}, about 1 in {round(1 / p_split, -3):,.0f}")Output
control 10.0%, treatment 10.9%: lift +0.90 points (+9.0% relative) z = 2.08, two-sided p = 0.0375 95 % interval of the lift: +0.05 to +1.75 points split 10,000 against 10,000: chi-square 0.00, p = 1.00 split 821,588 against 815,482: chi-square 22.77, p = 1.8e-06, about 1 in 549,000
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 ab_readout.py
Runs on this device, in your browser. The first run downloads Python (about 13.5 MB), which is kept for the next runs.
Your run, in this browser
The test of two proportions in NIST’s handbook gives The observed lift divided by its standard error, the size of the gap that chance alone typically makes. The further z is from 0, the smaller the p-value. = 2.08 and p = 0.0375: a difference this large would arise by chance about 4 times in 100 if the change did nothing. That is not a 96 % chance that the change works. Kohavi, Deng and Vermeer find that reversal even in books, vendor documentation and a published paper. The interval, +0.05 to +1.75 points in the figure below, says more: the lift is probably above zero, and its size is uncertain.
The interval clears zero, and the smallest lift worth shipping lies inside it
Text description of the diagram
A horizontal axis runs from −0.5 to 2.0 percentage points of lift, with ticks every half point. A solid vertical line at 0 is labelled "no change", and a dashed vertical line at 1.0 is labelled "worth shipping": the smallest lift the team chose before the test.
A bar above the axis shows the 95 % interval of the lift, from +0.05 to +1.75 points, and a dot on it shows the lift observed, +0.90 points. The whole bar lies to the right of the zero line, so the lift is probably real. The dashed line at 1.0 crosses the bar, and the dot lies just left of it: the data fit a lift below the one worth shipping as well as one above it.
The legend says that the bar is the 95 % interval and the dot the lift seen, and that the dashed line marks 1.0 point, the smallest lift worth shipping.
Key idea
Read the interval before the p-value: where it lies says whether the lift is likely real, and its width how sure you can be of its size.
Checkpoint
Check: Lift, interval and p-value
Check the split before you trust the result
A test configured 50/50 should split users close to 50/50. A sample ratio mismatch: A split of users between a test's groups that differs from the configured split by more than chance allows, such as 50.2 % to 49.8 % across 1.6 million users configured as 50/50. It usually means the results cannot be trusted. is a split further off than chance allows, and it usually means something removed or added users in one group, such as a crash before logging, a redirect or bot filtering. Fabijan and colleagues found one in about 6 % of experiments at Microsoft, and warn that it usually leaves a test’s results unusable. Their example looks harmless, 50.2 % to 49.8 %:
| Group | Configured | Expected users | Observed users |
|---|---|---|---|
| Control | 50 % | 818,535 | 821,588 |
| Treatment | 50 % | 818,535 | 815,482 |
Each group is 3,053 users from its expected count, and the readout’s A test of whether observed counts differ from the expected ones by more than chance allows. For a split of users it adds (observed − expected)² ÷ expected over the groups. gives 22.77 and p = 1.8 × 10⁻⁶, about 1 in 549,000.
Run the split check before reading any metric, and treat a failure as a stop: find the cause, fix it, run the test again. The site’s A/B test significance calculator flags a split below p = 0.01.
Peeking, many metrics and segments chosen later make false wins
A 5 % level holds for one planned look. peeking: Checking a test for significance again and again while it runs, and stopping at the first significant result. It raises the share of false wins well above the 5 % that the threshold promises. every day and stopping at the first significant result is many looks, and so is reading 20 metrics or cutting the result into segments afterwards. This program runs 2,000 tests in which both groups get the same product, so every “significant” result is false:
# 2,000 A/A tests: both groups get the same product, so every "significant" result is a false positive.
# Each test runs 14 days with 1,000 new users a day in each group, and a user converts with probability 1/8,
# drawn as three random bits that must all be 1 (whole-number arithmetic only, so every run prints the same).
import random
from math import sqrt
random.seed(7)
TESTS, DAYS, PER_DAY, Z95 = 2_000, 14, 1_000, 1.959963984540054
def conversions(n):
return (random.getrandbits(n) & random.getrandbits(n) & random.getrandbits(n)).bit_count()
def significant(conv_a, conv_b, n):
pooled = (conv_a + conv_b) / (2 * n)
z = (conv_b - conv_a) / n / sqrt(pooled * (1 - pooled) * 2 / n)
return abs(z) >= Z95
at_end = peeked = 0
for _ in range(TESTS):
a = b = 0
stopped = False
for day in range(1, DAYS + 1):
a += conversions(PER_DAY)
b += conversions(PER_DAY)
if not stopped and significant(a, b, day * PER_DAY):
stopped = True # a team that peeks every day stops here and ships
at_end += significant(a, b, DAYS * PER_DAY)
peeked += stopped
print(f"Read once, at day {DAYS}: {at_end / TESTS:.1%} of A/A tests look like a win or a loss")
print(f"Read every day, stopping at the first p < 0.05: {peeked / TESTS:.1%}")
for k in (5, 10, 20):
print(f"{k} independent metrics, each read once at 5 %: {1 - 0.95**k:.1%} chance at least one looks significant")Output
Read once, at day 14: 5.0% of A/A tests look like a win or a loss Read every day, stopping at the first p < 0.05: 21.8% 5 independent metrics, each read once at 5 %: 22.6% chance at least one looks significant 10 independent metrics, each read once at 5 %: 40.1% chance at least one looks significant 20 independent metrics, each read once at 5 %: 64.2% chance at least one looks significant
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 peeking.py
Runs on this device, in your browser. The first run downloads Python (about 13.5 MB), which is kept for the next runs.
Your run, in this browser
Read once at day 14, 5.0 % of the tests look like a win or a loss, as promised. Read every day and stopped at the first p below 0.05, 21.8 % do. With 20 independent metrics, the chance that at least one looks significant is 64.2 %. Kohavi, Deng and Vermeer count stopping when significant, removing data and reporting chosen segments among the habits that inflate false positives. Fix the length, the primary metric and any segments before the start.
Stopping on the best day
Wrong: check daily and ship on the first day that p drops below 0.05. Right: read the result once, at the planned end. In the simulation, daily checks turned 5.0 % false wins into 21.8 %.
Checkpoint
Check: The split check · Peeking, many metrics and late segments
A significant lift still has to be worth shipping
Statistical significance says the lift is probably not zero; it does not say the lift is worth its cost. Compare the interval with the smallest lift worth shipping, chosen before the test. Here that was 1 point: the interval, +0.05 to +1.75, reaches both sides of it, so the data fit a lift too small to matter as well as one above it. Reading A/B test results ends in a call: ship, ship to a segment named in advance, iterate, or stop. Answer the questions for a readout:
Answer the questions above: the table then marks what fits your case and says why the rest does not.
| Call | Risk if the result is wrong | Evidence it needs | Value it gives soon | Cost to undo |
|---|---|---|---|---|
| Ship to everyone Means: every user gets the change | Poor Every user gets a change that may not work, or may harm | Poor The most: interval above zero, guardrails within limits, checks passed | Good The whole gain, at once | Mixed Easy for a screen, hard for a price or a policy |
| Ship to one segment Means: only a group named before the test gets it | Mixed Only that segment is exposed | Poor The segment must have been named before the test | Mixed Part of the gain, sooner | Good A small group to roll back |
| Iterate and test again Means: change the idea or the test, then rerun | Good Nobody gets an unproven change | Good The next test brings it | Poor Weeks more before any gain | Good Nothing to undo |
| Stop Means: the idea goes no further | Good No one is exposed | Mixed A clear loss, or no effect worth its cost | Poor No gain at all | Good Nothing to undo |
How the answers decide
- “Did the split match the configured one, and did the other trust checks pass?” No, something is off: rules out Ship to everyone, Ship to one segment and Stop and adds a caveat to Iterate and test again. A split that does not match, or a broken metric, makes every number of the readout unreliable. Find the cause, fix it and run the test again before any call.
- “Where is the primary metric's 95 % interval?” Above zero, but reaching below that effect: adds a caveat to Ship to everyone and Ship to one segment. The lift is probably real but may be smaller than the effect worth shipping. Ship only if it costs little to run and keep; otherwise make it bigger and test again.
- “Where is the primary metric's 95 % interval?” It includes zero: rules out Ship to everyone and Ship to one segment. The data fit no effect at all, so no gain has been shown. Change the idea, or stop it if a bigger version is not worth building.
- “Where is the primary metric's 95 % interval?” Below zero: rules out Ship to everyone, Ship to one segment and Iterate and test again. The change made the primary metric worse. Stop, and keep what the test taught.
- “Did any guardrail cross its limit?” An interval reaches past a limit: adds a caveat to Ship to everyone and Ship to one segment. A guardrail may have crossed its limit. Run longer to narrow its interval, or fix the cause, before exposing more users.
- “Did any guardrail cross its limit?” Yes, one is clearly past its limit: rules out Ship to everyone and Ship to one segment. A guardrail clearly past its limit outweighs the primary win, by the rule set before the test. Fix the harm and test again, or stop.
- “Was the segment you want to ship to named before the test started?” No, it was found afterwards: rules out Ship to one segment. A segment found after the test is a new hypothesis. Test it again on that segment before shipping to it.
- “Can the change be undone within a day if it goes wrong?” No: adds a caveat to Ship to everyone and Ship to one segment. A change that cannot be undone needs stronger evidence, such as a replication or a stricter threshold, before it reaches users.
For this readout: the split passed, the interval clears zero but straddles the 1-point line, so ship only if the change costs little to keep, and otherwise test a bolder version.
Checkpoint
Check: Worth shipping, and the call
Practise: check a split
Write the split check, for any configured split, and run the sample tests:
Exercise · Medium · Python
Check a test's split before reading its result
Write srm_check(control, treatment, control_share=0.5, threshold=0.001) in split.py.
controlandtreatmentare the users each group received;control_shareis the share of users the test was configured to send to control (0.5for 50/50,0.9for a 90/10 test);- it returns a pair: the chi-square statistic of the observed counts against the configured split, and
Trueif the split passes, that is, if its p-value is at leastthreshold, orFalseif it is a sample ratio mismatch.
The chi-square statistic adds, for each group, (observed − expected)² ÷ expected, where expected is the total times the group's configured share. With two groups it has one degree of freedom, so its p-value is 2 × (1 − NormalDist().cdf(√chi_square)).
The default threshold, 0.001, is stricter than the 0.01 that the site's calculator uses: a stricter threshold flags fewer sound tests by chance, and needs a larger gap before it flags a mismatch.
srm_check(10_000, 10_000) # (0.0, True)
srm_check(821_588, 815_482) # (22.77..., False): about 1 in 549,000 by chanceThe sample tests also check a 90/10 test and a threshold of your own.
Starter code · split.py
def srm_check(control, treatment, control_share=0.5, threshold=0.001):
"""Return (chi_square, passes) for the observed split against the configured one."""
# Your code here.
return 0.0, TrueThe sample tests · test_split.py
from split import srm_check
def test_a_perfect_split_passes():
"""10,000 against 10,000 in a 50/50 test"""
assert srm_check(10_000, 10_000) == (0.0, True)
def test_the_papers_example_fails():
"""821,588 against 815,482: chi-square 22.77, far too unlikely by chance"""
chi_square, passes = srm_check(821_588, 815_482)
assert abs(chi_square - 22.77) < 0.01
assert passes is False
def test_a_small_wobble_passes():
"""10,000 against 9,900 is within chance"""
chi_square, passes = srm_check(10_000, 9_900)
assert abs(chi_square - 0.5025) < 0.001
assert passes is True
def test_a_ninety_ten_test():
"""90,200 against 9,800 in a 90/10 test: chi-square 4.44, p about 0.035"""
chi_square, passes = srm_check(90_200, 9_800, control_share=0.9)
assert abs(chi_square - 4.444) < 0.001
assert passes is True
def test_a_threshold_of_your_own():
"""the same 90/10 split fails a threshold of 0.05"""
assert srm_check(90_200, 9_800, control_share=0.9, threshold=0.05)[1] is FalseA hint
Compute the total first, then each group's expected count from its configured share: total * control_share and total * (1 - control_share). Add the two terms, take the p-value from NormalDist, and compare it with threshold.
Results of the sample tests
| Test | Result | Details |
|---|
What your code printed
The sample tests run on this device, in your browser (Pyodide): nothing is sent to mysmartcopilot.com. The first run downloads Python (about 13.5 MB), which is kept for the next runs. A check in your browser is feedback for you, not proof that the code is right for every input.
Then work through the self-check: it starts with the idea you have least practised and repeats a question you miss.
Practice that adapts to you
Practise this lesson
11 questions about 6 ideas of this lesson. Each answer is explained at once, and what you get right is kept in this browser.
Four readouts to decide, with the analysis run Pro: opens with a Pro, Premium or Ultimate pass
A readout is decided in a fixed order: the split, then the primary metric’s interval, then the guardrails, then the call. Here are four invented readouts, each analysed by the same program, with the call each supports.
In this part:
- A clean win on search ranking, shipped
- A win on a product page that cannot be trusted
- A win that a guardrail stops
- A flat test with one bright segment
- The four calls side by side
1 runnable example (Python) · about 5 minutes
Opening this part with your pass…
This part opens with any MySmartCoPilot pass.
- Pro pass · 1 week₹79$2.99
- 30 days₹179$5.99
- 1 year₹1,699$45
One payment; a pass never renews. The price shown is the total you pay: no fees or charges are added at checkout.
Passes can’t be bought right now, so this part can’t be opened at the moment. The free part above stands on its own. If you have a pass, enter your licence key.
It also opens every other in-depth part in Learn and every Pro tool. Sales are final, except where the law gives you a right to withdraw, for example in the EU and the UK (refund policy).
The follow-ups interviewers ask after a readout Pro: opens with a Pro, Premium or Ultimate pass
An interviewer who hears a readout asks what could still make it wrong and how sure you are. Six follow-ups come up again and again, and each needs its own evidence, with numbers wherever a number decides.
In this part:
- Is the lift novelty? Read it week by week
- How likely is a significant win to be false?
- What will it be worth in a year? Long-term holdouts
- Does it harm any group? An average can hide a loss
- How do you present the uncertainty to leadership?
1 runnable example (Python) · about 4 minutes
Opening this part with your pass…Pro part. It opens with the same pass as the part above.
Interview questions
The warm-up is answered in full. For the other three, say your call and the check before it, then compare:
-
Fresher to mid · Warm-up
Explain a p-value of 0.03 to a stakeholder without saying there is a 97 percent chance that the change works.
"If the change made no difference at all, a gap as large as the one we saw would show up by chance in about 3 tests out of 100. That is unusual enough that we treat the effect as probably real." Then add what the p-value does not say: it is not the chance that the change works, because that also depends on how often ideas like this one work at all, and it says nothing about how big the effect is. For the size, give the confidence interval in the stakeholder's own units, such as extra buyers per 100,000 users, from its low end to its high end. And say what the number assumes: that the test was run as planned, read once at its end, and that the split between the groups passed its check.
-
Mid to senior
A test configured 50/50 shows 10,000 users in control and 9,700 in treatment, and treatment wins. What do you conclude, and what do you do next?
-
Mid to staff
Treatment raises checkout conversion by 2 percent but increases refund requests by 5 percent. Do you ship? Explain how you decide.
-
Mid to senior
A test is flat overall, but new Android users show a large significant lift. The team wants to launch for that segment. What do you advise?
Flashcards · 7 cards
Recall it, then check
Say the answer to yourself before you look: recalling it is what makes it stick.
-
How do you compute and state the lift of an A/B test?
Treatment rate minus control rate: 10.9 % − 10.0 % = +0.90 points, which is 9.0 % in relative terms. Say which of the two forms you are quoting.
-
What does a p-value of 0.0375 mean, and what does it not mean?
If the change did nothing, a difference at least this large would turn up by chance about 4 times in 100. It is not a 96 % chance that the change works.
-
Why read the confidence interval before the p-value?
Its position says whether the lift is likely real, and its width says how sure you can be of the size: +0.05 to +1.75 points means probably above zero, of uncertain size.
-
What is a sample ratio mismatch, and what do you do when a test has one?
A split further from the configured one than chance allows, usually because something removed or added users in one group, such as a crash before logging, a redirect or bot filtering. Even 50.2 % to 49.8 % across 1.6 million users fails: stop, find the cause, fix it and run the test again.
-
How much does checking a test every day inflate false wins?
In the lesson's 2,000 simulated tests with no real effect, one look at day 14 gave 5.0 % false results; a look every day, stopping as soon as p fell below 0.05, gave 21.8 %.
-
Why do many metrics and segments chosen afterwards produce false wins?
Each extra look is another chance of a fluke: with 20 independent metrics, the chance that at least one looks significant is 64.2 %. Fix the length, the primary metric and any segments before the start.
-
Is a statistically significant lift always worth shipping?
No. Significance only says the lift is probably not zero; it says nothing about whether the lift repays its cost. Set the smallest lift worth shipping before the test and hold the interval against it: +0.05 to +1.75 against a 1-point bar reaches both sides, so ship only if the change is cheap to keep.
Key takeaways
- Read the lift with its interval: +0.90 points, between +0.05 and +1.75, is probably real and of uncertain size.
- A p-value is the chance of such a difference if the change did nothing, not the chance that it works.
- Check the split first: a mismatch such as 50.2 % to 49.8 % stops the readout.
- Daily peeking turned 5.0 % false wins into 21.8 %; fix the length, metrics and segments in advance.
- Compare the interval with the smallest lift worth shipping, then make the call.
Terms in this lesson
- confidence interval
- A range of effect sizes that fit the data, such as +0.05 to +1.75 points for a lift. A 95 % interval comes from a method that covers the true effect in 95 % of tests run the same way.
- p-value
- The probability of a difference at least as large as the one observed if the change did nothing and the test's assumptions hold. It is not the probability that the change works.
- peeking
- Checking a test for significance again and again while it runs, and stopping at the first significant result. It raises the share of false wins well above the 5 % that the threshold promises.
- sample ratio mismatch
- A split of users between a test's groups that differs from the configured split by more than chance allows, such as 50.2 % to 49.8 % across 1.6 million users configured as 50/50. It usually means the results cannot be trusted.
References
- Diagnosing Sample Ratio Mismatch in Online Controlled Experiments: A Taxonomy and Rules of Thumb for Practitioners (KDD 2019) (ACM (Fabijan, Gupchup, Gupta, Omhover, Qin, Vermeer and Dmitriev))
- A/B Testing Intuition Busters: Common Misunderstandings in Online Controlled Experiments (KDD 2022) (ACM (Kohavi, Deng and Vermeer))
- Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing (Cambridge University Press (Kohavi, Tang and Xu))
- Seven rules of thumb for web site experimenters (KDD 2014) (ACM (Kohavi, Deng, Longbotham and Xu))
- NIST/SEMATECH e-Handbook of Statistical Methods, 7.3.3: comparing two proportions (National Institute of Standards and Technology (NIST))
- statistics: Mathematical statistics functions (NormalDist) (Python Software Foundation)
Related tools
Report a problem with this lesson
Kept only in this browser. Your Learn progress