Data Analysis with Python (pandas, polars, DuckDB) Module 11 – Statistics an analyst uses every week
Comparing groups and A/B tests
Plan an A/B test before you look, test a lift by shuffling labels with NumPy, and report the effect with an interval, checked against SciPy.
What you will learn
- Define the metric, the groups and the decision rule of a comparison before looking at the data
- Run a permutation test for a difference in means or rates with NumPy
- Report effect sizes with intervals, not just p-values
- Recognise peeking, multiple comparisons and Simpson's paradox
Before you start
What you’ll learn · Before you start
What you will learn
- Define the metric, the groups and the decision rule of a comparison before looking at the data
- Run a permutation test for a difference in means or rates with NumPy
- Report effect sizes with intervals, not just p-values
- Recognise peeking, multiple comparisons and Simpson's paradox
Before you start
On this page
Comparing groups means asking whether a difference between them is larger than chance alone would produce. A/B tests make that question answerable: visitors are assigned at random between A and B, balancing other characteristics in expectation while ordinary chance imbalances remain. The method has four steps. Fix the metric, the groups and the rule for a decision before the data arrives. Measure the difference, the lift: A difference between comparison groups, stated either in original units such as percentage points or relative to a declared baseline.. Ask how often a lift that large would appear if the version made no difference, which a permutation test: A procedure that reassigns labels or pairings under a stated null and compares the observed statistic with the resulting null distribution. answers by shuffling the group labels. Then report the lift with an interval, so that a reader sees how large the effect might be and not only whether it exists. This lesson runs the whole method on an invented landing-page test with NumPy, and checks the answer against the classical tests.
A planned metric and stopping rule define the comparison
Comparisons and A/B tests are fair only if their rules were set before anyone saw the results. Write down four things first:
- The metric: the sign-up rate of new visitors, counted once per visitor.
- The groups: version A, the current page, and version B, the new one, assigned at random, half each.
- The sample size: how many visitors each version needs before the result is read. Here the plan is 5,000 per version, about a week of traffic.
- The decision rule: ship B if its sign-up rate is higher at p < 0.05.
The sample size decides which lifts the test can see. With a 5 % base rate, 5,000 visitors per version detect a lift of about 26 % with 80 % power, and a lift of 20 % needs 8,158 visitors per version.
A/B Test Sample Size Calculator Enter a 5 % base rate and a 20 % lift to see the 8,158 visitors per version, and how many days your traffic needs.Step through the same case and its result:
The drawing shows the whole scene; its 3 steps are listed below.
Every step as text
- Count new-visitor sign-ups once per visitor; fixing the unit avoids counting activity as if it were people.
- Plan random assignment to A and B with 5,000 visitors per version. Randomization balances other characteristics in expectation.
- Read the chosen metric once at the planned horizon. A p<0.05 rule is a decision criterion, while chance false alarms remain possible.
Checkpoint
Check: Planning the comparison
Shuffled labels put the observed lift against a null
The test ran: 250 of 5,000 visitors signed up with version A (5.0 %) and 300 of 5,000 with version B (6.0 %), a lift of 1 percentage point. If the version made no difference, the labels A and B would be arbitrary, and shuffling them should produce lifts like the real one quite often. The permutation test does exactly that: it pools the 10,000 visitors, deals the labels out again at random, recomputes the lift, and repeats.
The permutation test, step by step, on the landing-page test
Text description of the diagram
The test showed version A to 5,000 visitors, of whom 250 signed up (5.0 %), and version B to 5,000 visitors, of whom 300 signed up (6.0 %). The lift is 1.00 percentage point.
- Pool the 10,000 visitors, with their 550 sign-ups, and set the labels A and B aside.
- Shuffle the labels: give 5,000 visitors chosen at random the label A and the other 5,000 the label B.
- Compute the lift of these shuffled groups.
- Repeat steps 2 and 3 10,000 times. Because the labels were assigned by chance, the 10,000 shuffled lifts are centred on 0.
- The p-value is the share of shuffled lifts at least as far from 0 as the real lift of 1.00 point, in either direction: 304 of 10,000, so about 0.03.
import math
import numpy as np
# Invented results of a landing-page test: 1 = the visitor signed up, 0 = they did not.
a = np.repeat([1, 0], [250, 4_750]) # version A: 250 sign-ups from 5,000 visitors
b = np.repeat([1, 0], [300, 4_700]) # version B: 300 sign-ups from 5,000 visitors
observed = b.mean() - a.mean()
print(f"A {a.mean():.2%} B {b.mean():.2%} lift {observed * 100:+.2f} points ({observed / a.mean():+.0%})")
# If the version made no difference, the labels A and B are arbitrary: shuffle them and look again.
rng = np.random.default_rng(42)
pooled = np.concatenate([a, b])
diffs = np.empty(10_000)
for i in range(len(diffs)):
shuffled = rng.permutation(pooled)
diffs[i] = shuffled[len(a):].mean() - shuffled[: len(a)].mean()
extreme = np.sum(np.abs(diffs) >= abs(observed) - 1e-12) # as far from 0 as the real lift, in either direction
print(f"shuffles with a lift at least as large: {extreme} of {len(diffs):,}")
print(f"permutation p-value: {(extreme + 1) / (len(diffs) + 1):.4f}")
# The two-proportion z-test, the classical shortcut for the same question.
p = (a.sum() + b.sum()) / (len(a) + len(b))
z = observed / math.sqrt(p * (1 - p) * (1 / len(a) + 1 / len(b)))
print(f"z = {z:.2f}, two-sided p-value: {math.erfc(abs(z) / math.sqrt(2)):.4f}")Output
A 5.00% B 6.00% lift +1.00 points (+20%) shuffles with a lift at least as large: 304 of 10,000 permutation p-value: 0.0305 z = 2.19, two-sided p-value: 0.0283
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 permutation_test.py
Only 304 of the 10,000 shuffles produced a lift at least as far from zero as the real one, so the p-value is about 0.03: a result this extreme would turn up about 3 times in 100 if the version made no difference. Adding 1 to both counts, as the code does, keeps a permutation p-value from ever being exactly zero. The two-proportion z-test, the textbook shortcut for rates, gives 0.028. The answers differ a little, because the z-test is an approximation and the shuffles are random, but they say the same thing. By the rule written down before the test, B ships.
A/B Test Significance Calculator Enter 5,000 visitors and 250 sign-ups for A, 5,000 and 300 for B: the calculator's z-test gives p = 0.028.Step through the same case and its result:
The drawing shows the whole scene; its 3 steps are listed below.
Every step as text
- A has 250 sign-ups in 5,000 visitors and B has 300 in 5,000, so B−A is 1.00 percentage point.
- Pool the 550 sign-ups and 9,450 non-sign-ups; each shuffle changes labels while preserving every outcome and the two group sizes.
- 304 of 10,000 shuffles have an absolute lift at least as large. Including the observed case gives (304+1)/(10,000+1), approximately 0.0305.
Key idea
Checkpoint
Check: The permutation null
Choose the comparison for its question and assumptions
The permutation test is one of several ways of comparing groups, and each answers a slightly different question.
| Criterion | Permutation test | z-test or chi-square test | Welch's t-test | Bootstrap interval |
|---|---|---|---|---|
| Question it answers | Is the lift larger than shuffled labels produce? | Is the gap between two rates larger than chance? | Is the gap between two means larger than chance? | How large is the lift, and how sure are we? |
| Works for | Any statistic: rates, means, medians | Counts of yes and no | Numbers per visitor, such as revenue | Estimates with suitable bootstrap conditions |
| Relies on | Random assignment | Large counts in every cell | Large samples or roughly normal means | A valid resampling design and suitable statistic; assumptions checked |
| Result | p-value | p-value | p-value | An interval for the lift |
- The z-test and the uncorrected chi-square test compare two rates through a normal approximation; on a two-by-two table they give the same two-sided p-value.
- Welch’s t-test compares two means without assuming the groups have equal spread.
- A bootstrap interval resamples each group to show the range of lifts the data supports.
Three traps undo even a correct test. optional stopping: Choosing when to end an analysis from the observed outcomes; its inferential properties depend on a stopping rule evaluated for that use. means checking the result every day and stopping once p drops below 0.05, which turns chance into false wins. Multiple comparisons, testing many metrics or versions at once, make at least one false alarm likely. Simpson’s paradox is a lift in every segment that reverses in the total, when the segments are mixed differently in the two groups.
Step through the comparison and identify its result.
The drawing shows the whole scene; its 3 steps are listed below.
Every step as text
- For two large-count rates, the two-proportion z-test and uncorrected 2×2 chi-square calculation give the same two-sided p-value.
- Welch compares means without equal-variance assumptions. A permutation procedure needs a valid exchangeable-label null for the chosen statistic.
- A bootstrap interval estimates the range of effect sizes rather than assigning a probability that B is better. Peeking, metric selection and segment mixes need separate design checks.
Checkpoint
Check: Selecting a comparison
Effect sizes with intervals, and the classical tests side by side Pro: opens with a Pro, Premium or Ultimate pass
A p-value says whether a lift is surprising, not how large it is or whether it is worth having. This part puts an interval on the lift, runs SciPy’s classical tests on the same data and turns a significant result into a decision with numbers.
In this part:
- A confidence interval reports lift in points and per cent
- SciPy’s tests side by side, and why they agree here
- A significant lift that is not worth shipping, in numbers
- Your turn: analyse an A/B test end to end
4 diagrams · 6 runnable examples (Python) · about 10 minutes
Opening this part with your pass…
This part opens with any MySmartCoPilot pass.
- Pro pass · 1 week₹79$2.99
- 30 days₹179$5.99
- 1 year₹1,699$45
One payment; a pass never renews. The price shown is the total you pay: no fees or charges are added at checkout.
Passes can’t be bought right now, so this part can’t be opened at the moment. The free part above stands on its own. If you have a pass, enter your licence key.
It also opens every other in-depth part in Learn and every Pro tool. Sales are final, except where the law gives you a right to withdraw, for example in the EU and the UK (refund policy).
Peeking, many comparisons and Simpson's paradox, reproduced Pro: opens with a Pro, Premium or Ultimate pass
Three habits turn honest data into wrong decisions: stopping a test when it first looks good, judging it on many metrics at once, and pooling segments that the two versions saw in different mixes. Each one is reproduced here in code that runs, with a fix that holds up.
In this part:
- Peeking reproduced: daily checks on 10,000 tests with no real difference
- Ten metrics, one false alarm: Holm’s correction worked through
- Simpson’s paradox reproduced: B wins on every device and loses overall
3 diagrams · 3 runnable examples (Python) · about 7 minutes
Opening this part with your pass…
This part opens with any MySmartCoPilot pass.
- Pro pass · 1 week₹79$2.99
- 30 days₹179$5.99
- 1 year₹1,699$45
One payment; a pass never renews. The price shown is the total you pay: no fees or charges are added at checkout.
Passes can’t be bought right now, so this part can’t be opened at the moment. The free part above stands on its own. If you have a pass, enter your licence key.
It also opens every other in-depth part in Learn and every Pro tool. Sales are final, except where the law gives you a right to withdraw, for example in the EU and the UK (refund policy).
Practise: decide from the stated evidence
Practice that adapts to you
Practise this lesson
10 questions about 10 ideas of this lesson. Each answer is explained at once, and what you get right is kept in this browser.
Interview questions
-
Fresher to mid · Warm-up
What does a p-value of 0.03 in an A/B test tell you, and what does it not tell you?
It says that if both versions had the same true conversion rate, a lift at least as large as the one observed would appear in about 3 % of tests of this size under that null model. A small p-value indicates incompatibility with the specified null and assumptions; it does not establish a real effect. It is not the probability that B is better, nor the probability that the result is a fluke, and it says nothing about the size of the lift: 0.1 point on millions of visitors and 3 points on a few thousand can both give p = 0.03. It also assumes the test ran as planned, with random assignment and one look at the end; daily peeking or many metrics make small p-values far more common. Report the lift with its interval beside the p-value.
-
Fresher to mid
Your A/B test shows version B converting 6.0 % against 5.0 % for A, with 5,000 visitors per version and p = 0.03. The product manager wants to ship B today. What do you check first, and what exactly do you report?
-
Mid to senior
A team checks its A/B dashboard every morning and stops each test as soon as p drops below 0.05. Over a year, most of its "winning" changes show no effect after launch. Explain what is happening and how you would redesign the process.
-
Fresher to senior
Version B beats version A among phone visitors and among desktop visitors, but loses in the total. The designer says B wins everywhere; finance says B lost. Who is right, what do you check, and what do you report?
Key takeaways
- Write the metric, the groups, the sample size and the decision rule down before the test starts.
- A permutation test shuffles the group labels to see how often chance produces the observed lift; NumPy is enough.
- A p-value measures surprise, not size or value: report the lift with an interval.
- Peeking and metric selection can inflate false wins; differently mixed segments can confound a pooled comparison.
Terms in this lesson
- lift
- A difference between comparison groups, stated either in original units such as percentage points or relative to a declared baseline.
- optional stopping
- Choosing when to end an analysis from the observed outcomes; its inferential properties depend on a stopping rule evaluated for that use.
- permutation test
- A procedure that reassigns labels or pairings under a stated null and compares the observed statistic with the resulting null distribution.
References
- scipy.stats.permutation_test (SciPy)
- scipy.stats.ttest_ind (SciPy)
- scipy.stats.fisher_exact (SciPy)
- scipy.stats.chi2_contingency (SciPy)
- Random Generator (numpy.random) (NumPy)
- Two-proportion tests (NIST/SEMATECH e-Handbook of Statistical Methods) (National Institute of Standards and Technology)
- Permutation p-values should never be zero: calculating exact p-values when permutations are randomly drawn (arXiv (Statistical Applications in Genetics and Molecular Biology))
- A Simple Sequentially Rejective Multiple Test Procedure (Scandinavian Journal of Statistics)
- ASA Statement on Statistical Significance and P-Values (American Statistical Association)
Related tools
Report a problem with this lesson
Kept only in this browser. Your Learn progress