A/B Test Significance Calculator

About the A/B Test Significance Calculator

Find out whether the difference between the versions of an A/B test is real or could just be chance. For conversions — visitors who bought, signed up or clicked — the calculator runs the two-proportion z-test; for averages such as revenue per visitor it runs Welch’s t-test from each group’s size, mean and standard deviation (or from the raw values you paste).

You get the p-value and whether it is significant at your confidence level, the uplift with its confidence interval, the difference in percentage points, an optional Bayesian chance to beat the control and the smallest change the test could reliably detect. With very few conversions it switches to Fisher’s exact test. Compare up to five variants with one control — Holm’s correction keeps the chance of a false winner at your α — and every test is checked for a sample ratio mismatch, the traffic-split problem that silently breaks many tests. Everything runs in your browser.

Common uses

• Deciding whether a new landing page, checkout or headline really converts better before rolling it out.
• Checking revenue per visitor or average order value, not just conversion rate, for a pricing or shipping test.
• Reviewing a test with several variants without being fooled by the one that won by luck.
• Spotting a broken test early: a traffic split that does not match the plan points to a bug in assignment or tracking.

The formulas

• Conversions — two-proportion z-test: z = (p̂_B − p̂_A) ÷ √(p̂(1 − p̂)(1/n_A + 1/n_B)), with the pooled rate p̂ = (x_A + x_B) ÷ (n_A + n_B) (NIST/SEMATECH e-Handbook of Statistical Methods, §7.3.3). The p-value is 2 × P(Z > |z|) for a two-sided test, P(Z > z) for a one-sided one.
• Difference in percentage points: p̂_B − p̂_A ± z × √(p̂_A(1 − p̂_A)/n_A + p̂_B(1 − p̂_B)/n_B).
• Relative uplift: p̂_B ÷ p̂_A − 1, with the log-ratio interval exp(ln(p̂_B/p̂_A) ± z × √(1/x_B − 1/n_B + 1/x_A − 1/n_A)) − 1 (Katz et al., Biometrics 34), which can never go below −100%.
• Very few conversions: when an expected count — visitors × pooled rate, or visitors × (1 − pooled rate) — is below 5 (Cochran’s rule), the z-test’s normal approximation is unreliable. The p-value then comes from Fisher’s exact test: given the totals, it adds up the probabilities of every possible table that is no more likely than the one observed (R documentation of fisher.test); one-sided, the chance of at least as many conversions in the variant. The difference then gets Newcombe’s hybrid score interval (Newcombe 1998, Statistics in Medicine 17), which stays sensible with zero conversions.
• Averages — Welch’s t-test: t = (m_B − m_A) ÷ √(s_A²/n_A + s_B²/n_B) with the Welch–Satterthwaite degrees of freedom (Welch 1947, Biometrika 34). The uplift interval uses the delta method described by Kohavi, Tang & Xu, Trustworthy Online Controlled Experiments (Cambridge University Press, 2020).
• Several variants: each is compared with the control, and Holm’s step-down method (Holm 1979, Scandinavian Journal of Statistics 6) adjusts the p-values: sort them, multiply the smallest by the number of comparisons m, the next by m − 1 and so on, keeping the running maximum.
• Bayesian chance to beat the control: with a uniform Beta(1, 1) prior, each rate’s posterior is Beta(1 + conversions, 1 + non-conversions), and Pr(p_B > p_A) is computed exactly with the closed form in Evan Miller’s Formulas for Bayesian A/B Testing. For averages it uses normal posteriors with flat priors, an approximation.

Reading the result

The p-value is how often a difference at least this large would appear if the versions performed exactly the same. Below α (0.05 at 95% confidence) the result is called significant. It is not the probability that the variant is better, and a significant result can still be a small one — look at the interval: it shows the range of uplifts that fit the data.

Not significant does not mean “no effect”: it means the test cannot tell. The Detectable (80% power) figure shows the smallest change these visitor numbers could reliably pick up; if a smaller change would still matter to you, the test needs more visitors — the sample size calculator tells you how many.

Sample ratio mismatch (SRM)

When the groups get a noticeably different share of the visitors than planned, something in the experiment is broken: a redirect that loses visitors, a bot filter that only hits one version, a tracking tag that fires late. Fabijan et al., Diagnosing Sample Ratio Mismatch in Online Controlled Experiments (KDD 2019), report that about 6% of experiments at Microsoft had an SRM; a chi-square test detects it.

The calculator runs that chi-square goodness-of-fit test on the visitor counts against your planned split (equal unless you enter one) and flags p < 0.01 — the threshold the SRM Checker browser extension used (SRM Checker FAQ). With an SRM, do not trust the result, whichever way it points, until you have found the cause.

Do not stop at the first significant result

These tests assume you looked at the data once, after a sample size you fixed in advance. Checking every day and stopping as soon as the result turns significant makes false winners far more likely: in Evan Miller’s example in How Not To Run an A/B Test, testing after every observation and stopping at the first result significant at 5% (or after 150 observations) declared a winner 26.1% of the time between two identical versions. Decide the sample size first with the A/B test sample size calculator, and run whole weeks so every weekday is included.

Limitations

• Each visitor must be counted once and belong to one group. For conversions, count visitors who converted (not orders); for several orders per visitor use Averages.
• The z-test and t-test rely on normal approximations. With an expected count below 5 the conversion test switches to Fisher’s exact test (which is conservative: its p-values tend to be a little high), and the calculator warns when a group has fewer than 10 conversions or non-conversions, or fewer than 30 visitors for averages. Revenue data with a few very large orders need large samples.
• The p-values are valid for one look at a sample size fixed in advance — not for a test that was stopped when it first looked significant.
• Holm’s correction covers the comparisons with the control. The confidence intervals are for each comparison on its own and are not adjusted.
• The Bayesian chance uses uninformative priors; it says how likely the variant is to be better at all, not by how much.

Privacy

Everything happens in your browser. What you enter or open here is not uploaded or stored by MySmartCoPilot.