Logo

MonoCalc

/

Ad A/B Test Significance Calculator

Social Media

The two variants you tested

Enter how many people saw each variant and how many of them acted. Shorthand such as 5,000 or 12k works. Everything is computed in your browser — no account is connected and no ad platform is queried.

Variant A

Control
Everyone who saw it
How many acted

Variant B

Challenger
Everyone who saw it
How many acted
Pick it before you look at the data
Is there any difference, in either direction?
Changes the wording only, never a number
Variant wins
At 5.00% versus 4.00%, Variant's lift of +25.0% against Control is statistically significant at the 95% level (p = 0.0159, 98.4% confidence the difference is not chance). On this evidence the true lift sits between +4.2% and +49.9%.

Control rate

4.00%

200 / 5,000

Variant rate

5.00%

250 / 5,000

Relative lift

+25.0%

B against A

z statistic

2.4119

pooled standard error

p-value

0.0159

two-tailed

Confidence

98.41%

not chance

What that confidence figure is, and is not
98.41% is the confidence that a gap this large would not have appeared by chance if the two variants were truly identical. It is not the probability that Variant is better — that is a different (Bayesian) question, and this test does not answer it.

The two rates, with their uncertainty

Each bar is an observed rate; each whisker is the 95% Wilson score interval around it — the range of true rates that could plausibly have produced that count. Overlapping whiskers are the picture of an unsettled test.

0.0%1.6%3.2%4.7%6.3%Control200 / 5,0004.00%3.49%–4.58%Variant250 / 5,0005.00%4.43%–5.64%Bars: observed rate. Whiskers: 95% Wilson score interval.

Wilson score intervals rather than the textbook Wald interval: a Wald interval on a low rate extends below zero, which cannot be drawn on a percentage scale without either lying or clipping. Note that two overlapping intervals do not by themselves prove the difference is insignificant — the test below is the arbiter, and this chart is the intuition.

The difference, measured against zero

The dot is the observed gap in percentage points. The bar is the 95% interval around it. Everything the test concluded is visible in one fact: whether that bar touches the zero line.

-0.22pp+0.34pp+0.91pp+1.47pp+2.03ppno difference+1.00pp95% confidence interval on the difference, in percentage points. The bar sits clear of zero.Two-tailed test.

Every number behind the verdict

QuantityValueHow it is built
Pooled rate4.5000%(c1 + c2) / (n1 + n2) — the single rate the null hypothesis claims both variants share
Pooled standard error0.0041461√(p̄(1−p̄)(1/n1 + 1/n2)) — used for the test statistic only
Unpooled standard error0.0041449√(p1(1−p1)/n1 + p2(1−p2)/n2) — used for the difference interval only
z statistic2.4119(p2 − p1) / SE_pooled
Critical value1.9600Two-sided cut-off at 95%, computed from the inverse normal
p-value0.01592(1 − Φ(|z|)) — significant when below 0.05
Absolute difference+1.00ppp2 − p1, in percentage points
Difference, 95% interval+0.19pp to +1.81pp(p2 − p1) ± z SE_unpooled
Relative lift+25.0%(p2 − p1) / p1
Lift, 95% interval+4.2% to +49.9%Delta method on ln(p2/p1), not the absolute interval divided by p1
Smallest cell200The normal approximation wants 5 at the absolute minimum and 10 to be comfortable
Read the result once, at the end
Fix the sample size before the test starts and read the result once, at the end. Checking daily and stopping the moment the number goes green inflates the false-positive rate far above the level you selected: the p-value assumes exactly one look at the data.

Where do these four numbers come from?

Any report that counts how many people saw something and how many acted will do. The test does not care what the two numbers are called, only that both variants were counted the same way over the same window.

Meta Ads Manager

Split the ad set by creative and read impressions or reach against results for each ad.

Google Ads

Ad variation and experiment reports give impressions and conversions per variant.

GA4

Landing-page reports give sessions and key events per page, which is the same pair of numbers.

Your email platform

Subject-line splits report delivered messages and unique clicks per variant.

Any funnel you own

Anything that counts how many saw it and how many acted works — the test does not care what the two numbers are called.

Two-proportion z-test. The test statistic uses the pooled standard error (the null hypothesis says both arms share one rate); the confidence interval on the absolute difference uses the unpooled standard error; the interval on relative lift uses the delta method on the log risk ratio. Per-variant error bars are Wilson score intervals.

About This Tool

Ad A/B Test Significance Calculator – Real Win or Noise?

Creative A pulled 200 clicks from 5,000 impressions. Creative B pulled 250 from the same 5,000. B is ahead by a quarter, the dashboard is green, and someone wants to kill A this afternoon. The only question worth asking first is whether that gap would survive being run again, and that is what an A/B test significance calculator exists to answer. Most free ones print a bare “Significant!” and stop. This one shows you the uncertainty, because the uncertainty is the finding.

The two-proportion z-test, in four lines

Two rates, two sample sizes, one test. The null hypothesis is that both creatives share a single true rate, so the two arms get pooled into one estimate before the standard error is computed.

p1 = c1 / n1 4.00% p2 = c2 / n2 5.00% p_pool = (c1 + c2) / (n1 + n2) 0.045 SE_pool = √(p_pool(1 − p_pool)(1/n1 + 1/n2)) 0.0041461 z = (p2 − p1) / SE_pool 2.4119 p = 2(1 − Φ(|z|)) 0.0159

A p-value of 0.0159 means a gap this large would appear by chance in about 1.6% of tests where the two creatives were truly identical. Flip it round and you get the figure marketers quote: 98.41% confidence the difference is not chance. That wording is deliberate and worth guarding. It is emphatically not “a 98% probability that B is better” — that is a Bayesian claim about your specific test, and a frequentist p-value cannot make it.

Why the interval matters more than the verdict

“Significant” is one bit of information. The confidence interval is the whole distribution of plausible answers, and it is what you actually plan against. For the example above the absolute difference is +1.00pp with a 95% interval of +0.19pp to +1.81pp, and the relative lift is +25.0% with an interval of +4.2% to +49.9%. B won — but the honest forecast is “somewhere between a rounding error and a transformation”, not “25%”.

Note that the interval uses the unpooled standard error while the test uses the pooled one. That is not an inconsistency: pooling assumes the null hypothesis is true, which is right for a test statistic and wrong for an interval that has to cover a difference which may be real. The lift interval, meanwhile, comes from the delta method on the log risk ratio rather than from dividing the absolute interval by p1 — the shortcut produces a symmetric range where the truth is skewed, and it understates the upside every time.

The peeking problem

The single most common way marketers fool themselves is checking the test every morning and stopping the moment it turns green. A p-value is computed on the assumption of exactly one look at the data. Armitage, McPherson and Rowe showed in 1969 that repeating a fixed-level test on accumulating data pushes the false-positive rate above the nominal level, and it climbs with every extra look. Fix the sample size before the test starts, run to it, read the result once.

A green number is not a finish line
If you find yourself checking daily, the test is not telling you when to stop — you are. Decide the stopping point in advance, or use a method built for continuous monitoring, but do not do both halves of the wrong thing.

Statistical significance is not practical significance

With enough traffic, almost any difference becomes statistically detectable. At ten million impressions per arm, moving a 0.50% rate to 0.51% — a gap of one hundredth of a percentage point — clears the 95% bar comfortably at z = 3.15, and can still be worth nothing, because rebuilding the creative costs more than the extra conversions return. Before running the test, write down the smallest lift that would actually change your decision. If the confidence interval sits entirely below that number, the correct conclusion is “B wins and it does not matter”.

Small samples, and the warning you should not skip

The z-test approximates a binomial with a normal curve, and that approximation needs enough successes and failures in both arms — conventionally five at the absolute minimum and ten to be comfortable. Three conversions out of forty thousand impressions is an enormous sample and nowhere near normal. This is why the calculator checks all four cells and says so loudly when one is thin: a confident p-value computed off three conversions is arithmetic dressed up as evidence.

Reading a null result properly

An inconclusive test is not a tie. It is a statement that your data cannot separate the options at the precision you asked for. A gap of +8% with an interval running from −8.7% to +27.7% leaves a real win entirely on the table — keep going. An interval of −1% to +1% is a genuine answer: the creatives are interchangeable, so decide on production cost instead and spend the traffic on a bolder test.

Frequently Asked Questions

Is the Ad A/B Test Significance Calculator free?

Yes, Ad A/B Test Significance Calculator is totally free :)

Can I use the Ad A/B Test Significance Calculator offline?

Yes, you can install the webapp as PWA.

Is it safe to use Ad A/B Test Significance Calculator?

Yes, any data related to Ad A/B Test Significance Calculator only stored in your browser (if storage required). You can simply clear browser cache to clear all the stored data. We do not store any data on server.

How does this ad A/B test significance calculator work?

You enter four numbers — how many people saw each variant and how many converted on each — and the page runs a two-proportion z-test over them. The test statistic uses the pooled standard error, because the null hypothesis being tested is that both creatives share one true rate; the confidence interval on the absolute difference uses the unpooled standard error, because an interval has to cover a difference that may be real. Relative lift gets its own interval from the delta method on the log risk ratio, and each variant's error bar is a Wilson score interval. Everything runs in your browser: no account is connected, no ad platform is queried, and nothing you type leaves the page.

What does 95% confidence actually mean here?

It means that if the two creatives really performed identically, a gap at least this large would show up in fewer than 5% of tests by pure chance. That is a statement about how often this procedure is fooled, not about your particular test. It is specifically not the probability that the variant is better — that is a Bayesian claim, and a frequentist z-test cannot make it no matter how the result is worded. The tool therefore labels the figure as confidence the difference is not chance.

Why is 95% the convention rather than some other level?

There is no mathematical reason for it. The 5% threshold is a historical convention from early twentieth-century agricultural statistics that stuck, and 90% and 99% are the two neighbours people reach for when they want to be more or less cautious. Choose the level before you look at the data. Picking it afterwards, once you can see which side of the line the p-value fell on, is not a choice of rigour but a choice of result.

How long should I run a test before checking the result?

Decide the sample size before the test starts, run to it, and read the result once. The p-value assumes exactly one look at the data. Checking every morning and stopping the moment the number turns green is called peeking, and it inflates the false-positive rate above the level you selected: Armitage, McPherson and Rowe showed in 1969 (Journal of the Royal Statistical Society, Series A, 132:2) that repeating a fixed-level test on accumulating data drives the chance of a spurious "significant" result upward with every extra look. Running a test to a fixed, pre-declared end is the cheapest reliability you can buy.

Why does it warn me when my conversion counts are small?

The two-proportion z-test approximates a binomial distribution with a normal one, and that approximation needs enough successes and failures in both arms — conventionally at least five of each, comfortably ten. Three conversions out of forty thousand impressions is an enormous sample and still nowhere near normal. When any of the four cells falls below the threshold the calculator says so above the verdict, because a p-value computed on that data is arithmetic rather than evidence.

What should I do with an inconclusive result?

Either keep collecting data or accept that the difference, if any, is too small to matter to you. An inconclusive test is not a tie — it is a statement that the data cannot separate the two options at the precision you asked for. Look at the confidence interval on the difference: if it spans −7% to +26%, a meaningful win is still perfectly possible and more data will resolve it. If it spans −1% to +1%, you have your answer already, which is that the creatives are interchangeable and the decision should be made on cost or effort instead.