AB Test Calculator (Z-test)

This statistical significance calculator calculates if the test Variation B has beat the Control A, shows advanced statistics, and outputs recommendations such as how many additional days are recommended to reach a big enough sample size (e.g. how many conversions you need in A/B testing).
In addition, you get a permalink to share/save your test results.

Settings

• {Significance}: Choose the level of confidence (95% is standard). This means there's a 90%, 95% or 99% chance the results are due to an actual difference between A and B, and not to chance.
• {Power}: Choose the level of statistical power (80% is standard). In A/B testing, power reflects the sample size.
• {Hypothesis}: Choose whether you want to know if the variation performs significantly worse (2-tailed) or not (1-tailed).
• Modify {Variation names}, {Test name} and {Duration} to your needs.

Understand results

• A winner is declared if {significance} is >= {your settings}.
• Recommendations for {Additional users needed} and {Additional days needed}. This recommendation is based on calculating the Minimal Detectible Effect (MDE) for your current result and settings.

Share / save

Click the {Share / Save Results} button to get a permalink containing all your data & settings.

AB test calculator Excel integration

No more copy & paste!
Create hyperlinks in your local Excel file and let this calculator do all the work for you.
Download the AB test calculator Excel demo.

Credits

• Evan Miller: Sample size calculator (recommended sample size)
• Annemarie Klaassen: A/B deviation graph

Created by

Nico Fonce