# 7 A/B Testing Statistical Mistakes That Invalidate Your Results > The most common statistical mistakes in A/B testing that cause product teams to ship wrong decisions — and how to avoid each one before you run your next test. - **Published:** 2025-08-05 - **Author:** Clixo - **Reading time:** 5 min read - **Tags:** a/b-testing, statistics, growth-engineering, experimentation - **Canonical URL:** https://clixo.sh/blog/ab-testing-statistical-mistakes-that-invalidate-results Running an A/B test is easy. Running one that produces trustworthy results is not. The majority of failed experiments are not caused by bad ideas or insufficient traffic — they are caused by statistical errors that were baked into the design before a single user was assigned. These are the mistakes that most commonly invalidate results, how they happen, and what to do instead. ## Statistical Mistakes in A/B Testing That Teams Repeat Constantly ### 1. Peeking at results and stopping early This is the single most common and most damaging mistake in experimentation. A team launches a test, checks results daily, and stops as soon as they see a significant p-value — usually because the uplift looks exciting. The problem: if you check your results repeatedly and stop when significance appears, your false positive rate explodes well above the 5% your threshold implies. You are not running a test with 95% confidence. You are running a slot machine. The fix: decide your stopping date before the experiment launches. Do not look at results until that date. If you need to peek, use a sequential testing method (like always-valid p-values) designed for interim analysis. ### 2. Not computing sample size before launch A team runs a test for two weeks, sees a 3% lift, and calls it a winner. The confidence interval spans from -4% to +10%. The result is noise. Sample size must be calculated before launch, using your baseline conversion rate, minimum detectable effect, desired power, and significance threshold. If your traffic cannot support the test within a reasonable time frame, either increase the MDE (only care about larger effects) or narrow the audience. Running an underpowered experiment does not just produce inconclusive results — it produces confident-sounding wrong conclusions. ### 3. Testing multiple metrics and declaring a winner on the best-looking one If you run an experiment and check 20 metrics, the probability that at least one will appear significant by chance alone is very high — even if your variant had zero real effect. This is called multiple comparisons inflation. The fix: designate one primary metric before launch. Track secondary guardrails (error rates, page speed, downstream funnel steps), but your test lives or dies on the primary. If you want to measure many things, apply a correction like Bonferroni or use a Bayesian framework that handles multiple outcomes natively. ### 4. Sample ratio mismatch (SRM) You allocated 50/50 between control and variant. After two weeks, control has 12,000 users and variant has 9,400. Something went wrong with assignment. Sample ratio mismatch (SRM) is an often-missed validity check. When the observed split differs significantly from the expected split, your data is contaminated. Common causes: bots being filtered post-assignment in only one variant, caching serving stale pages to control users, or a bug in the assignment logic itself. Always run an SRM check before reading results. If the split is off by more than a few percent, the experiment data is not trustworthy regardless of what the metrics show. ### 5. Novelty and primacy effects Users behave differently toward new things. A redesigned interface may spike engagement in week one because users are exploring it, not because it is genuinely better. Conversely, users who are deeply habituated to an existing UI may initially perform worse with a change even if the change is correct long-term. Neither effect is stable. If you run a test for only a few days, you may be measuring novelty rather than real preference. The fix: run experiments for at least one full business cycle, typically a minimum of seven days, with two weeks preferred for anything that touches user habits. ### 6. Network effects and interference between users Standard A/B testing assumes that each user's behavior is independent of other users. This assumption breaks in products with social features, referral programs, shared workspaces, or marketplaces. If a treated user can influence a control user — by inviting them, by affecting inventory they both see, or by changing their own behavior in a way that ripples outward — your experiment is measuring a contaminated signal. For products where interference is a real risk, cluster-based randomization (assigning entire cohorts or social groups to the same variant) is the appropriate method, at the cost of requiring significantly more traffic. ### 7. Changing the experiment while it runs Resizing images, changing copy in both variants, modifying the audience targeting, or pushing a backend change that affects all users mid-experiment invalidates the test. Once an experiment is live, treat it as frozen infrastructure. If a critical bug requires a change, stop the experiment, fix the issue, and restart with fresh assignment. Do not try to salvage data from a contaminated run. ## How to Build Cleaner Experimentation Practice ```mermaid flowchart TD A[Define primary metric] --> B[Calculate sample size] B --> C[Set stopping date] C --> D[Launch experiment] D --> E["SRM check — day two"] E --> F{Split ratio correct?} F -->|No| G[Invalidate run and restart] F -->|Yes| H[Wait until stopping date] H --> I[Read results once] ``` Most of these mistakes are preventable with a pre-launch review process. Before any experiment goes live, verify: - Primary metric is defined and logged - Sample size calculation is documented - Stopping date is set - SRM check is scheduled for day two - No dependent experiments are running on the same population The discipline of experimentation is mostly the discipline of doing boring things correctly, before you are excited about the results. If your team wants to build an experimentation system with statistical rigor baked in from the start, [Clixo builds growth infrastructure](https://clixo.sh/#contact) for product and engineering teams. --- Clixo · 1141 W Bryn Mawr Ave, Itasca, IL 60143, US · [hello@clixo.sh](mailto:hello@clixo.sh) [Start a build](https://clixo.sh/#contact) · [All services](https://clixo.sh/services) · [Agent guide (llms.txt)](https://clixo.sh/llms.txt)