Ecommerce A/B Testing: What Should You Test First?
How to run ecommerce A/B tests you can trust: hypotheses, prioritization, what to test first, duration, sample size, false positives and analysis.
Quick answer
Test first where your own data shows a problem, on high-traffic pages close to purchase: product pages, cart and key landing pages. Start every test with a hypothesis based on evidence, prioritize by reach, likely impact and effort, and choose one primary metric plus guardrails such as revenue per visitor. Calculate the sample size before starting, run for full weekly cycles and don't stop early when results look good, because peeking inflates false positives. A/B testing doesn't automatically improve conversion. It tells you which changes work, including which ones would have hurt.
What A/B Testing Is
An A/B test randomly splits visitors between the current version (control) and a changed version (variant) and compares a metric. Because the groups are randomized and run at the same time, differences in outcome can be attributed to the change rather than to season or traffic mix, provided the test has enough data and is analyzed properly.
Start With a Hypothesis
A good hypothesis links evidence, a change and an expected outcome: “Because recordings show mobile shoppers scrolling past the size guide link and support tickets ask about fit, adding size guidance next to the size selector will increase mobile add-to-cart rate.” Tests without a hypothesis produce results nobody can learn from.
Evidence comes from funnel analysis, recordings, heatmaps, surveys, reviews and user tests. See ecommerce conversion funnel and ecommerce heatmaps.
How to Prioritize Tests
Rank candidate tests on four questions rather than on opinion.
| Question | Why it matters |
|---|---|
| How many shoppers see it? | More traffic means faster, more reliable results |
| How strong is the evidence? | Tests grounded in observed problems win more often |
| How big could the effect be? | Small cosmetic changes rarely produce detectable effects |
| How much effort and risk? | Cheap, low-risk tests can run first |
What to Test First
Common starting points, each only worth testing if your evidence points to it.
| Page | Test ideas |
|---|---|
| Product page | Delivery cost and date near the button; review placement; size or fit guidance; image order; offer presentation |
| Homepage | Clearer value proposition; category entry points; search prominence |
| CTAs | Label clarity; sticky add-to-cart on mobile; express payment visibility |
| Cart | Cost estimate; free-delivery messaging; cross-sell placement; checkout button position |
| Checkout | Payment method order; express checkout placement; reassurance near payment (within platform limits) |
| Mobile | First-screen content; filter access; gallery height |
Choosing Metrics
Pick one primary metric that the change should move, such as add-to-cart rate for a product page test. Add guardrail metrics that shouldn't get worse, such as revenue per visitor, average order value and checkout completion. A variant that raises add-to-cart while lowering purchases isn't a win.
Want a testing program built on evidence?
ZSpace researches your funnel, writes testable hypotheses and runs experiments you can trust.
Sample Size and Test Duration
Before starting, decide the smallest effect worth detecting and use a sample size calculator with your baseline conversion rate to work out how many visitors each variant needs. Run the test until it reaches that sample, and for full weekly cycles so different days are represented. Small expected effects on low-traffic pages may need more traffic than you have; in that case, test bigger changes or improve the page without a formal test.
Statistical Interpretation
Statistical significance tells you how surprising the result would be if there were no real difference; it doesn't measure how big or valuable the effect is. Look at the estimated effect and its confidence interval, not just a “winner” label, and consider whether the effect is large enough to matter commercially.
False Positives and Peeking
Checking results repeatedly and stopping as soon as they look significant greatly increases the chance of declaring a winner that isn't real. Evan Miller's classic explanation, How Not to Run an A/B Test, shows how peeking distorts significance. Fix the sample size in advance, or use a testing method designed for continuous monitoring.
Multiple Testing
Testing many variants, many metrics or many segments at once raises the chance that something looks significant by luck. Limit variants, name one primary metric in advance, and treat segment results as ideas for future tests rather than conclusions.
Check the Split
If the traffic split you configured (say 50/50) differs noticeably from what you observe, something may be wrong with assignment or tracking, a problem known as sample ratio mismatch. Investigate before trusting the result.
Post-Test Analysis
- Confirm the test ran to its planned sample and duration
- Check the traffic split matches the configuration
- Review the primary metric's effect size and uncertainty
- Check guardrail metrics for harm
- Look at major segments cautiously, as hypotheses for next time
- Document the hypothesis, result and what you learned
- Implement winners properly, not as permanent test code
Tools and Platforms
Most ecommerce platforms work with third-party testing tools. On Shopify, the changelog describes Rollouts for testing theme and checkout configurations; see Shopify A/B testing for platform detail and Shopify CRO testing ideas for a longer idea list. Client-side testing tools can add script weight and flicker, so check their performance impact.
Common A/B Testing Mistakes
- Testing ideas with no supporting evidence
- Stopping early when results look good
- Running tests without enough traffic to detect realistic effects
- Too many variants or metrics at once
- Declaring winners on segments after the fact
- Ignoring revenue and margin
- Not documenting results
Ready to test what matters?
Talk to ZSpace about CRO and experimentation, plus design and development for test variants.
Conclusion
Test first where your evidence points and traffic allows: product pages, cart and key landing pages. Write a hypothesis, plan the sample size and metrics, run for full cycles, don't peek, analyze honestly and document everything. The goal is reliable learning, not a string of lucky winners. For ideas tied to evidence, see 25 ecommerce A/B testing ideas; to run testing as a program, see the experimentation framework.
For related guides, see A/B testing framework, hypothesis testing and experiment prioritization.
Common questions
Showing two versions of a page or element to randomly split groups of visitors and comparing a metric such as add-to-cart rate or revenue per visitor, to learn which version performs better.