At a glance
A/B testing means pitting two versions of a page against each other and letting the data pick the winner. Long missing from Shopify, it is now available natively: the Rollouts tool, available since June 2026, lets you split traffic between two theme versions starting on the Grow plan. Its limits are real (no price testing, no statistical significance), and apps such as Intelligems or Shoplift take over for advanced use cases. But the real question lies elsewhere: do you have enough traffic for a test to be conclusive? At Stellar, we keep most of our A/B testing for email marketing, where the volume exists, and in CRO we favor a qualitative approach built on proven best practices. This page gives you the tools, the method and, above all, the cases where you are better off not testing.
Contents
Test two variants and let the numbers decide: on paper, A/B testing is the ultimate CRO tool. In practice, it is one of the most misused topics in e-commerce, because a poorly calibrated test produces wrong conclusions with great confidence. Here is how to test properly on Shopify, and how to know whether you should be testing at all.
Our practice, in full transparency: we do a lot of A/B testing on email marketing, where the volumes make tests genuinely conclusive. For store CRO, we work mostly qualitatively: we keep track of best practices with our partners, media agencies in particular, and we draw on this proven knowledge base to implement it for our clients. In practice, this beats A/B tests that are often inconclusive for lack of sufficient traffic.
Rollouts: Shopify's native A/B testing
This is the big change of 2026: Shopify finally offers native A/B testing. The tool is called Rollouts, it has been available directly in the admin since June 2026, and it splits traffic between two versions of a theme through Shopify's servers, with no app to install. Traffic splitting requires the Grow plan or higher.
What Rollouts tests well: theme variants, meaning the layout of a product page, a homepage banner, navigation, collection grids. What it does not do yet:
- Test prices, discounts or offer logic.
- Calculate statistical significance or declare a winner automatically.
- Segment the audience beyond the market, or test an isolated section without duplicating the theme.
In other words, Rollouts is perfect for comparing two design choices in real conditions, provided you interpret the results yourself with rigor.
Specialized apps: going further
As soon as your needs go beyond theme testing, specialized apps remain the benchmark. Here are the main ones in 2026 and what they are good at.
| Tool | Strength | Best for |
|---|---|---|
| Rollouts (native) | Theme version testing, no app, from Grow | Comparing two designs in real conditions |
| Intelligems | Price, shipping and offer tests | Stores that want to test their prices |
| Shoplift | Section-level tests, significance analysis | Targeted tests with guided statistical reading |
| ABConvert | Price and shipping threshold tests | Fine-tuning average order value |
The choice depends on what you are testing: design (Rollouts is enough), price (Intelligems or ABConvert), or targeted hypotheses with a real statistical engine (Shoplift). In every case, the tool does not replace the method.
The method: sample, duration, mistakes
An A/B test is only worth something if its framework is solid. Three rules make the difference between an informed decision and a statistical illusion.
Sample size first. To detect a realistic effect (a few points of conversion), you generally need several thousand visitors per variant. Below that threshold, the result is noise: a test on 500 visitors that shows a winner at +20% proves nothing.
Duration next. Two weeks is a good minimum ballpark: covering at least two full weekly cycles smooths out day-of-week, payday and campaign effects. Stopping a test as soon as it looks like a winner is the most common mistake.
Pitfalls last. The most frequent ones: testing several elements at once (impossible to attribute the effect), segmenting after the fact until you find a winning subgroup, ignoring atypical periods (sales, holidays), and generalizing a mobile result to all traffic. To measure properly, rely on a solid analytics setup, as we explain in our guide to GA4 on Shopify.
Prioritizing your tests with a simple framework
Not all test ideas are equal. A prioritization framework like ICE scores each hypothesis on three criteria: expected impact, confidence in the hypothesis, and ease of implementation. You test the highest scores first.
In practice, a hypothesis born from an observed friction point (a blocker spotted in heatmaps or in the funnel analysis) will always score better than an aesthetic whim. The A/B test validates a documented intuition, it does not replace the diagnosis.
When not to test: our approach
The question almost nobody asks: does your store have the traffic to test? Below a few thousand weekly sessions, most tests will remain inconclusive, and you will waste weeks staring at curves that mean nothing.
That is why we own our practice: we do intensive A/B testing on email marketing, where send volumes allow fast and reliable conclusions. For store CRO, we prefer qualitative work: active monitoring of best practices with our partners, then direct implementation of what has proven itself elsewhere. This approach ties in with our core conviction about the customer journey: simplicity and common sense deliver most of the gains, testing comes after.
If you are torn between launching tests and going back to the fundamentals, that is exactly the kind of trade-off covered by our e-commerce support, as part of our guide to CRO on Shopify.