Choosing an A/B testing tool: 10 questions before you decide
Feature lists all look alike. These ten questions separate tools you can work with from tools you replace later.
What this article covers
Every testing tool can swap a headline. The difference only shows in month three: with revenue measurement, with several clients, when explaining a result to a room that is not convinced.
Measuring
- Can the tool compare revenue per visitor, or only conversion rates?
- Are there guardrails, that is metrics which must not get worse?
- Does it report a skewed traffic split by itself, or do I have to notice it?
Deciding
- Does it calculate the sample size before launch, or only when I ask?
- Does it show intervals, or just a percentage with a green tick?
- Does it correct when several variations are compared at once?
Operating
- Where is the data, and can the script be delivered through our own domain?
- Does the tool stay quiet until consent is given?
- What happens to experiments and results if we cancel?
- Does the pricing model fit our growth, or does it punish success?
How to check the answers
Not through the feature list, but through a trial run: connect a real page, set up a revenue goal, let a small test run. After two weeks you know more than after ten demos.
Pay particular attention to how the tool presents an inconclusive result. Tools that always find a winner do not solve a problem, they postpone it.