Experiments (A/B tests)

An experiment is an A/B test: visitors are split between two or more variants of a page, and each variant is measured against a goal. The Experiments screen appears once the site has put a session in one with lynq.experiment. It answers two questions: which variant converts better, and whether that is real yet.

The list

Every experiment in the range, with its variants, when it started and how many sessions it holds. The Result column reads each one against the KPI goal:

• Confident: the best variant is at least 95% likely to beat control, with how much better it converts.
• Likely: 90% or better.
• No clear difference: the arms are within noise of each other.
• Not enough data: a variant is under 100 sessions or 5 conversions.

An experiment

Select a row and the variants appear against a goal, the KPI goal first and any other goal a click away. One sentence gives the verdict: which variant converts how much better than control, the two rates, and how likely the difference is to be real. Under it, the table: sessions, conversions, the conversion rate, revenue per session and the lift against control for each variant.

When there is no clear difference, the sentence says how many more sessions per variant a 10% lift would need to show at the current rate, so you know whether to wait or call it.

Filter on a variant narrows every screen to its sessions, so the Pages, Sources and Goals screens all read for that variant alone.

How it is judged

Each variant’s conversion rate is compared with control’s with a two-proportion test; the probability shown is the chance that the variant’s true rate is above control’s. The interval under the sentence is the 95% range of the difference in points. The range and the filters above apply, so a test can be read for mobile alone, or for last week.