Experiments (A/B tests)

An experiment is an A/B test: visitors are split between two or more variants of a page, and each variant is measured against a goal. The Experiments screen appears once the site has put a session in one with lynq.experiment. It answers two questions: which variant converts better, and whether that is real yet.

Running, and concluded

Running lists the tests still going, with their variants, how long each has run and what it says so far against the KPI goal. Concluded lists the ones you have finished, with what shipped and why.

A running test says one of four things:

• Winner: one variant has separated from control and will stay separated.
• Still running: nothing separates them yet.
• Not enough data: a variant is under 100 sessions or 5 conversions.
• Split is off: the traffic is not being divided the way it was asked for, so nothing can be read yet.

An experiment

Select a row and three checks appear before any result, because a result computed on a broken test is worse than no result:

• Split: the share each variant got against the share it was meant to get. When they disagree, something is deciding who sees what before Lynq does, usually a cache, a redirect, or a variant set on the server for some visitors only.
• Time running: a test should cover two full weeks, so both weekends are inside it. A Tuesday audience is not a Sunday audience.
• Traffic: enough sessions and conversions in every variant to say anything at all.

Under them, the verdict in a sentence, then the variants: sessions, conversions, the conversion rate, the lift with the range it sits in, revenue per session, and two guardrails, bounce and engaged time, so a variant that wins by pushing people out is not mistaken for a win. Under the table, conversion by day per variant, so a lift that is only a first-week novelty shows as one.

Filter on a variant narrows every screen to its sessions, so the Pages, Sources and Goals screens all read for that variant alone.

How a result is judged

You can look as often as you like. That sounds obvious and it is not: the usual textbook test is only valid if you look once, at a number of visitors fixed before the test started, and a test watched daily with that arithmetic calls a winner far more often than it should.

Lynq uses a test built for being watched. The range shown beside a lift covers the true lift whatever day you read it, and a variant is only called a winner when that whole range sits on one side of zero, which means the result would still hold if the test kept running. The range is wider than the textbook one; that width is the price of being allowed to look.

Two more rules follow from it. Comparing three variants against control is three chances to be wrong, so each is held to a stricter standard than one would be. And when nothing has separated, the sentence says how much longer it would take at the current rate, so you know whether to wait or to call it.

Concluding

Conclude records what shipped and one line saying why. The test moves to Concluded and stops being read as running. Remove the lynq.experiment call from your code when you ship the winner: Lynq records the decision, it does not stop the assignment.