How the router learns

Thompson sampling over Beta posteriors: traffic moves toward the variant that converts, while the others keep enough to be measured.

variant	visitors	conv	p(best)
a (control)	1200	38	1.8%
b	2900	131	90.1%
c	380	11	8.0%

Computed on this page by the same code the API runs. Narrow curves are confident. The router sends most new visitors to b; c, still uncertain, keeps a share until its curve narrows.

Posteriors

For each variant the API keeps two counts: visitors exposed (n) and visitors who converted (c, at most once each). The belief about its true conversion rate is Beta(1 + c, 1 + n - c). More visitors make the curve narrower.

From posteriors to weights

Every 30 seconds the API draws 4,000 samples from each posterior and counts how often each variant has the highest sample. That share is its weight: the probability that it is the best variant. A floor of 2% per variant keeps every arm measured while the test runs. The proxy then hashes each new visitor into those weights.

Compared with a fixed split, fewer visitors see the losing variants during the test, so the test costs fewer conversions.

Segments

With --segment source,device, the API also keeps counts per segment value (source:x, device:mobile, country:fr, visitor:returning). A segment gets its own weights once it has 30 visitors; until then it uses the global weights. A variant can win on mobile and lose on desktop; the router serves each the one that works.

When to stop

Promote when one variant's p_beat_control reaches 0.95 and the lower end of its uplift interval is above zero, with at least 300 visitors per variant. If all intervals still straddle zero after 2,000 visitors per variant, the change did not matter: stop and test a bolder idea.