AgentBIAgent with Business Intelligence for DTC sites

Back to the knowledge base

BIZ-4

Set sample size and read-out rules before conversion experiments

Published:2026-11-21Views:0

Conversion experiments most often fail by concluding after a few days: before the sample is large enough and the variance has settled, a single day's lift may just be noise. Write the sample size and read-out rules before the experiment starts — which metric, how many samples before reading, how large a difference counts — so results are not explained by cherry-picking afterwards.

Fix the primary metric and denominator up front, estimate a sample size for the difference you want to detect and log it; do not change the definition mid-run or declare success because an interim looks good. Anything touching payment or checkout is verified in sandbox only, never A/B tested on real orders; archive the sample size, start/end time and metric version together when reading out.

This article covers sample size and read-out discipline only; it issues no compliance verdict on any result and decides no launch for anyone. Write conclusions as recorded — an unclear difference stays unclear, flat numbers stay flat — and do not use them to push more experiments or a change of direction.

2026-11-22