Lesson objectives
- Design a valid A/B test
- Avoid common statistical mistakes
Before you start
- Write the hypothesis and primary metric in advance.
- Estimate the sample size needed with a calculator, based on your baseline rate and the smallest change worth detecting.
- Run for full weeks to smooth out day-of-week effects.
- Test one meaningful change - or a clearly different page - not a dozen tiny tweaks at once.
Mistakes that create false winners
- Stopping as soon as a result looks significant ('peeking').
- Running too many variants on too little traffic.
- Ignoring down-funnel quality.
- Declaring small differences on small samples as wins.
Action steps
- Write a test plan: hypothesis, metric, sample size, duration
Checklist
- Sample size estimated
- Duration fixed in advance
- Quality metric tracked