Member-only story
How to Early Peek👀 at Your AB Test?
Previously we’ve covered AB testing on multiple metrics and Holm’s method, today let’s continue with the AB testing series.
Can you peek early?
The short answer is yes.
In general, the advice is to let your experiment run its course and only analyze results when your pre-determined sample size is achieved.
But there are all the good reasons that you or your stakeholders might want to peek early. The main one is efficiency gain.
Repeatedly analyzing incoming data while data collection is in progress has many advantages.
You can stop collecting data at an interim analysis when you can reject the null hypothesis at the smallest effect size of interest, or if something goes wrong with the study. For instance, if you discover a new drug being effective at an interim analysis, you may very well want to end the trial early and provide the drug to the control group which may improve their health condition, or even save their lives.
Another example is that in the experiment design of the Pfizer–BioNTech COVID-19 vaccine, the researchers planned five interim analyses and controlled the overall Type 1 error rate by lowering the alpha level for each interim analysis.
