AB Test Design with Outliers 🤯 — Solutions Part I
Two weeks ago, we discussed the issue of outliers in the context of designing your A/B test. If you need a quick reminder, you can find the relevant post here:
AB Test Design with Outliers 🤯
So what are your options?
There are several options you could choose when dealing with outliers.
The simplest solution is to use the trimmed-mean t-test.
Essentially this is a t-test using the trimmed means of each sample as the means and the Winsorized (trimmed) variances as the variances.
The idea behind this approach is straightforward: if you have data points that are either too big or too small, you throw them away.
Of course, if you have a strong statistics background, you might question the validity of this approach…