AB Test Design with Outliers 🤯 — Solutions Part I

Two weeks ago, we discussed the issue of outliers in the context of designing your A/B test. If you need a quick reminder, you can find the relevant post here:

AB Test Design with Outliers 🤯

Today we’ll discuss a common problem with regards to A/B testing: What to expect when there’re outliers in your population? Two-sample t-tests are widely used for A/B testing when the primary objective is to compare the means of two groups (Group A and Group B) to determine if there is a statistically significant difference between them. It is also a special case of ANOVA (Analysis of Variance).

Read full story

So what are your options?

There are several options you could choose when dealing with outliers.

The simplest solution is to use the trimmed-mean t-test.

Essentially this is a t-test using the trimmed means of each sample as the means and the Winsorized (trimmed) variances as the variances.

The idea behind this approach is straightforward: if you have data points that are either too big or too small, you throw them away.

Of course, if you have a strong statistics background, you might question the validity of this approach…