AB Test Design with Outliers — What is CUPED?

A few weeks ago, we explored the issue of outliers in the context of designing your A/B test and provided potential solutions, such as trimming outliers. Let’s recap:

Why does this matter?

• The stakes of your experiment are high. For instance,

A mere 0.1% increase to revenue at Facebook is worth over $100 million per year!

• The time to action based on your experiment outcome is mission-critical for your business.

Businesses don’t want their experiments to run for a year just to gather enough sample size. Moreover, features that have a negative impact should be stopped as soon as possible. Therefore, we aim to find ways to decrease the standard error, reduce the sample size, and consequently reduce the learning time from the experiments.

If you need a quick reminder, you can refer to the relevant post here:

AB Test Design with Outliers 🤯

Two weeks ago, we discussed the issue of outliers in the context of designing your A/B test. If you need a quick reminder, you can find the relevant post here: So what are your options? There are several options you could choose when dealing with outliers.