AB Test Design with Outliers — What is CUPED?

Angelina Yang
6 min readOct 16, 2023

A few weeks ago, we explored the issue of outliers in the context of designing your A/B test and provided potential solutions, such as trimming outliers. Let’s recap:

Why does this matter?

  • The stakes of your experiment are high. For instance,

A mere 0.1% increase to revenue at Facebook is worth over $100 million per year!

  • The time to action based on your experiment outcome is mission-critical for your business.

Businesses don’t want their experiments to run for a year just to gather enough sample size. Moreover, features that have a negative impact should be stopped as soon as possible. Therefore, we aim to find ways to decrease the standard error, reduce the sample size, and consequently reduce the learning time from the experiments.

If you need a quick reminder, you can refer to the relevant post here:

AB Test Design with Outliers 🤯

Two weeks ago, we discussed the issue of outliers in the context of designing your A/B test. If you need a quick reminder, you can find the relevant post here: So what are your options? There are several options you could choose when dealing with outliers.

Read full story

CUPED Method

Today we will discuss a different approach called CUPED (Controlled-experiment Using Pre-Experiment Data). Microsoft proposed and named CUPED in 2013, and it has since become widely used in online experimentation.

The guiding principle of CUPED is that not all variance in an experiment are random. In fact, a lot of the differences in user outcomes are based on pre-existing factors that have nothing to do with the experiment.

To demonstrate this concept, consider the simple example as the following:

Suppose we want to test whether people run slower when weights are attached to their bodies (obviously 🤓). We divide the participants into two groups: the test group, which runs with weights, and the control group, which runs without weights. The collected data may appear as follows:

--

--