From A/B to RL (1/3): Bayesian A/B Testing

Before policies learn, experiments assign.

This series connects A/B testing (randomized comparisons between variants A and B) with reinforcement learning (RL) through a Bayesian view of decision-making. We start with a fixed experiment, where options are assigned in advance. Then we move to situations where each observed result helps determine what to try next.

The aim is not to give a complete introduction to RL. Instead, we use a sequence of small examples to get familiar with key RL concepts: action, reward, policy, state, episode, and delayed feedback.

This is the first post in a 3-part series:

Bayesian A/B Testing

We start the series with a fixed randomized experiment: two variants (A and B), an immediate binary outcome (success or failure), and one final decision about which variant to keep.

An A/B test compares two variants of a system or intervention, such as two webpages, two button labels, two recommendation policies, or two treatments in a randomized controlled trial.

This post uses the simplest version of that setup:

  • two fixed variants, A and B,
  • random assignment to either A or B,
  • one binary outcome per assignment,
  • a fixed batch of data,
  • one final decision.

This small setup keeps two questions separate:

  1. how to model uncertainty about each variant's performance,
  2. how to use that uncertainty to make the final decision at the end of the experiment.

From this point on, variant and action mean the same thing. I'll use variant when discussing the A/B test and action when making the RL connection.

In [1]:

The Toy Problem

To make the setup concrete, let's start with a simple A/B-testing example. Imagine we want to test a change to a button on a website, such as a Buy or Subscribe button. We want to compare the current button (variant A) with a new design (variant B).

Our goal is to determine whether variant B produces more clicks than variant A. In this toy example, a click is the immediate reward: $1$ for a click and $0$ otherwise. We measure each variant's performance with its click-through rate (CTR), the fraction of users who click after being shown the button.

Each exposure is a single interaction: one user sees either button A or B, and we observe whether they click it. In RL terms, the choice is the action, and in this example the click outcome is the immediate reward. Every exposure starts from the same situation, so there is no changing state to track.

The causal question is whether showing button B causes more clicks than showing button A. We use the same random-assignment rule throughout the experiment. At the end, we make one final decision about which variant to ship.

This leaves one question:

Which variant should we ship?

We will simulate one randomized experiment. The true CTRs, or expected rewards, are fixed but hidden from the inference procedure. We keep them only so we can check later whether the posterior behaves sensibly.

In [2]:

The next cell generates one randomized experiment. For simplicity, the simulation generates an equal number of outcomes for each arm; the randomization is represented by the experimental design rather than simulated user by user.

In [3]:
# Simulate one fixed randomized A/B test.
TRUE_RATE_A = 0.12  # Hidden from the learner
TRUE_RATE_B = 0.15  # Hidden from the learner
NB_TRIALS = 200
experiment_rng = np.random.default_rng(20260425)

# Simulate user outcomes for each variant. Each user either clicks (1) or doesn't click (0).
observations_a = experiment_rng.binomial(n=1, p=TRUE_RATE_A, size=NB_TRIALS)
observations_b = experiment_rng.binomial(n=1, p=TRUE_RATE_B, size=NB_TRIALS)

experiment = ABTestData(
    observations_a=observations_a,
    observations_b=observations_b,
    true_rate_a=TRUE_RATE_A,
    true_rate_b=TRUE_RATE_B,
)

clicks_a, clicks_b = experiment.clicks
no_clicks_a, no_clicks_b = experiment.no_clicks

print(f"Variant A: {clicks_a} clicks, {no_clicks_a} non-clicks, observed CTR = {clicks_a / NB_TRIALS:.3f}")
print(f"Variant B: {clicks_b} clicks, {no_clicks_b} non-clicks, observed CTR = {clicks_b / NB_TRIALS:.3f}")

Variant A: 23 clicks, 177 non-clicks, observed CTR = 0.115
Variant B: 32 clicks, 168 non-clicks, observed CTR = 0.160

This chart shows the noisy click counts observed in the experiment. The Bayesian model uses these counts as evidence about the two unknown CTRs.

In [4]:
Observed click and no-click counts for variants A and B in the simulated A/B experiment.

What Is Actually Observed?

The important distinction is that we do not observe the true click-through rate (CTR) directly. Under randomized assignment, we observe only finite, noisy counts:

  • how many users saw A and how many saw B,
  • how many clicks each variant produced.

The observed CTR is the number of clicks divided by the number of users who were shown that variant. It estimates the variant's expected immediate reward. Since each exposure is a one-step interaction, that expected reward is also the action value. We use $\theta$ to represent the unknown long-run click probability.

That leaves us with the real question: how should we reason about the unknown click probability $\theta$ from limited observations?

A Bayesian view helps because it turns observed click counts into a posterior distribution over $\theta$ for each variant. We can then ask decision questions directly, such as how likely it is that B has the higher expected reward.

To answer those questions, we begin with a simple probabilistic model that connects each click outcome to an unknown click-through rate. This is not the only way to analyze an A/B test, but it gives us a useful starting point.

A Minimal Bayesian Model

Let's express this setup in probabilistic terms.

For one user, the outcome $x$ is binary, so we model it with a Bernoulli distribution:

$$x \sim \mathrm{Bernoulli}(\theta)$$

Here $x = 1$ means a click and $x = 0$ means no click. We use $\theta$ for the unknown click-through rate of one variant. In decision-making terms, $\theta$ is the expected immediate reward from showing that variant. In this one-step setting, it is also the action value.

If we group $n_{\mathrm{users}}$ users together and count their clicks, the total $n_{\mathrm{clicks}}$ follows a Binomial distribution:

$$n_{\mathrm{clicks}} \sim \mathrm{Binomial}(n_{\mathrm{users}}, \theta)$$

This toy model assumes that each variant has a stable click probability and that user outcomes are conditionally independent given that probability.

This describes how clicks are generated if we know $\theta$. In the A/B test, we face the inverse problem: we observe clicks and want to infer the unknown rate.

Bayesian inference uses Bayes' theorem to move from the observed clicks back to the unknown rate.

For a single observed outcome $x$, that relationship can be written as:

$$p(\theta \mid x) = \frac{p(x \mid \theta) p(\theta)}{p(x)}$$

The posterior is proportional to the likelihood times the prior. In practice, we combine the likelihood of the observed clicks with a prior over $\theta$. The result is a full posterior distribution over the unknown click-through rate, rather than a single best guess.

For the prior, we use a Beta distribution:

$$\theta \sim \mathrm{Beta}(\alpha, \beta)$$

The Beta distribution describes plausible click-through rates between 0 and 1. For a more in-depth introduction to the beta distribution, see Beta distribution: Probabilities and binary data. Here we use the uniform prior $\mathrm{Beta}(1, 1)$, which represents the assumption that, before seeing any data, we do not favor any particular rate.

This prior is convenient because the Beta distribution is a conjugate prior for the Bernoulli/Binomial likelihood. After observing clicks and non-clicks, the posterior is also a Beta distribution, with updated parameters.

For one observation, the update is simple:

  • after a click, $\mathrm{Beta}(\alpha, \beta) \to \mathrm{Beta}(\alpha + 1, \beta)$,
  • after a non-click, $\mathrm{Beta}(\alpha, \beta) \to \mathrm{Beta}(\alpha, \beta + 1)$.

If we summarize the observations by their total number of clicks, the same update becomes:

$$\theta \mid n_{\mathrm{clicks}}, n_{\mathrm{users}} \sim \mathrm{Beta}(\alpha + n_{\mathrm{clicks}}, \beta + n_{\mathrm{users}} - n_{\mathrm{clicks}})$$

The update has a simple interpretation: a click adds 1 to the first shape parameter, and a non-click adds 1 to the second, updating the posterior distribution in either case. Starting from $\mathrm{Beta}(1, 1)$ and observing $n_{\mathrm{clicks}}$ clicks out of $n_{\mathrm{users}}$ users gives $\mathrm{Beta}(1 + n_{\mathrm{clicks}}, 1 + n_{\mathrm{users}} - n_{\mathrm{clicks}})$. Here, however, we update once using the full experiment.

For the A/B test, we apply this model separately to variants A and B, whose unknown rates $\theta_A$ and $\theta_B$ are the expected immediate rewards of the two actions. We assume independent priors for A and B and independent outcomes within each arm, so the two posterior distributions are independent. We then compare the two posterior distributions to decide which variant to ship.

From Observed Counts to Posterior Distributions

Let's apply the model to the toy experiment. The model uses the observed click and no-click counts from variants A and B. The dashed lines mark the hidden true CTRs, or expected rewards, used to generate the simulation; they are not available to the posterior calculation, which combines the observed counts with the fixed Beta prior.

In [5]:
# Compute Beta posterior parameters from observed click counts.
def beta_posterior_parameters(
    successes: int,
    trials: int,
    alpha_prior: float = 1.0,
    beta_prior: float = 1.0,
) -> tuple[float, float]:
    """Return the Beta posterior shape parameters after Bernoulli or Binomial observations."""
    return alpha_prior + successes, beta_prior + trials - successes

alpha_a, beta_a = beta_posterior_parameters(successes=clicks_a, trials=nb_trials)
alpha_b, beta_b = beta_posterior_parameters(successes=clicks_b, trials=nb_trials)
In [6]:
Beta posterior densities for variants A and B; dashed lines show the true simulated click-through rates hidden from the learner.

Making the Decision

With posterior distributions for variants A and B, we can ask how likely it is that B's true CTR, or expected immediate reward, exceeds A's. In terms of the unknown CTRs:

$$ P(B > A \mid \mathrm{data}) = P(\theta_B > \theta_A \mid \mathrm{data}). $$

Here $\theta_A$ and $\theta_B$ are the unknown true CTRs of variants A and B. In the one-step reward interpretation, they are also the unknown one-step action values.

For this post, $P(B > A \mid \mathrm{data})$ is a decision quantity: it helps us decide which variant to ship after the experiment. It is not an online rule for choosing variants: assignments were fixed in advance. In the next post, a similar posterior probability will become an online policy distribution.

There are two useful ways to compute this probability:

  1. approximate it with Monte Carlo sampling: draw posterior samples from A and B and count how often $\theta_B > \theta_A$,
  2. compute it exactly for Beta posteriors using the finite-sum formula for comparing two Beta distributions.

Monte Carlo is more general because it works whenever we can sample from the posterior. In the Beta-Bernoulli case, we can also compute the same probability directly. The exact calculation integrates over the two independent Beta posteriors rather than relying on Monte Carlo samples.

Estimating $P(B > A \mid \mathrm{data})$

We can estimate $P(B > A \mid \mathrm{data})$ by Monte Carlo: repeatedly draw one plausible CTR from each posterior and, for each pair of draws, record whether B's draw is larger. If B wins 90% of the paired draws, our estimate of $P(B > A \mid \mathrm{data})$ is about 0.90.

Each comparison pairs one draw of $\theta_A$ with one draw of $\theta_B$ and records whether $\theta_B > \theta_A$.

$$ P(B > A \mid \mathrm{data}) = P(\theta_B > \theta_A \mid \mathrm{data}). $$

For each pair, we can also compute the posterior difference $\Delta$: B's sampled CTR minus A's sampled CTR. In this one-step setting, it is also the sampled difference in expected reward:

$$ \Delta = \theta_B - \theta_A. $$

A positive $\Delta$ means that B is higher in that paired draw. When A is a baseline or control, this difference is often called the absolute lift of B over A.

The figure shows three views of the same paired draws. The scatter plot on the left shows a readable subset. The histogram in the center uses all paired draws to show the posterior differences $\Delta$. The bar chart on the right summarizes those comparisons as the posterior probability that B beats A. In all three panels, green indicates $\theta_B > \theta_A$.

In [7]:
Observed A: 23/200 = 0.115
Observed B: 32/200 = 0.160
P(B > A | data), posterior sampling: 0.902
P(B > A | data), exact Beta formula: 0.903
Three views of paired posterior samples for variants A and B: their joint samples, the distribution of their difference, and the probability that B exceeds A.

How the Posterior Tightens with More Data

In this post, the A/B test is a choose-once, fixed-policy problem, so we care most about the final posterior. It is still useful to watch the posterior change as data accumulates: the distributions narrow, uncertainty shrinks, and the overlap between A and B becomes easier to interpret.

The next cell runs a fresh, longer simulated experiment and looks at prefixes after 20, 100, 500, and 2,000 observations per variant. This illustrates how the posterior evolves in a new run; it is not a continuation of the toy experiment above.

In [8]:
Beta posterior densities for variants A and B after increasing numbers of observations, showing uncertainty narrowing over time.

How Much Evidence Do We Need?

In this post, we collect the data first and make one decision at the end, so the amount of data matters a lot. The previous section showed how the posterior distributions over $\theta_A$ and $\theta_B$ tighten as observations accumulate within one experiment, making the decision quantity $P(B > A \mid \mathrm{data})$ more decisive. Now we ask what happens across many repetitions of the same experiment.

If we repeated the same randomized A/B test many times, how often would our decision rule produce strong evidence in favor of variant B?

Under the assumed model, the heatmap below gives the exact answer to that question. We fix variant A's true CTR at 0.10 and set variant B's true CTR to 0.10 plus the absolute lift shown on the y-axis. For each combination of observations per variant (x-axis) and absolute lift (y-axis), we enumerate all possible observed click counts for variants A and B. For each possible outcome, we compute $P(B > A \mid \mathrm{data})$ and sum the probabilities of the outcomes for which this posterior probability exceeds 0.95.

Each cell gives the exact probability, under the assumed Binomial model, that an experiment with that sample size and true lift would produce strong evidence in favor of variant B, defined here as $P(B > A \mid \mathrm{data}) > 0.95$.

In [9]:
Heatmap of the fraction of simulated experiments where the posterior probability that B beats A exceeds 0.95, by sample size and true lift.

Why Random Assignment Matters

The calculations above rely on random assignment. Here is why that matters.

Random assignment gives the comparison a causal interpretation. In causal inference, it lets us treat showing A or B as an intervention rather than as a characteristic of the user. It also helps protect against confounding: differences in clicks are less likely to reflect pre-existing differences between the groups.

Short Note on Real-World Use

Real A/B tests are messier than this toy example. In practice, we need to consider data quality, sample-ratio mismatches, multiple metrics, stopping rules, business constraints, and how the change affects the people who experience it. We also need to ask whether the measured outcome is a good reward signal for the value we care about.

Summary

The toy A/B test maps to RL as follows:

  • action: show either variant A or variant B,
  • policy: a fixed randomized assignment rule, written as $\pi(a \mid s_0)$ in the one-state notation; randomization gives the comparison its causal interpretation,
  • feedback and reward: an immediate click or no-click outcome; its expected value is the variant's CTR,
  • state: the same simple state $s_0$ for every exposure; outcomes are modeled as independent given each variant's CTR,
  • what we learn: posterior uncertainty about each variant's CTR,
  • decision quantity: the posterior probability that variant B has the higher expected immediate reward, $P(\theta_B > \theta_A \mid \mathrm{data})$.

This post considers a fixed experiment: feedback is immediate, we collect the data first, and make one final ship decision without changing traffic allocation while the experiment runs. In the next post, the same posterior-probability idea becomes an adaptive action distribution as evidence arrives.

References

Further Reading

In [10]:
Python: 3.13.12
numpy: 2.4.6
scipy: 1.17.1
matplotlib: 3.11.0
seaborn: 0.13.2

This post is generated from an IPython notebook file. Link to the full IPython notebook file