Did the Ads Actually Work?

// the question

Every campaign report counts conversions among people who saw ads. That number is nearly meaningless: heavy platform users see more ads and buy more anyway. The honest question is counterfactual — how many conversions would have happened without the ads? This experiment (Kaggle, Marketing A/B Testing) answers it by design: users were randomized to ads or to public-service announcements in the same slots. The control's conversion rate is the counterfactual.

// is it real · exact inference first

Ad arm: 2.55% conversion. Control: 1.79%. Lift: +0.77pp. Before any asymptotics, the sharpest test available uses only the randomization itself: if ads did nothing, the arm labels are arbitrary, so shuffle them. Because a full relabeling of 588,101 users only changes how many of the 14,843 total conversions land in the control arm, each shuffle is one draw from a hypergeometric — which makes one million exact permutations instant:

# one label-shuffle == one hypergeometric draw of control-arm conversions
perm_x0   = rng.hypergeometric(K, N-K, n0, size=1_000_000)   # K = all conversions
perm_diff = (K-perm_x0)/n1 - perm_x0/n0                      # lift under H0
p_exact   = (perm_diff >= observed_diff).mean()              # -> 0 of 1,000,000

The permutation null. The observed lift sits far beyond every one of a million no-effect worlds; the classical z-test (z = 7.4, p = 1.7e-13) agrees.

// the bayesian read · from “significant” to decisions

The p-value answers “how surprising is this data if ads did nothing?” A client asks a different question: “given the data, what is the lift — and how sure are we?” That is a posterior. Each arm's rate gets a flat Beta(1,1) prior; because the Beta is conjugate to the Binomial, observing x conversions in n users updates it by addition, no fitting required:

# conjugacy: prior Beta(1,1) + data (x of n) -> posterior Beta(1+x, 1+n-x)
post_ad  = rng.beta(1 + 14_423, 1 + 550_154, size=4_000)   # ad arm draws
post_psa = rng.beta(1 +    420, 1 +  23_104, size=4_000)   # control draws
rel_lift = post_ad / post_psa - 1                          # posterior of the lift
P_gt_25  = (rel_lift > 0.25).mean()                        # -> 0.997

The control's posterior is visibly wider — 24× less data — and the machinery prices that honestly. Any decision quantity is then a matter of counting draws: P(lift > 0) ≈ 1.000, P(lift > +10%) ≈ 1.000, P(lift > +25%) = 99.7%, with a 95% credible interval of +30% to +57% — an interval that means what everyone always wanted intervals to mean: a 95% probability the true lift is inside.

Left: posterior conversion rates by arm. Right: the posterior of relative lift; essentially no mass below +25%.

// the trap · dose is not response

Within the ad arm, conversion rises from 0.24% (1–4 impressions) to 16.7% (81+). The naive read — “frequency works, buy more” — is confounded: impressions go to whoever shows up most, and heavy users convert more regardless. The control breaks the tie, since its heavy users saw heavy PSAs. Ad-vs-control within activity strata isolates what the ads add:

Both arms climb with activity (log scale) — that slope is user behavior, not advertising. The causal part is the gap, significant only at 41+ impressions.

Where the campaign’s 4,343 incremental conversions actually come from.

// uplift · from “does it work” to “on whom”

The average effect hides its own budget plan. A T-learner fits two logistic regressions — my usual IRLS, coded from the likelihood up — one on the treated arm, one on the control, each predicting conversion from user features (log exposure, weekend, daypart). The difference of the two fitted probabilities is a per-user predicted uplift τ̂(x): how much the ad changes this user's chance of converting.

b_treated = logit_irls(X[train & ad ], y[train & ad ])   # P(convert | ad, x)
b_control = logit_irls(X[train & psa], y[train & psa])   # P(convert | no ad, x)
tau_hat   = sigmoid(X_test @ b_treated) - sigmoid(X_test @ b_control)

Ranking a 294,050-user holdout by τ̂ and sweeping down the list traces the Qini curve — incremental conversions captured vs. share of audience targeted, against the random-targeting diagonal:

The Qini curve on the holdout. Steep early: the model finds the persuadables first.

And the model keeps its promises out of sample — predicted uplift by decile tracks the realized ad-minus-control gap, including correctly identifying deciles the ads barely move:

Calibration by holdout decile: the top decile was promised ~4.9pp and delivered ~6.7pp.

// interactive · posterior economics

1,500 posterior draws per arm are embedded in this page. Set a CPM and a conversion value: every readout below is recomputed across all draws — medians with 95% credible intervals, and the probability the campaign pays for itself.

// interactive · who to target

The Qini curve, priced. Choose what share of the audience to target (ranked by predicted uplift) — the widget applies the holdout curve and the actual impression volumes of the targeted users. High-uplift users are heavy-exposure users, so cost falls slower than reach: the model tells you that too, honestly.

// what this demonstrates

Three complementary ways to read one experiment — exact (assumption-free), classical (fast), Bayesian (decision-ready) — plus the modeling step agencies actually buy: heterogeneous treatment effects with an honest holdout, a Qini curve that turns causal evidence into a media plan, calibration to prove the model isn't flattering itself, and the endogeneity trap (dose ≠ response) called out rather than monetized. The recommendation prices itself: target the top decile by predicted uplift, keep ~86% of the effect, and buy half the impressions — lifting iROAS from 2.35× to 4.4× at an $8 CPM.