Criteo Uplift.
A 14 million row randomized ad experiment that clears every statistical bar and still loses money. An X-Learner isolates who the ad actually persuades, a profit-aware bandit turns the loss into a profit, and the model is distilled to a sub-millisecond decision tree for real-time bidding.
+59.4% lift
statistically significant, p < 0.001
-$0.07 / user
$0.10 cost per ad, treat everyone
+$0.08 / user
same data, who to bid on
The lift is real and the loss is real. The X-Learner finds who to actually target.
Rows processed
85/15 randomized treatment split
A/B lift, still a loss
net -$0.07/user at $0.10 cost per ad
Bandit turnaround
profit-aware LinUCB replay, same data
Production latency
~45µs distilled tree vs ~120ms teacher ensemble
Test coverage
11 pytest files: validator, bandit, distillation, evaluator
CI jobs / push
pytest backend + npm build frontend, GitHub Actions
Memory reduction
1706MB to 693MB, float64 to float32 downcast
Deployed surfaces
pipeline, live FastAPI (Render), live dashboard (Vercel)
A significant lift, a losing campaign.
A standard A/B test on this dataset clears every bar an analyst would ask for: a large sample, a low p-value, a positive lift. It also describes a population average, not an individual, so it cannot say who the ad actually persuades, who would have converted anyway, and who it might actively repel.
Multiply the measured lift by real unit economics, $10 per conversion, $0.10 per impression, and the same statistically significant campaign is losing money on every user it reaches. The rest of this project exists to close that gap: from a population-average verdict to a per-user one that can actually be acted on.
0.292% global conversion rate x $10 value
$0.10 per impression, shown to everyone
Breakeven needs an absolute lift of cost over value, 1%. The measured absolute lift is about 0.115 percentage points, roughly 9x too small to justify blanket targeting. The +59.4% relative lift is real and still not the number that decides whether the campaign is a good idea.
14 million rows, no missing values.
The Criteo Uplift Modeling dataset: a real randomized ad-exposure experiment, not a synthetic benchmark.
Rows
0 missing values across 16 columns
Treatment / control
randomized ad exposure experiment
Global conversion rate
the rare-event problem every model fights
Anonymized features
no public semantic mapping, Criteo dataset
One pipeline, a training-free deploy.
8 pipeline stages produce everything the API and dashboard need. The production image cannot retrain or load the teacher model.
$ python pipeline.py
Load
Polars read + float64 to float32, binary flags to int8. 1706MB to 693MB, a 59.4% reduction.
Validate
SRM chi-square (alpha 0.001) and per-feature SMD balance check before any effect is estimated.
Classical A/B
Welch's t-test ATE plus CUPED variance reduction, theta fit on the control arm only.
X-Learner
Propensity, outcome, and effect models: 4 LightGBM boosters producing a per-user CATE.
Evaluate
Decile ranking check, bootstrapped Qini curve, 95% confidence interval on AUUC.
Segment
Surrogate decision tree explains the CATE surface as human-readable targeting rules.
Bandit replay
LinUCB contextual bandit, off-policy replay against the logged randomized experiment.
Distill
Depth-5 decision tree trained on the teacher X-Learner's soft labels, ~2,667x faster.
FastAPI loads only the distilled tree at startup. The training stack (Polars, LightGBM) never ships to production.
Reads a pipeline-time insights.json for analysis pages, calls the live API only for the real-time bid demo.
Validate the experiment before trusting a number.
Two integrity checks run before any effect is estimated, and a variance-reduced estimate that cannot leak the treatment effect into its own adjustment.
Chi-square goodness of fit, alpha = 0.001
p = 0.9989, passA conventional alpha of 0.05 would flag noise constantly at 14M rows. The strict alpha is deliberate.
Standardized Mean Difference, all 12 features
max |SMD| = 0.0488Below the 0.1 threshold on every feature. The randomization held.
CUPED reduces variance by regressing the outcome on pre-experiment covariates. Fitting that regression on the full population would let the treatment effect leak into the adjustment itself, biasing the very estimate it is meant to sharpen.
Theta is estimated on the control arm only, then applied everywhere. The adjustment runs on a 10% sample rather than the full 14M rows: at that scale standard error already scales as one over the square root of n, so the full dataset buys negligible further precision for far more compute.
1# Theta estimated on the control arm only,2# so the treatment effect cannot leak into3# its own variance-reduction adjustment.4control_df = self.df.filter(5 pl.col(treatment_col) == 06)7X_control = control_df.select(covariates)8Y_control = control_df.select(target_col)9 10theta, *_ = np.linalg.lstsq(11 X_control_centered, Y_control, rcond=None12)13 14# Applied to the full population afterward.15y_cuped = Y - (X - X_mean) @ thetaAn X-Learner, not a T-Learner.
4 LightGBM boosters instead of 2, chosen specifically because the control arm is only 15% of a dataset with a 0.29% base conversion rate.
Propensity g(x)
One LightGBM classifier, trained on all users, predicts P(treatment = 1 | x). Used to weight the final estimate.
Outcome models M0, M1
Separate LightGBM classifiers for the control and treatment arms, each predicting conversion probability within its own arm.
Imputed effects D0, D1
D1 = Y1 minus M0(X1), and D0 = M1(X0) minus Y0. Cross-arm counterfactual residuals, computed in plain NumPy.
Effect models tau0, tau1
Two LightGBM regressors, each trained to predict the imputed effect from its own arm's features.
CATE(x) = g(x) · tau0(x) + (1 − g(x)) · tau1(x)
Weighting looks inverted at first glance, tau0 (fit on control-arm residuals) gets weighted by the propensity to be treated. That is the correct formulation from the original X-Learner paper: it down-weights each arm's effect estimate in the region of feature space where that arm has less data support, which is exactly the imbalance problem the X-Learner exists to solve.
Does the model actually rank users?
Two independent checks on the same held-out test set, a decile ranking audit and a bootstrapped Qini curve.
A sqrt scale is used so decile 9 does not flatten every other bar to invisible, the real value is 0.9087% against a population average of 0.1152%. Middle deciles are genuinely not monotonic, decile 0 (lowest predicted uplift) shows more actual lift than deciles 1 through 7, an unsmoothed result from a rare-event target, not a cleaned-up staircase.
The amber curve stays above the dashed random-targeting diagonal across the informative region of the population, and the lower bound of the 95% interval stays above it too. The lift the X-Learner finds is not noise. Max incremental conversions on this axis: 4008.
Two features explain almost everything.
A depth-5 surrogate decision tree turns the X-Learner's predictions into human-readable rules.
f4 and f3 together account for 77% of the surrogate tree's explanatory power. The other 9 features, f0, f1, f5, f6, f7, f11, and the remainder, contribute close to nothing.
Large, consistent differences, not marginal noise: f6 differs by -106%, f9 by +104%. "Persuadable" is a real, feature-distinguishable population.
Four archetypes, one bidding rule.
CATE and baseline conversion probability alone classify every user into a business-meaningful segment.
CATE >= 0.02
The ad drives incremental conversion. The single highest-value leaf in the surrogate tree (f4 > 11.77, f3 <= 0.25, f2 <= 8.32) predicts CATE of 0.08 here.
CATE < 0
The ad actively suppresses conversion. Bidding here is worse than doing nothing, not merely wasteful.
0 <= CATE < 0.02, baseline prob above control CR
Converts regardless of the ad. High baseline probability means the spend buys nothing incremental.
0 <= CATE < 0.02, baseline prob at or below control CR
Low conversion probability with or without treatment. The ad cannot move them.
A profit-aware bandit, replayed on real data.
LinUCB, evaluated by the Replay Method: an event only counts when the bandit's chosen arm matches what the logged randomized experiment actually did.
Only bidding when predicted uplift times conversion value exceeds cost turns the same logged experiment from a loss into a profit, evaluated on the 16.5% of events the replay method can honestly score.
Compressed for speed, audited on profit.
The X-Learner ensemble is too slow for real-time bidding. Before shipping the compressed model, its realized profit was audited against the teacher's, not just its prediction fidelity.
4 LightGBM boosters, propensity + outcome + effect
distilled on the teacher's CATE predictions as soft labels
Compressing to a depth-5 tree does not cost profit here, it gains 104.7% of it: $0.0932 vs $0.0891 per user, audited by replaying both policies against the identical held-out set. The depth constraint appears to regularize away noise in the teacher's CATE surface that was never predictive of real profit.
What if ad costs doubled overnight?
The student policy re-evaluated across a 10x cost range, with no retraining between points.
At $0.05, the policy bids on 19.8% of traffic to capture volume. At $0.50, a 5x cost increase, it automatically retreats to the top 2.0% of elite persuadables and stays profitable. The implicit threshold (uplift over cost, over value) self-adjusts without retraining.
Measured, not estimated.
Every plot below is a real matplotlib figure the pipeline's own evaluator saved, not a recreation.
Bootstrapped Qini curve
The pipeline's own saved figure, 20-bootstrap 95% CI, AUUC 2448 on this run. A separate run plotted above reports 3125.
Actual lift by predicted decile
Decile 9 dominant, middle deciles genuinely non-monotonic, an unsmoothed result from the pipeline's own evaluator.
Cumulative conversions, bandit vs fixed strategy
The real history_reward trajectory from BanditSimulator.run_replay, not a client-side reconstruction.
Surrogate-tree feature importance
f4 and f3 account for 77% of explanatory power between them. Source for the featureImportance values above.
How the bandit actually decides.
The funnel above shows what LinUCB did to the numbers. This is the disjoint linear bandit itself: a ridge regression per arm, plus the confidence term that makes it a bandit rather than a lookup table.
Per-arm ridge regression
Each arm keeps its own running A = X'X + I and b = X'r, a ridge-regularized linear model updated online as the replay stream is processed, one event at a time.
Parameter estimate θ
θ = A⁻¹b, the arm's current best linear estimate of expected reward given context x. This is the "exploit" half of the decision.
Upper confidence bound
UCB(a) = θ·x + α·√(x'A⁻¹x): the predicted reward plus an exploration bonus that shrinks automatically as an arm accumulates data.
Reward scaled to [0, 1]
Realized profit is rescaled, (profit − min) / (max − min), before each update. Raw dollar values span a wide, mostly-zero, occasionally large range that would otherwise destabilize the linear update.
bid only if predicted_uplift × conversion_value > cost
The exploration bonus is what makes this a bandit and not just a lookup table: an arm the model hasn't seen much of gets a wider confidence interval and a temporary boost to its UCB score, so the policy keeps sampling arms it's uncertain about instead of collapsing onto whatever looked best on the first few events.
Not every number agrees with itself.
Five things this case study found by reading the code and rerunning the pipeline, rather than trusting a single number.
| Source | Baseline profit | Bandit profit | Student R2 |
|---|---|---|---|
| Executed notebook (2025-12-04) | $-0.0708 | +$0.0776 | 0.6955 |
| insights.json (live dashboard) | $-0.0691 | +$0.0720 | 0.6307 |
| README (reported) | $-0.0500 | +$0.0900 | not reported |
All three runs agree qualitatively, loss becomes profit, R2 sits in the mid 0.6s. None of the three headline numbers match exactly, because the 80/20 train and test split has no fixed seed while every other randomized step in the pipeline does.
The primary train/test split has no fixed seed
Every other random step in the pipeline (the CUPED sample, the bandit replay sample, the bootstrap resampling) passes seed=42. The 80/20 split that everything downstream depends on does not. Three independent runs of the same code, the executed notebook, insights.json, and the README, report baseline profit of -$0.0708, -$0.0691, and -$0.05 respectively, and bandit profit of +$0.0776, +$0.072, and +$0.09.
Not fixed. Documented directly instead: all three runs agree qualitatively (loss becomes profit, R squared in the mid 0.6s), and this case study reports a specific run's numbers labeled by source rather than one number presented as the result.
One row of a "live" export is a documented estimate, not a simulation
The Power BI policy comparison table is designed to show three strategies side by side. The middle row, "Uplift Model (Greedy)", has no corresponding simulation anywhere in the codebase. Its profit figure is a hardcoded constant.
Caught by reading the exporter's own code comment, which states this plainly. This case study only cites the two genuinely simulated policies (the A/B baseline and the LinUCB bandit) as measured results.
The Power BI export directory does not exist in this checkout
pipeline.py and exporter.py both reference results/data/, 16 structured CSVs plus the dashboard's hardcoded archetype population percentages (0.85% Persuadable, 9.87% Sure Thing, 89.22% Lost Cause, 0.06% Sleeping Dog). That directory is not present in the source repository.
Those percentages are not reproduced anywhere on this page. Every number shown here traces to an artifact that does exist and was independently read: segment_profile.csv, segment_rules.txt, the notebook's saved outputs, or insights.json.
The live dashboard's trajectory chart is a smoothed reconstruction
The deployed React dashboard's cumulative-profit chart is generated client-side from an easing function anchored only to the final metric values, not the raw history_reward array the bandit simulator actually produced.
This case study uses the real matplotlib figure the pipeline itself saved (bandit_performance.png) instead of recreating the dashboard's illustrative version.
The README rounds a measured number down
The README states the memory optimization saves "approximately 50%". The executed notebook's own logged output shows 1706.49MB to 693.26MB, a 59.4% reduction.
A minor discrepancy, not corrected in the source repo, but reported here as the measured figure rather than the rounder marketing number.
Significant and profitable are not the same.
- A statistically significant lift and a profitable decision are different questions, and this project exists because conflating them is the single most consequential mistake it is built to avoid.
- Off-policy evaluation by replay is a sharp but lossy tool: accepting only 16.5% of sampled events to stay unbiased means every profit number should be reported alongside its effective sample size, not the nominal one fed in.
- Auditing a compressed model against the metric that actually matters, realized profit, rather than a proxy metric, R squared, can overturn the intuitive assumption that compression is strictly a cost.
- Unseeded randomness in an otherwise reproducible pipeline is easy to miss because everything downstream still runs and still looks sensible. It only shows up when multiple runs are actually diffed against each other.
The API is live, and the dashboard reads real numbers.