Personalization Policy Report

The Short Version

This project is about choosing the best website variant for each visitor, using historical randomized logs. The winning approach is a support-aware hybrid: a leak-free empirical-Bayes table when data is thin, switching to a gradient-boosted T-learner once a segment has enough support to justify a more flexible functional form.

Deliverables

The end deliverable is a public GitHub repo plus a recorded video. This HTML page is designed to be the "open this first" artifact for reviewers.

What We Are Solving

This is a one-step contextual bandit problem, not a full reinforcement learning problem. At a single moment, we observe context, choose one action, and get one reward.

For each visitor:
  x = context
  A = available variants
  choose a in A
  observe reward only for chosen a

Goal:
  estimate mu(x, a) = P(convert | context x, variant a)
  serve argmax_a mu_hat(x, a) among available variants

What The Data Looks Like

There are six datasets from May 2, 2025 through May 31, 2025. The observed logs are the only data the policy may train on. The hidden truth files are used by the evaluator and by offline diagnostics, not by the policy.

Randomization And Propensity Refresher

The logs are randomized, but not always uniformly randomized. That is the key distinction. Some datasets show each variant about equally often; others use a known logging probability called a propensity.

Variance, Small Weights, And Shrinkage

The user insight that matters most is variance: when propensities are small, inverse-propensity weights are huge. A technically unbiased estimate can still be useless for choosing an action if it is too noisy.

Inverse propensity weight:
  weight_i = 1 / P(action_i was assigned | context_i)

Example:
  propensity = 0.05
  weight = 20

Effective sample size:
  ESS = (sum weights)^2 / sum(weights^2)

The Harness

The evaluator mimics production by walking forward in time. Training always happens before the window being scored.

Implementation

The final policy is a support-adaptive direct reward model. It starts with an additive empirical-Bayes T-learner, then switches to a histogram-gradient-boosted T-learner once the training history is large enough for flexible nonlinear modeling.

EB policy formula

score(context x, variant a)
  = logit(global_arm_prior[a])
    + sum_feature deviation(feature_value, a)
    + sum_cross deviation(cross_value, a)

shrunk_rate(segment, a)
  = (reward_sum(segment, a) + k * parent_prior[a])
    / (count(segment, a) + k)

GBM and hybrid logic

gbm_tlearner:
  for each arm a:
    fit HistGradientBoostingClassifier on rows where variant_id == a
    predict P(reward=1 | x, arm=a)

hybrid_eb_gbm:
  if n_train < 50,000 or any arm has fewer than 2,000 rows:
    use seg_eb
  else:
    use gbm_tlearner

What Kind Of ML Model Is This?

The shipped model is a direct-method, multi-action T-learner for a contextual bandit. The final submitted policy is a hybrid: EB shrinkage for low-support regimes, gradient boosting for high-support regimes. Both estimate the expected conversion rate for every variant in the current context, then choose the highest score among live variants.

The exact model shape

hybrid_eb_gbm is a support-gated T-learner. "T-learner" means each action has its own estimated outcome surface:

mu_a(x) = E[reward | context = x, action = a]

serve:
  choose argmax_a mu_hat_a(x)
  subject to a being in available_variants

In low-support windows, the policy uses shrunk segment tables in log-odds space. The prediction for a variant is:

score(x, a)
  = global arm prior
    + country effect for arm a
    + language effect for arm a
    + platform/device/numeric-bin effects for arm a
    + selected country-cross effects for arm a

Is this uplift modeling?

It is related, but the final policy is not only an "uplift model" in the narrow binary-treatment sense. Classic uplift modeling asks, "How much better is treatment than control for this user?" Here there are multiple variants and every visitor must receive one of them. We can express the decision as uplift against a baseline:

uplift_a(x) = mu_a(x) - mu_baseline(x)

But because the baseline term is common when comparing arms, choosing the largest mu_a(x) gives the same action as choosing the largest uplift. So the submitted policy models absolute reward per action, while the business interpretation is "personalized lift over best fixed."

How T, X, and R learners fit

Walk-forward retraining, not RL

We are not discovering by changing the future data collection process in this benchmark. The harness replays fixed historical windows. At each time step, we train only on the past and score on the next unseen window:

iteration 11:
  train on all data through iteration 10
  recommend for iteration 11

iteration 15:
  train on all data through iteration 14
  recommend for iteration 15

That is why there is no data leakage: the model can keep retraining as time moves forward, but it never uses reward, action, or truth from the window it is currently scoring.

How to think about stale data and drift

Stale data matters when P(reward | context, action) changes over time — then old rows are not just less fresh, they are actively misleading. zephyr is the case: the best arm flips at the midpoint of the timeline.

A full-history model averages two incompatible regimes, so it keeps "remembering" that variant_2 used to be great and adapts too slowly. Recency weighting discounts old rows — weight = 0.5 ** (age_days / half_life), so with a 4-day half-life a row 8 days old counts as a quarter of a row. That is why on zephyr: EB full-history 47.4% → EB recency 58.9%, and the time-aware gbm_tlearner reaches 75.9%.

But recency is not free: it trades bias from stale data for variance from a smaller effective sample size. Forgetting old rows on a stable dataset throws away valid signal. A single global 4-day half-life proved this — it helped zephyr but hurt every stationary dataset:

vega is hit hardest (−7.1) because it already has only 6,000 rows — recency made a sparse problem sparser. So "always weight recent data higher" isn't wrong, it's a bias-variance choice: keep long memory while the process is stable, switch to short memory only once it has changed.

Observability Displays

These plots make the data and harness inspectable. They are useful for the video because they show why the final policy is shaped the way it is.

Results

Full walk-forward evaluation over 50 iterations, comparing the starter baselines, provided examples, our data-display experiments, and the final EB policies.

Why the hybrid wins

hybrid_eb_gbm is not a new model — it is a per-window router between two base learners with opposite error profiles, gated on a leak-free signal: how much training data exists so far.

The switch is on support, every window

each walk-forward window:
  if len(train) >= 50,000  AND  every arm has >= 2,000 rows:
      use gbm_tlearner        # rich enough to trust the flexible model
  else:
      use seg_eb              # sparse / early -> shrink, do no harm

The gate reads only training support, so it is leak-free. Because it runs every window, it is also a cold-start-to-mature switch over time: the data-poor early windows use EB (which converges faster from little data), and once history is large the same dataset graduates to GBM.

What the gate produces, dataset by dataset

Edge Cases

These are the cases to call out in the video. They show that the solution is not just chasing a leaderboard number.

Productionization

In production there is no truth file and no guaranteed clean randomization. The machinery we deliberately set aside for the offline score — propensities, off-policy evaluation, exploration, drift handling — is exactly what comes back.

Off-policy evaluation when counterfactuals are unavailable

Offline here we score against the hidden per-arm oracle. In production there is no oracle, so you estimate a new policy's value from logged data. The doubly-robust (DR) estimator is the workhorse — it combines a reward model with propensity weighting:

V_DR(pi) = average over logged rows of:
    m_hat(x, pi(x))                                # reward-model guess for the new policy's choice
  + 1{a == pi(x)} / p(a | x) * ( y - m_hat(x, a) ) # correction, ONLY when the log happened to show
                                                   # the same arm the new policy would have chosen

It is "doubly robust" because the estimate stays consistent if either the reward model or the propensities are correct — two parachutes, you only need one to open. Intuition: start from the model's prediction; on the rows where the logged arm matches the new policy's choice, correct it by the propensity-weighted residual (actual − predicted). SNIPS (self-normalized) and weight clipping tame the variance that small propensities create.

Exploration vs. greedy exploitation

Greedy argmax is optimal for this offline score, but in production greedy starves the arms it disfavors — without overlap you can no longer estimate those arms, and the model decays. You keep a controlled amount of exploration (Thompson sampling or ε-greedy, or a small randomized holdout) to preserve overlap and keep OPE valid. This is the Fisher-information / value-of-information idea, scoped concretely: spend exploration where an arm's value confidence interval still overlaps the current argmax and the segment is high-traffic — i.e. where information is both uncertain and decision-relevant.

Retraining cadence

Keyed to drift, not a fixed clock. Monitor per-arm and per-segment CVR and the model's rolling log-loss; a change-point detector (CUSUM / Bayesian online change-point) triggers down-weighting of stale data — the adaptive version of seg_eb_recency. zephyr is the cautionary tale: a static expanding-window model craters when the best arm flips mid-stream, while an adaptive one recovers within a few windows.

Serving latency

The empirical-Bayes table is a hash lookup plus argmax — microseconds per impression, trivially cacheable. A per-arm gradient-boosted model costs n_arms model evaluations per impression (10× on rotation's ten arms). The shipped hybrid_eb_gbm serves the cheap EB path for thin segments and the GBM only where support justifies it, so most traffic takes the fast path; hot-segment GBM scores can be precomputed and cached if needed.

Learnings And Discussion

This section consolidates the discussion from our thread, the local research log, and the parallel agent's strategy notes.

What came from the parallel work

The parallel strategy file emphasized the same core framing: this is a contextual bandit, not RL; the scoring oracle makes greedy exploitation appropriate for the take-home; propensities should be handled carefully; and EB shrinkage is the right response to sparse or high-variance cells. It also flagged two important harness details we adopted: the stored oracle_value behavior on rotation, and the canonical score_variants column-order contract.

What is next

1. Adaptive recency: keep the zephyr gain without hurting stationary datasets.
2. Propensity experiment: add clipped/stabilized IPW as a measured ablation, mainly for helios and rotation.
3. LightGBM tuning: tune the LightGBM hybrid (currently ~69.1% on first-pass hyperparameters) to match or beat the sklearn hybrid, as the faster production backend.
4. Production OPE: add IPS/SNIPS/doubly robust evaluation with confidence intervals.
5. Exploration design: use Fisher-information or optimal-design thinking to collect data where uncertainty is decision-relevant.

Video Outline

This is the clean story for the recorded submission. The goal is to sound rigorous without burying the reviewer in implementation detail.

Runbook

Commands used to reproduce the policy comparisons and regenerate this report's figures.

uv run run_eval.py --policies random,my_policy,best_fixed,example,feature_demo,seg_country,seg_eb,seg_eb_recency,gbm_tlearner,hybrid_eb_gbm,lgbm_tlearner,hybrid_eb_lgbm --workers 1 --threads 2
uv run plot_results.py
uv run python solution/observability.py
uv run python solution/eval_observability.py
uv run python solution/arm_sensitivity.py

AVERAGE across headroom datasets (helios, meadow, rotation, vega, zephyr)
  random         mean captured=   0.2%
  my_policy      mean captured=   0.3%
  best_fixed     mean captured=   0.3%
  example        mean captured=  18.3%
  feature_demo   mean captured=   3.6%
  seg_country    mean captured=  42.7%
  seg_eb         mean captured=  62.7%
  seg_eb_recency mean captured=  61.9%   (drift ablation; helps zephyr, hurts stationary)
  gbm_tlearner   mean captured=  65.4%   (boosted trees; overfits sparse vega -> 32.7%)
  hybrid_eb_gbm  mean captured=  69.6%   <-- SHIPPED: EB when thin, GBM when rich
  lgbm_tlearner  mean captured=  62.9%   (LightGBM backend; faster than sklearn)
  hybrid_eb_lgbm mean captured=  69.1%   (LightGBM hybrid; ~same score, faster)