RL in Recommendation and Advertising

One-liner: this page covers how RL is actually used in recommendation and advertising systems — why the problem is naturally an RL setting, why the industry started with contextual bandits, how list ranking and real-time bidding put RL to work, and the ever-present engineering dilemma that offline evaluation is hard while online trial-and-error is expensive.

1. Why recommendation/advertising is an RL setting

1.1 Three "sequential" properties

Traditional recommendation treats "what to show you" as an independent classification/ranking problem (each request decided on its own). But a real business has three sequential properties that open the door for RL:

Property	What it means	Modeling implication
User state	The user is an evolving state: what they've watched and clicked changes future behavior	Per-request decisions lose the state; MDPs fit naturally
Delayed feedback	Click → conversion (order/registration) often lags by minutes to days	Rewards are delayed, so value learning beats immediate supervision
Exploration–exploitation	Pushing unknown content is how you discover new interests, but exploration costs short-term revenue	This is the raison d'être of bandits/RL

In short: the recommendation/advertising problem is not "predict what the user likes" but "decide what to show next." The former is supervised learning (click-through-rate prediction); the latter is sequential decision-making — RL's home turf.

1.2 Mapping business metrics to "rewards"

User (state s) → platform serves content (action a) → user behavior (feedback) → business reward r

s = user profile + recent behavior sequence + context (time/device)
a = one item or a candidate list (slate)
r = click? conversion? dwell time? GMV? — a weighted sum set by the product goal

TIP

Recommendation/advertising is the best teaching ground for reward engineering: click-through rate, conversion rate, GMV, dwell time, diversity, and freshness each carry their own weight, and the weights are product decisions. For the same user, crank up the "dwell time" weight and the system serves more addictive content; crank up "GMV" and it pushes pricier goods. Behavior grows however the reward is shaped — see reward engineering for the details.

2. First stop: the contextual bandit

2.1 Why the industry started with bandits

Full RL requires modeling "state transitions," which is expensive to build and hard to validate. The contextual bandit cuts state transitions out: each request is decided independently, the reward shows up immediately, and the goal is "pick the best action for the current context." It suits a first RL-style upgrade of recommendation because:

• It's simple to implement, updates online, and comes with regret guarantees;
• It's compatible with existing CTR models (build the bandit on top of CTR estimation);
• Launch risk stays controllable.

Aspect	Contextual bandit	Full RL
State	Has context, no state transition	State + transition
Decision horizon	Single step	Multi-step sequence
Needs value function/planning	No	Yes
Engineering complexity	Low	High
Best for	Cold start, single-decision optimization	Sequential recommendation, long-term value

2.2 Three core algorithms

Algorithm	Idea	Characteristics
ε-greedy	Explore randomly with probability ε	Simplest; exploration is undirected
LinUCB	Linear model + confidence bound: CTR prediction + c·√(xᵀA⁻¹x)	Theoretically grounded; common in industry
Thompson Sampling	Sample from the posterior over parameters, act on the sample	Bayesian; works well in practice

Li et al. (2010) compared LinUCB against ε-greedy and others on Yahoo! News recommendation: under offline evaluation (replay on historical logs), LinUCB improved click-through rate by roughly 12%+ over human-selected content. The full derivation of this classic case is on the multi-armed bandit page.

2.3 The limits of bandits

A bandit assumes "this decision doesn't affect the next" — but in recommendation, once you serve a video, the user is looking at a completely fresh feed next time, so the state really does evolve. Hence the industry consensus: a bandit is the first stop, not the destination. When you need to "factor in long-term return" (say, turning users into regulars), it's time for full RL.

2.4 Cold start: the most mature bandit use case

The most resource-starved scenario in a recommendation system is the cold start: new content has no click data, and new users have no behavior history. Uniform random exploration is terrible value for money here, and the bandit's "uncertainty-aware" nature is exactly what helps:

Cold-start object	What the bandit does	Typical setup
New content	Decide how much traffic to send it, and to whom	UCB/TS, exploration decays with exposure
New users	Probe broadly first, then converge on interests	Larger ε, decreasing as behavior accumulates
New item pools	Rank the recommendation pool	Thompson Sampling to diversify items

Engineering detail: industrial implementations almost never run a bandit standalone — they embed it inside the CTR estimation model. A base predictor does the heavy lifting, and the bandit layers an "uncertainty correction" on top. That preserves the ML model's core strength while adding exploration. This two-stage "base model + exploration layer" architecture is the mainstream form of bandit deployment in recommendation and advertising.

3. List ranking: from pointwise to sequence-wise RL

3.1 Three ways to model ranking

pointwise:  score each item independently → rank
pairwise:   compare pairwise item preferences
listwise:   take the whole list as input/output → sequential decision-making

The opening for RL is listwise (the list view): items in a list have combinatorial effects — what you show and where, whether items duplicate each other, and the user will probably click only one. In RL terms: the action is choosing the whole slate, not picking items one by one.

3.2 Classic case: YouTube's slate RL

YouTube (Chen et al., 2019) published a framework that makes the combinatorics tractable: decompose "choose the top-k list" into additive terms, and solve off-policy evaluation — because online logs come from the old policy (the behavior policy), you can't just "compute the new policy's return directly from the old policy's data." Their fix is top-k off-policy correction: the item at each position is reweighted by the difference in its probability of being selected, yielding an unbiased (or at least low-bias) value estimate.

Key idea: recommendation logs are sampled by the "old policy", so evaluating
the "new policy" requires importance weighting:
  Û(π_new) ≈ Σ_logs  [ π_new(a|x) / π_old(a|x) ] · r

But the top-k combinatorial space is too large; YouTube decomposes it
per position, which makes it engineering-feasible.

3.3 A DQN variant for Taobao search

Zhao et al. (2018) formulated Taobao search recommendation as an MDP and trained a ranking policy with a DQN variant. The engineering challenges are highly representative:

• Huge action space: one ranking pass must handle thousands of candidate items, so enumerating actions is infeasible — they resolve it by deciding sequentially over the candidates.
• State representation: the user behavior sequence (clicks/purchases) plus candidate item features.
• Reward: a weighted combination of user behavior signals (click, add-to-cart, purchase, etc.).
• At launch they used a "mixed policy": the RL policy decides the items for only some positions, while the rest keep the legacy ranking, to keep risk under control.

3.4 Combinatorial explosion in slates and engineering decompositions

The first enemy of list-ranking RL is combinatorial explosion: choosing and ordering 10 items out of 1000 candidates gives on the order of 1000^10 permutations, so any "enumerate all slates" approach is dead on arrival. Four decomposition strategies are used in practice:

Strategy	How	Cost
Decide position by position	Decide one position at a time; already-chosen items are excluded (Taobao's approach)	Ignores high-order cross-position interactions
Additive decomposition	Assume slate value ≈ sum of item values (YouTube)	Ignores complementarity ("if A is in, B isn't needed")
Candidate pre-filtering + RL ranking	Pre-filter to a few hundred with a lightweight model, then RL ranks the top slots	Pre-filter quality becomes the bottleneck
Implicit order modeling	Feed positions as a sequence and model with an RNN/Transformer	High training and inference cost

The key trade-off: the more important cross-position interactions (duplication, complementarity, competition) are, the more you should use full sequence modeling — but sequence modeling is more expensive and harder to evaluate. Industry defaults to starting with "position-by-position + decomposition," and only upgrades once ablations confirm significant combinatorial effects. That's exactly the "start simple" discipline from the evaluation in practice page.

WARNING

The gain numbers in the Taobao/YouTube papers (e.g., "conversion rate up X%") are A/B results under specific traffic, specific time windows, and a mixed policy. They're real in engineering terms, but they don't support the general conclusion that "RL always beats existing ranking." When reading industrial papers, go straight to the "experimental setup" section: what's the control group, what share of traffic, were only some positions changed — this is precisely the "the evaluation protocol determines the conclusion" point from the evaluation & benchmarks page.

4. Real-time bidding (RTB): RL in auctions

4.1 The ad auction mechanism

In programmatic advertising, every ad impression goes through a real-time auction (RTB): advertisers (or their agents) bid for the impression, and the highest bidder wins. The decision becomes "given an impression, how much to bid":

state s: info about this impression (user profile, page, context) + budget remaining + time remaining
action a: the bid (continuous or discretized into buckets)
reward r: win → an impression, settled on click/conversion; lose → 0
constraint: budget (how much to spend per day) — this is a constrained MDP

4.2 RL bidding strategies

Cai et al. (2017) and others model bidding as RL, learning the "bid–win–return" mapping with the goal of maximizing conversions under a budget constraint. Public technical reports from industry teams (Alimama, ByteDance, Tencent Ads, and others) show that bidding strategies have broadly gone RL (variants like DRLB and non-linear bidding), though the details are mostly internal.

4.3 A multi-agent view: auctions are games by nature

The key insight about RTB: all advertisers bid with strategies, and the market-clearing price emerges from the behavior of every participant — this is fundamentally a game problem in multi-agent reinforcement learning. Advertisers learning to bid independently will get into an "arms race" that pushes bids up. So the industry's pragmatic move is to treat the other participants as "environment noise" and learn a one-sided optimal bid with bandits/RL, rather than explicitly modeling opponents (that's MARL research territory).

4.4 Engineering points for bidding RL

Several details are unavoidable when productionizing bidding RL:

1. Second-price auction: the winner pays the second-highest bid → bid ≠ price paid
   The RL objective is "win cheap impressions" — the bidding policy should exploit this
2. Budget constraint: with a limited daily budget, whether to bid "loose early, tight late"
   or the reverse depends on the goal
   → often encode the budget in the state, using "remaining budget / remaining time" features
3. Delayed settlement: clicks/conversions happen long after the bid → delayed reward,
   requiring value estimation rather than immediate supervision
4. Bid granularity: discretized buckets vs. continuous bids — affects the design of the RL action space

These details are highly isomorphic to the order-execution problem in RL for financial trading: same idea of "use a policy to optimize bids/orders, rewards are delayed, constraints (budget/slippage) are explicit." Ad bidding is financial RL with a clean simulator — which is also why it could land first.

5. Engineering reality: hard offline evaluation, cautious online steps

5.1 Why evaluating recommendation RL is so hard

Obstacle	Explanation
Off-policy evaluation	Online logs come from the old policy; computing the new policy's return directly is biased, so counterfactual correction (importance weighting) is needed
The counterfactual problem	Logs contain "no feedback for content that was never served," so you can't know what would have happened if you had
Delayed feedback	Conversions lag by days, so training and evaluation time windows are hard to align
Business guardrails	Too much exploration hurts revenue immediately; too little and you never learn

Bottou et al. (2013), Counterfactual Reasoning and Learning Systems, is the foundational treatment of this problem: "running A/B tests on historical logs" is counterfactual reasoning in essence, and unbiased correction is mandatory. This is also one of the core difficulties discussed on the offline RL page.

5.2 The industrial three-tier evaluation ladder

Level 1  Offline: log replay, counterfactual evaluation → quickly weed out clearly bad policies
Level 2  Shadow: the new policy outputs alongside without serving traffic → see "what would happen if launched"
Level 3  Online A/B: small traffic slice (e.g. 1%–5%) → full rollout only after statistical significance

Every tier has its traps: offline evaluation gets fooled by the log distribution, shadow mode ignores "the policy would have influenced user state," and A/B needs heavy traffic to reach significance. The engineering discipline is progress tier by tier, skipping none.

DANGER

The classic "evaluation cheating" in recommendation/advertising is tuning hyperparameters against the same historical logs until the offline metric looks best, then claiming the policy works. That's tuning on the test set, so the offline score is inevitably inflated. The right approach is to split the offline data into time windows too: a training window, a validation window, and an evaluation window, strictly in temporal order. This is the protocol issue the evaluation & benchmarks page keeps stressing.

5.3 Implementation details of counterfactual evaluation

"Evaluating a new policy on historical logs" (off-policy evaluation, OPE) has standard implementation points:

Basic formula (importance weighting):
  V̂(π_new) = (1/N) · Σᵢ [ π_new(aᵢ|xᵢ) / π_old(aᵢ|xᵢ) ] · rᵢ
  weight = P(new policy picks this action) / P(old policy picks this action)

Three engineering fixes:
  1. Clip the weights: prevents a single sample's weight from exploding → biased but variance drops a lot
  2. Self-normalize: divide weights by their sum → better unbiasedness
  3. Evaluate only where the old policy had coverage: when the new policy's favorite
     actions barely appear in the logs, weight variance explodes

Three questions to judge whether an offline evaluation is trustworthy: Which behavior policy produced the logs? Were the weights clipped/normalized? Is action coverage sufficient? — if any one of these can't be answered, the evaluation can't ground a decision. This is the heart of "offline evaluation is hard" on the offline RL page.

6. The cost of exploration and business guardrails

6.1 The "money" question of exploration

In recommendation/advertising, exploration isn't just an academic problem — it's real money: take 1% of traffic for "unknown content" experiments, and if that content earns only half the average revenue, you lose about 0.5% of GMV per day — an astronomical figure for a mega-platform.

Strategy	Exploration share	Revenue impact	Best for
ε-greedy	Fixed ε (e.g. 1%–5%)	Direct loss ≈ ε × the gap	Simple, controllable
UCB/TS	Adaptive	Exploration concentrates where uncertainty is high	Cold start, new content
Personalized exploration	Tuned per user/context	Low	Mature platforms

6.2 Business guardrails

• Fallback policy: before going live, a new policy must be "one-click rollback"-able to the old policy at any moment.
• Exploration quota: cap exploration per day/per user; beyond the quota, switch back to exploitation.
• Hard constraints on freshness/diversity: even if RL says "serve it," the content must pass compliance and diversity checks — RL only ranks; it doesn't override product boundaries.

These guardrails are summarized as the general "safety guardrails" principle on the common pitfalls & anti-patterns page.

7. The role of offline RL here

7.1 Why recommendation is offline RL's biggest application field

Recommendation/advertising has RL's scarcest resource: massive, continuous, real historical interaction logs (hundreds of millions of entries per day). That fits the offline RL setup exactly: "only historical data, no further environment interaction."

• Safe: offline training doesn't touch live traffic, so there's no exploration risk.
• Cheap: no expensive online experiments needed.
• Compliant: no extra personal information is collected for exploration.

7.2 Forms of offline RL in recommendation

Form	What it is	Challenges
Offline pretraining + online fine-tuning	Learn offline on the large logs first, then continue online with a small traffic slice	Distribution shift, value overestimation
Conservative methods (CQL/IQL ideas)	Prevent value hallucination on OOD actions	Sensitive to tuning; many hyperparameters
Counterfactual ranking	Evaluate new rankings with importance weighting	High variance; needs lots of logs

Note the distinction: many papers claiming "offline RL" for recommendation are actually counterfactual evaluation + supervised re-ranking, not full offline RL training. When reading papers, first ask "did it really learn a policy, or just evaluate on logs?" Methodological details are on the offline RL page.

7.3 A landing path: from bandit to full RL

Here's an upgrade roadmap for teams that want to land in this field:

Stage 1  Bandit cold start + counterfactual evaluation → steady gains, low risk
Stage 2  Partial list RL (RL for some positions, mixed policy) → validate sequential value
Stage 3  Full sequential policy + offline RL pretraining → optimize long-term return
Each stage must prove it beats the previous one before moving on; otherwise stay put.

Why this path is mandatory: the payoff of full RL (long-term return) only materializes once the whole engineering chain works — and every link in that chain (environment, reward, evaluation, guardrails) can zero out the gains. Staging keeps the risk under control and leaves every step rollback-able. This aligns exactly with the "baselines first, iterate in small steps" principle on the RL design principles page.

7.4 The mathematical relationship between bandits and full RL

From bandits to RL is a continuous spectrum, and knowing where you stand on it helps with method selection:

Multi-armed bandit (no context)   : pick one action per round, immediate reward, no state transition
Contextual bandit                 : the choice depends on context x, but actions don't affect the future
RL (full MDP)                     : state evolves, actions affect future returns

The only difference is whether there is a "state transition". A bandit removes the
temporal dimension, so it is a "degenerate RL" — which is why the
[multi-armed bandit](/concepts/bandits) page positions it as a
"simplified laboratory for exploration and exploitation".

Engineering corollary:

Problem shape	Which layer	Why
Each decision is independent (recommend one product to a new user)	Contextual bandit	State transitions are negligible
Decisions affect what follows (serving A in a video feed affects the next impression)	Full RL	State evolution can't be ignored
In between	Bandit + state features	Use context as a state approximation

Many industrial systems billed as "RL ranking" actually operate on "context + single-step reward," which is essentially a contextual bandit. That's not a bad thing — bandits are simple, stable, and rollback-able. But when reading papers and designing architectures, know where you stand on the spectrum, and don't attribute bandit gains to RL.

7.5 Engineering details of delayed feedback and sequence modeling

The most maddening part of recommendation/advertising is delayed feedback: a user clicks but orders three days later — which recommendation gets the reward? Four engineering responses:

Approach	How	Cost
Waiting window	Settle the reward N days after the decision	Slow training, stale data
Immediate proxy reward	Use "click-through" as the reward proxy	Can be optimistic (click ≠ conversion)
Decay/probabilistic attribution	Attribute conversions with time decay	Needs an attribution model
Sequence state modeling	Encode "what was last served and what the user did" into the state	Requires complete behavior-sequence data

Relation to RL: delayed feedback is fundamentally the "delayed reward" problem — exactly what "TD and credit assignment" on the value learning page addresses (long-term value estimation discounts distant rewards back). But in industrial practice, few teams deploy full RL just for this; most use "conversion attribution + click proxy reward" as an engineering approximation — pragmatic, but know where the approximation lies.

8. Case roundup: public technical routes

System	Scenario	RL form	Highlights from public material
Yahoo! News	Article recommendation	LinUCB contextual bandit	The classic bandit experiment (Li et al. 2010)
YouTube	Video lists	Slate RL + off-policy correction	Tractable decomposition of combinatorial actions
Taobao Search	Search ranking	DQN variant (launched with a mixed policy)	Huge action space, user-behavior-sequence state
Alimama (DeepLight)	Ad CTR estimation	(Deep CTR engineering, not RL proper)	Feature-interaction acceleration; shows the full display-ads engineering stack
RTB across platforms	Real-time bidding	Bandit/RL bidding	Budget constraints, game-theoretic view

INFO

Since 2020, "ByteDance and others' public engineering" has appeared as tech blogs and papers covering bandit/RL practice in recommendation and advertising (UCB cold start, Thompson-Sampling item diversification, bidding strategies), but most of it has no citable formal publication. This page lists only cases with public literature, to avoid spreading unverifiable claims.

Further reading

• Multi-armed bandits — LinUCB derivation, Thompson Sampling, and the definition and intuition of regret.
• Exploration & exploitation — a unified view of exploration in deep RL and its business cost.
• Offline RL — recommendation is offline RL's biggest application field; includes the CQL/IQL mechanisms.
• Evaluation & benchmarks — hard offline evaluation, counterfactual correction, and evaluation protocols.
• Multi-agent RL — the game-theoretic view of RTB auctions and CTDE.
• RLHF and alignment with human feedback — modeling human preference data (Bradley–Terry) and its isomorphism with click/conversion preferences in recommendation.

References

• Li, L., Chu, W., Langford, J., & Schapire, R. E. (2010). A Contextual-Bandit Approach to Personalized News Article Recommendation. WWW 2010. (arXiv:1003.0146)
• Auer, P., Cesa-Bianchi, N., & Fischer, P. (2002). Finite-time Analysis of the Multiarmed Bandit Problem. Machine Learning, 47, 235–256. (UCB1)
• Thompson, W. R. (1933). On the Likelihood that One Unknown Probability Exceeds Another in View of the Evidence of Two Samples. Biometrika, 24(3–4), 285–294.
• Bottou, L., et al. (2013). Counterfactual Reasoning and Learning Systems: The Example of Computational Advertising. JMLR 14, 3207–3260. (arXiv:1209.6875)
• Zhao, J., et al. (2018). Deep Reinforcement Learning for Search Recommendation in Taobao. arXiv:1801.02057.
• Chen, M., et al. (2019). Top-K Off-Policy Correction for a REINFORCE Recommender System. WSDM 2019. (arXiv:1812.02353)
• Cai, H., et al. (2017). Real-Time Bidding by Reinforcement Learning in Display Advertising. WSDM 2017. (conference paper)
• Wang, R., et al. (2021). DeepLight: Deep-Lightweight Feature Interactions for Accelerating Inference in Ad Click Prediction. Alimama team technical paper.