Reinforcement Learning Handbook

Appearance

Frontier Advances

On this page The RL frontier of the 2020s — world models (DreamerV3, TD-MPC2), offline RL (IQL/CQL), the RLHF and DPO family, scalable RL (massive parallelization, the JAX ecosystem), RL for LLM reasoning (R1, RLVR), and general agents — with a read on where the trends point.

Frontier Advances ​

One-liner: this page is a telescope and a filter for the RL frontier of the 2020s — it takes six directions that are exploding or about to explode (world models, offline RL, the alignment family, scalable RL, RL for reasoning, and general agents), dissects each one, and tells you "what the core mechanism is, who the representative work is, and whether it's a real breakthrough or hype." It's for anyone who wants to track the frontier, find a research direction, or judge which papers are worth following. After reading it, you'll browse arXiv's daily feed with a critical eye.

Bottom line up front: the 2020s brought a tectonic shift in where RL gets applied. Over the previous decade, RL's signature victories came from games and robotics (DQN, AlphaGo, PPO on MuJoCo). In the 2020s, the most important new battlegrounds for RL are language models and agents: RLHF taught large models to "speak human," DeepSeek-R1 used RL to teach models to "reason," and WebArena uses RL to train browser agents. Meanwhile, scalable infrastructure that "makes RL more like engineering" (the JAX ecosystem, massive parallelization) is maturing fast. These six threads are the sequels to the milestone papers in Close Reading of Classic Papers.

1. Frontier Overview: A Map of Six Tables ​

DirectionCore QuestionRepresentative Work (Year)Key MechanismsMaturity
World modelsCan we plan "in our heads"?DreamerV3 (2023), TD-MPC2 (2023), MuZero (2020)latent-space dynamics, imagined rollouts, fixed hyperparametersMature in research, heating up in engineering
Offline RLCan we learn from historical data alone?IQL (2022), CQL (2020), RvS/DT (2021)OOD actions, conservatism, implicit QMature in research, cautious adoption
Alignment familyMake models obey human preferencesInstructGPT (2022), DPO (2023), RLAIF (2022)preference optimization, KL constraints, AI feedbackProduction-ready, problems unresolved
Scalable RLHow compute turns into algorithmic gainsBrax (2021), PureJaxRL (2025)JAX, GPU parallelism, algorithms as pure functionsMaturing rapidly
RL for ReasoningCan RL teach models to "think"?DeepSeek-R1 (2025), RLVRverifiable rewards, group-relative advantages, long chains of thoughtEarly explosion
General agentsCan RL drive general-purpose agents?WebArena (2023), VLA (RT-2 et al.)environments as APIs, action-execution rewardsExploration phase
text
Internal structure of the three main lines (suggested reading):
· Learn the environment: world models ──► offline RL (treat historical data as "environment logs")
· Set the objective: the alignment family ──► RL for reasoning (turn "objectives" into "verifiable rewards")
· Build the platform: scalable RL ──► general agents (needs massive parallelism and real environment APIs)

2. World Models: Plan "in Your Head" Instead of Bumbling Around in the Real World ​

1. From MuZero to World Models ​

The paper map covered MuZero's (2020) core idea: extending AlphaZero-style planning to "unknown environments" — learning representations, dynamics, and rewards in latent space, then running MCTS with the learned model. The world-model line of follow-up work pushed it in two directions: task generalization (one set of hyperparameters for all tasks) and continuous control (robotics).

2. DreamerV3 (Hafner et al., 2023, arXiv:2301.04104) ​

  • Mechanism: learn a latent-space world model (an RSSM — recurrent state-space model) that predicts the next representation and reward, then train an actor-critic on "imagined" trajectories, with no real environment interaction. Three engineering keys: discrete latent representations (symbolic latents — more stable learning), fixed hyperparameters across domains (the same set for Atari, Minecraft, and robotics), and return normalization (the symlog transform, which tames differences in reward scale).
  • Results: outperforms specially designed methods on 150+ tasks — the strongest proof yet of the "one algorithm for every environment" philosophy.
  • How to read it: don't just count "how many environments it won"; look at how it handles "differences in reward scale" and "long-term memory" — that's the transferable engineering mechanism. Background in model-based RL.

3. TD-MPC2 (Hansen et al., 2023, arXiv:2310.16828) ​

  • Mechanism: a continuation of the TD-MPC family. Do model predictive control (MPC) and temporal-difference learning simultaneously in latent space: learn a joint state-action representation, align it with a latent-consistency objective, and add target networks plus regularization. The core innovation is one model, many tasks: a single model masters 80+ continuous-control tasks (from 5 DoF to 84 DoF).
  • Significance: world models go from "one environment each" to "multi-task with shared representations," paving the way for multi-task learning in robotic manipulation.
  • How to read it: focus on how "multi-task sharing" is achieved through latent consistency — that's what distinguishes it from simply "one model per task."

4. An Honest Assessment of the World-Model Hype ​

ClaimReality
"World models plan without real interaction"Training the world model itself still requires massive real or simulated interaction; imagined rollouts accumulate model error
"One set of hyperparameters rules them all"DreamerV3 pulled it off, but at no small compute cost; "universal hyperparameters" rest on accumulated normalization and regularization engineering
"World models will replace model-free methods"Not in the near term: model-free methods (PPO/SAC) are still simpler and more robust; world models are the other road to sample efficiency and planning

3. Offline RL: Can You Learn from Historical Data Alone? ​

1. The Core Difficulty: Value Hallucinations on OOD Actions ​

The offline RL setup: you only have a batch of historical interactions (s,a,r,s') — no further environment interaction. The difficulty isn't "training," it's evaluation: actions the policy learns to take may fall outside the data distribution (OOD), and the Q function hands out inflated value estimates for actions it has never seen — because max_a' will pick "the hallucination with the most noise." This is the "bootstrapping hallucination" discussed on the offline RL concept page. Three mainstream defenses:

IdeaRepresentativeHow It WorksCost
Constrain the policyBCQ (arXiv:1812.02900)only pick actions close to the data distributionceiling capped by the data distribution
Pessimistic valueCQL (arXiv:2006.04779)penalize the value of OOD actions in the Q-learning objectiveone extra hyperparameter α; too pessimistic and you underperform
Implicit QIQL (arXiv:2110.06169)learn a value backbone with expectile regression; never query OOD actionssensitive to hyperparameters; policy quality depends on value quality

2. Choosing Between IQL and CQL ​

  • CQL: the value-method family; suits "decent data coverage, want to squeeze everything out of policy optimization"; tuning α requires an offline validation protocol (which is itself hard).
  • IQL: never touches OOD actions, so it's the better-behaved engineering citizen; the D4RL benchmark proved it one of the most robust general choices of the past two years, and it's the default starting point for many downstream systems (e.g., training on robotic data).
  • Simple rule of thumb: get a baseline with IQL first, then bring in CQL for comparison — the performance gap between the two is usually far smaller than the gap created by data quality.

3. The RvS Debate: Does Offline RL Even Need "RL"? ​

From 2021 to 2023 there was a famous debate: return-conditioned supervised learning (RCSL) vs. standard offline RL. The representatives are Decision Transformer (arXiv:2106.01345 — model trajectories as sequences, conditioned on the "target return") and RvS (Emmons et al., ICLR 2023). Their claim: "sequence modeling + supervised learning is enough; you don't need dynamic-programming-style value learning." The IQL/CQL camp fired back: "RCSL underperforms standard offline RL on key benchmarks."

The engineering verdict of that debate:

  • RCSL-style methods (DT, RvS) are simple, stable, and naturally fit transformer architectures — good enough when "trajectory data is plentiful and the environment structure is simple";
  • Standard offline RL methods (IQL, CQL) are stronger when you need to beat the data distribution's performance, but tuning and evaluation are more complicated;
  • Don't mistake "who won in the papers" for "the answer in engineering" — run both baselines on your data first.

4. The Alignment Family: From RLHF to DPO, and the Return of "Online RLHF" ​

1. The Rise of the DPO Family ​

InstructGPT's three stages (SFT → RM → PPO) are costly and hard to keep stable. DPO (arXiv:2305.18290) uses the closed-form solution of the KL-constrained RLHF objective, implicitly encoding the reward model into the policy and training with nothing but preference pairs in a single classification-style step. It put alignment within reach of the open-source community (Zephyr, simplified reproductions of Llama-2-Chat). Variants like KTO and IPO followed, but the core insight is the same: preference optimization has an analytical solution; you don't have to model the reward explicitly. Mechanism details on the RLHF concept page.

2. RLAIF: Let AI Do the Labeling ​

Constitutional AI (arXiv:2212.08073, Bai et al., 2022) uses a written "constitution" to have AI judge AI outputs and propose revisions, then trains the RM on AI feedback — RL from AI feedback (RLAIF). It targets RLHF's biggest bottleneck: human preference labeling is too expensive, too slow, and inconsistent. The limitation is just as clear: AI judges inherit the model's biases, and there's a ceiling on feedback quality.

3. Online RLHF: Reward Model and Policy Co-Evolve ​

The post-2024 trend is online RLHF: instead of scoring with a fixed RM, let the policy produce new samples in deployment, keep labeling with humans or AI, and update the RM and the policy alternately. It's isomorphic to classic online RL (a policy interacting with an environment) — except the "environment" becomes "humans + RM." This is the "from offline alignment to online alignment" route in the LLM alignment case study.

4. The Unavoidable Problem: Reward Overoptimization ​

Gao, Schulman, and Hilton (2022), in Scaling Laws for Reward Model Overoptimization (arXiv:2210.10760), gave RLHF its empirical Goodhart's-law curve: as KL distance grows, the reward-model score first rises, then falls — past a certain point, optimizing the reward model actually degrades true quality. This curve is the sword of Damocles hanging over all alignment work: there's no free reward, and optimizing a proxy reward always has a price. Being able to cite this curve in an interview or a paper is the watershed between "has memorized RLHF" and "understands RLHF."

5. Scalable RL: Compute Is Becoming Algorithmic Gain ​

1. The JAX Ecosystem: RL Algorithms as Pure Functions on a GPU ​

Traditional RL training runs environments on CPUs and networks on GPUs, with bandwidth as the bottleneck. The JAX ecosystem's answer is end-to-end vectorization:

  • Brax (Google, 2021): physics simulation runs directly in JAX/on GPU; a single GPU generates hundreds of thousands of environment steps per second;
  • PureJaxRL (2025, arXiv:2502.19643, Reinforcement Learning at Light Speed): writes the entire PPO training loop as a JAX pure function (environment, policy, and updates in one), using evojax-style parallelism primitives, making "PPO finishes a task in one minute" routine.
text
Traditional RL training loop (CPU/GPU split):
   CPU: environment stepping (slow, hundreds to thousands of steps/sec)
   GPU: policy forward / gradient updates (fast, but starved for data)
   → bottleneck: shuttling data between environment and learner

JAX end-to-end vectorization:
   environment + policy + updates all written as GPU kernels
   → tens of thousands of parallel environment steps + immediate updates; the bandwidth bottleneck disappears

2. The Paradigm Shift Brought by Massive Parallelization ​

  • The number of parallel environments becomes a hyperparameter: thousands of parallel environments plus a handful of update rounds — that's the 2020s redefinition of "sample efficiency";
  • Algorithms as pure functions: CleanRL/PureJaxRL turn reproduction, ablation, and hyperparameter sweeps from "script management" into "function calls," greatly improving research reproducibility;
  • Infrastructure dividend: the cost of the "learning" part of RL keeps falling, and the cost center shifts back to "environments and rewards" — once again confirming "the environment is the product" from Anatomy of an RL System.

An engineering takeaway

If you're doing RL research or building an RL product, "writing a training loop from scratch" is no longer necessary as of 2026 — the JAX ecosystem's ready-made skeletons (Brax, PureJaxRL) are faster and more stable than anything you'd roll yourself. Framework selection is covered in detail in Choosing Frameworks and Tools.

6. RL for Reasoning: RL Teaches Large Models to "Think" ​

1. Background: From "Aligning Preferences" to "Optimizing Verifiable Correctness" ​

RLHF's reward is the reward model's subjective rating; but for tasks like math and code, there exist objectively verifiable rewards (is the answer right, do the tests pass). "Using RL to optimize verifiable rewards" (Reinforcement Learning with Verifiable Rewards, RLVR) became the hottest direction of 2024–2025: no reward-model bias — the reward is hard fact.

2. DeepSeek-R1 (DeepSeek-AI, 2025, arXiv:2501.12948) ​

DeepSeek-R1 is the landmark work of this direction, and its two contributions must be remembered separately:

Contribution one: R1-Zero proves that pure RL alone is enough for reasoning to emerge. No supervised fine-tuning (SFT) at all — just RL straight from the base model — and the model "spontaneously" learns to think longer on math problems and to reflect on itself. The paper's famous "aha moment" refers to exactly this kind of emergence.

Contribution two: GRPO solves the infrastructure problem of large-scale RL. Standard PPO needs a critic network to estimate the value function; at scale the critic is as large as the policy — double the memory, double the instability. GRPO's (Group Relative Policy Optimization) idea:

text
PPO:      advantage ≈ r - V(s)          # requires learning a value network V
GRPO:     for each question, sample a group of responses {y1..yG}
          normalize rewards within the group: Â_i = (r_i - mean(r)) / std(r)
          → no critic needed; the within-group relative advantage serves directly as the advantage

"Relativization" eliminates the value network and simplifies the training infrastructure by an order of magnitude — the key mechanism that let R1 scale RL training within limited compute. Its intellectual lineage runs in the same direction as PPO replacing absolute returns with advantages (see policy gradient).

3. How to Read This Line ​

ClaimReality and Risks
"RL teaches models to reason"True, but strong only on tasks with verifiable rewards (math/code); for open-ended questions, rewards still come from model/LLM judges, degrading back into "RLHF-style subjective alignment"
"Process rewards (PRM) beat outcome rewards"Contested: process rewards provide denser signal but are expensive to label and can encourage shortcut-taking
"Reasoning RL will replace all of RLHF"No: reasoning RL optimizes "correctness," RLHF optimizes "preference" — they solve different problems and commonly coexist in practice

7. General Agents: RL Meets Agents ​

1. Environments as APIs ​

When the "environment" shifts from games to browsers, APIs, and the physical world, the agent paradigm of RL changes with it: an agent's actions are tool calls, web clicks, or shell commands, and rewards come from "was the task completed" (executability, task success rate). The Agents and Dialogue Systems case study works through the MDP-ification of this paradigm in detail.

2. WebArena (Zhou et al., 2023, arXiv:2307.13854) ​

  • What it is: a self-hosted environment of real websites (e-commerce, forums, CMS, etc.) with 1091 tasks, evaluating agents by functional correctness (whether the site reaches the goal state) rather than just "did it click the right thing."
  • Why it matters: it's "the Atari of general agents" — giving agent research a reproducible, evaluable, realism-adjacent testbed. The core problems it exposes are evaluation (how to decide whether a sequence of browser operations "completed the task") and long-horizon credit assignment (success or failure only becomes clear dozens of steps later).

3. VLA (Vision-Language-Action): The RL-ification of Embodied Agents ​

VLA models merge vision, language, and action into a single model (the representative RT-2 family, Brohan et al., 2023, arXiv:2307.15818), transferring "web knowledge" to robotic control. RL's role here is still being explored: the mainstream recipe today is large-scale imitation pretraining first, then RL fine-tuning to push success rates past the bottleneck — structurally isomorphic to AlphaGo's "SL first, RL later" route. Progress and criticism in the Robotics Sim2Real case study.

8. A Critical Perspective: Real Breakthroughs vs. Hype ​

The skill most needed for reading frontier papers isn't "keeping up" — it's telling signal from noise. A usable evaluation framework:

DimensionSigns of a Real BreakthroughSigns of Hype
Mechanismproposes a reusable mechanism (e.g., GRPO dropping the critic, TD-MPC2's latent consistency)reports results but no mechanism — "XX surpasses humans for the first time"
Evaluationmultiple independent tasks / multiple seeds / comparisons against strong baselines / public codea single task, a single seed, comparisons only against weak baselines
Reproducibilityofficial implementation public, ablations completecode not public or "coming soon"
Contextclearly states applicable conditions and failure modesclaims generality, "one algorithm to rule them all"

Apply this framework to the six threads on this page:

  • Real breakthroughs: the OOD handling ideas of IQL/CQL, DPO's closed-form insight, GRPO's infrastructure simplification, DreamerV3's fixed-hyperparameter engineering — all of them left behind transferable mechanisms.
  • Watch this space: general agents (evaluation and credit assignment unsolved), online RLHF (cost and stability unsolved), RL for reasoning on open-ended questions.
  • Most likely hype: any "sweeps all tasks, no tuning needed" headline; any method claiming superiority on a single private benchmark.

A final word of advice

Frontier pages go stale; mechanisms don't. The "frontier" you read about here in 2026 will be old news by 2028 — but mechanisms like "within-group relative advantages eliminate the critic," "value hallucinations on OOD actions," and "the KL curve of reward overoptimization" will become foundations of the next generation of classic papers, just like "experience replay" and "target networks" did. When reading frontier work, always ask one question: what mechanism does this paper leave behind?

Further Reading ​

References ​