REINFORCEMENT LEARNING HANDBOOK
A systematic path from trial-and-error to decision-making — MDPs · Value Learning · Policy Gradients · Actor-Critic · Offline RL · Multi-Agent · RLHF · Career Paths
50+ pages aren't a library to read cover to cover — they're routes you combine as needed
Interviewing within 1–3 months? Go straight to JD benchmarking, resume rewriting, and the interview question bank — every page maps to what RL roles actually test.
See the path →Have time to build a solid foundation? Work from the Guide through Core Concepts, Case Studies, Practice, and Papers week by week, with a checkpoint for each week.
See the path →Practitioners don’t read cover to cover: use the "problem → page" index to look things up on demand — treat it as a reference book.
See the path →If you only read ten pages, read these
These 50+ pages are not a library to read cover to cover but a roadmap — a two-week interview sprint for anyone with interviews in 1–3 months, an ~8-week systematic track for building solid foundations, and a "question → page" desk-reference index for builders already on the job.
GuideReinforcement learning is the paradigm of learning "what to do" through trial and error — no labels, only after-the-fact rewards. This page gives a precise definition, the agent–environment interaction framework, the reward hypothesis, a three-way comparison with supervised and unsupervised learning, and a complete walkthrough of a minimal RL example.
GuideThe unified language of nearly all RL problems — the 5-tuple, the Markov property, discounted return, policies, value functions, and the Bellman equation; plus the three solution routes from a known model to an unknown one.
Core ConceptsLearning "what to do" directly — the intuition behind the policy gradient theorem, REINFORCE and its high-variance problem, baselines and advantage, the motivation for TRPO and PPO's clipped objective, and implementation notes for continuous action spaces.
Core ConceptsThe key technique that makes language models "speak human" — the limits of behavior cloning, reward model training, and the three-stage PPO pipeline; the alignment tax, reward overoptimization, DPO and the simplification of direct preference optimization; how RLHF compares with classic RL.
Core ConceptsThe full arc of value-based learning — policy evaluation and policy improvement, value iteration; Monte Carlo and temporal-difference TD(0); how SARSA and Q-learning differ; the engineering tricks of the DQN family (replay, target networks, Double, Dueling, Rainbow).
Core ConceptsWhy Go "cannot be searched"; the four steps of MCTS and UCB1; how policy and value networks "trade learning for compute"; AlphaZero's pure self-play and generality; and the complementary principles of search and learning.
Case StudiesThe three-stage training of InstructGPT/ChatGPT: SFT -> reward model -> PPO; open-source reproductions (LLaMA-2-chat, Zephyr); the real costs of reward overoptimization and the alignment tax; how DPO simplifies it all; large-model training from an RL engineer's perspective.
Case StudiesImplement three versions of CartPole from scratch with Gymnasium — tabular Q-learning, a simple policy gradient, then PPO. Each version ships with complete code, key pitfalls, and a "definition of done," plus a minimal reproducible experiment template.
Practice GuideBrowse by module, or search directly
Seventy years from the Bellman equation to RLHF: dynamic programming lays the foundations, temporal-difference learning and Q-learning, TD-Gammon, the deep RL trinity (DQN/PPO/AlphaGo), and on to LLM alignment. A decade-by-decade table of the key milestones.
The six layers of a complete RL system — environment, experience collection, learning algorithm, policy, evaluator, and safety guardrails — and why "the environment is the product": 80% of RL project cost sits in the environment.
These 50+ pages are not a library to read cover to cover but a roadmap — a two-week interview sprint for anyone with interviews in 1–3 months, an ~8-week systematic track for building solid foundations, and a "question → page" desk-reference index for builders already on the job.
Boundary analysis of reinforcement learning versus supervised learning, unsupervised learning, optimal control, operations research, planning & search, and behavior cloning — a decision framework for "when to use RL, and when not to."
Reinforcement learning is the paradigm of learning "what to do" through trial and error — no labels, only after-the-fact rewards. This page gives a precise definition, the agent–environment interaction framework, the reward hypothesis, a three-way comparison with supervised and unsupervised learning, and a complete walkthrough of a minimal RL example.
Why RL evaluation is hard — return variance, random seeds, sample efficiency, and compute budgets; a map of the mainstream benchmarks (Gymnasium, Atari, MuJoCo, DM Control, Procgen, Meta-World, Maze); evaluation reporting standards.
RL's first-order tension — from ε-greedy to entropy regularization, parameter-space noise, intrinsic rewards (curiosity), count-based methods, and RND; the concrete forms and frontiers of exploration in deep RL (Go-Explore, E3B).
The unified language of nearly all RL problems — the 5-tuple, the Markov property, discounted return, policies, value functions, and the Bellman equation; plus the three solution routes from a known model to an unknown one.
Learn the environment's dynamics first, then plan "in your head" — MPC, Dyna, Dreamer, World Models, TD-MPC; the trade-off between data efficiency and model error; a model-based view of AlphaGo/AlphaZero.
From single agent to games — non-stationarity, Nash equilibrium, the oscillation of independent learning; the CTDE paradigm, MADDPG, QMIX, and value factorization; MARL in games, traffic, and auctions, plus the hard truth about deployment.
A simplified laboratory for exploration and exploitation — problem formulation and the definition of regret, ε-greedy, the derivation and intuition of UCB1, the Bayesian view of Thompson Sampling, and the bridge from contextual bandits to recommendation and advertising.
Only historical data, no further environment interaction — out-of-distribution (OOD) actions and value overestimation, the conservatism idea, CQL and IQL; the relationship to behavior cloning / imitation learning; deployment realities and "when offline RL is the wrong tool."
Learning "what to do" directly — the intuition behind the policy gradient theorem, REINFORCE and its high-variance problem, baselines and advantage, the motivation for TRPO and PPO's clipped objective, and implementation notes for continuous action spaces.
Rewards are the product spec — reward shaping and potential functions, sparse rewards and curriculum learning, a casebook of reward hacking, inverse RL (IRL) and preference learning; how to write a reward that can't be gamed.
The key technique that makes language models "speak human" — the limits of behavior cloning, reward model training, and the three-stage PPO pipeline; the alignment tax, reward overoptimization, DPO and the simplification of direct preference optimization; how RLHF compares with classic RL.
Where value meets policy — the family tree of A2C/A3C, DDPG, TD3, SAC, and PPO; a unified view of on-policy and off-policy; and why continuous control is SAC's home turf.
The full arc of value-based learning — policy evaluation and policy improvement, value iteration; Monte Carlo and temporal-difference TD(0); how SARSA and Q-learning differ; the engineering tricks of the DQN family (replay, target networks, Double, Dueling, Rainbow).
Dialogue policy learning, tool-use and web-navigation agents, embodied agents — RL fine-tuning that treats the LLM as the policy; the latest progress from ReAct to RL-trained agents (e.g., AgentLM, RFT); the environment is an API.
Why Go "cannot be searched"; the four steps of MCTS and UCB1; how policy and value networks "trade learning for compute"; AlphaZero's pure self-play and generality; and the complementary principles of search and learning.
How DQN beat humans across 49 Atari games — end-to-end learning from pixels to actions, the birth of experience replay and target networks; Rainbow's seven improvements; and the limits and lessons of game RL.
Where RL sits in the perception–prediction–planning–control stack — hierarchical decision-making, mixing imitation learning with RL, and safety constraints (constrained RL, safety guardrails); the Waymo/Tesla technical routes and "why RL hasn't grabbed the wheel yet."
The three-stage training of InstructGPT/ChatGPT: SFT -> reward model -> PPO; open-source reproductions (LLaMA-2-chat, Zephyr); the real costs of reward overoptimization and the alignment tax; how DPO simplifies it all; large-model training from an RL engineer's perspective.
Portfolio optimization, order execution, market making — the allure and the traps of financial RL; non-stationary markets, extremely low signal-to-noise, and poor sample independence; an honest look at why most financial RL papers fail in live trading.
Treat the user as the environment — list-page ranking, real-time bidding, and the exploration–revenue trade-off; public engineering from Taobao/Alimama, ByteDance and others; the evolution from bandits to full RL, and the ever-present offline evaluation dilemma.
Inventory management, bin packing, machine scheduling, network routing: writing combinatorial optimization as MDPs; learned solvers (L2S, GPN); comparison and complementarity with classical OR (MILP, heuristics).
AlphaFold's end-to-end structure prediction and geometric constraints, molecular design and drug discovery with RL (GFlowNets, RL-based molecule generation), experimental design as active learning, and the mathematical-reasoning RL of AlphaProof/AlphaGeometry — the big picture of RL for Science.
Real-world physical trial-and-error is too expensive, so learning in simulation has become the mainstream — the simulator ecosystem (MuJoCo, Isaac), domain randomization, the Sim2Real gap and how to close it; case studies from OpenAI's dexterous hand and the ANYmal quadruped to robotic grasping.
Close, paper-by-paper readings of the classics that changed RL — Bellman's dynamic programming, Watkins's Q-learning, Mnih's DQN, Schulman's PPO, Silver's AlphaGo, and Ouyang's InstructGPT — each with background, method, experiments, limitations, and "how to answer in an interview."
The RL frontier of the 2020s — world models (DreamerV3, TD-MPC2), offline RL (IQL/CQL), the RLHF and DPO family, scalable RL (massive parallelization, the JAX ecosystem), RL for LLM reasoning (R1, RLVR), and general agents — with a read on where the trends point.
A map of seventy years of key RL papers, organized by theme and timeline — the dynamic programming branch, the classic tabular branch, the policy gradient branch, the deep RL branch, the RLHF branch, and the multi-agent branch — with a one-line positioning for every paper.
A methodology for reading RL papers: how to take notes (the three-column method), what to do when the math won't yield, how to judge whether a paper is any good, which papers you can skip, and the discipline of reproducing experiments.
Goal-driven roadmaps for reading RL papers — concrete reading lists and orderings for the two-hour quick start, the engineering track, and the research track, plus how deeply to read each paper.
Why RL learners should read papers — understand principles instead of memorizing APIs, keep up with the frontier instead of chasing hype, and ace interviews; the three reading routes; a map of the section's five pages; "take away mechanisms, not numbers."
Evaluation protocols in practice — fixed seed matrices, multiple runs, learning curves with IQR, the twin dimensions of sample efficiency vs. final performance; fairness pitfalls in algorithm comparisons; offline and online evaluation.
The complete pipeline of an RL project, from problem definition to production — choosing an environment (build vs. off-the-shelf), baselines first, the training loop, evaluation protocols, logging and reproducibility, deployment and monitoring — at the "make it run first, tune later" rhythm.
Reward hacking, seed overfitting, environment bugs, evaluation cheating, sample-efficiency illusions, missing logs, and the engineering traps of large-scale parallelism — the ten most common ways RL projects go wrong, and how to detect them.
Stable-Baselines3, Ray RLlib, Tianshou, CleanRL, the JAX stack (Brax, PureJaxRL), and rolling your own — a comparison table of six options for different scenarios, plus the battle-tested engineering advice "copy CleanRL first, then modify."
Types of RL projects worth showing in a job search — paper reproductions, custom environments, real business problems; how to write a README covering the four essentials of environment, algorithm, evaluation, and reproducibility; common mistakes.
Implement three versions of CartPole from scratch with Gymnasium — tabular Q-learning, a simple policy gradient, then PPO. Each version ships with complete code, key pitfalls, and a "definition of done," plus a minimal reproducible experiment template.
Ten principles grown out of failed projects — reward before algorithm, environment first, simplicity first, baselines first, reproducibility, safety guardrails, sample-efficiency awareness, iteration cadence, documenting failures, business alignment.
RL is brutally sensitive to hyperparameters — learning rate, γ, GAE λ, entropy coefficient, clip range, network width; the right order to tune from defaults; AutoRL and batch experiments; and a "diagnose before you tune" methodology.
Maps JD skill terms onto a four-quadrant self-assessment — fluent / half-fluent / can't / not needed — across RL theory, deep learning & coding, engineering & simulation, and business & math; generates a personal catch-up list linked to pages on this site.
The standard formula for RL resumes — the four elements of environment, algorithm, metrics, and reproducibility; STAR-ified project descriptions; common mistakes (algorithm name-dropping, no evaluation protocol); and before/after rewrite examples.
Worked answers to high-frequency RL interview questions — MDP and Bellman, MC vs TD, Q-learning vs SARSA, DQN engineering tricks, PPO details, SAC and continuous control, exploration vs exploitation, RLHF, reward-design scenario questions — plus an answer framework and follow-up predictions.
Breakdowns of live RL job descriptions in China and abroad — research scientist, RL algorithm engineer, LLM alignment, robotics learning, and more; word-frequency stats of high-frequency requirements and skill-mix profiles; includes dataAsOf.
A panoramic view of the RL job market — the responsibilities, core skills, and pay comparisons for roles like research scientist, RL engineer, LLM alignment engineer, and simulation/robotics engineer; a three-dimension decision checklist; and a usage map for the career module's four pages.
Textbooks (Sutton & Barto), courses (DeepMind/Berkeley/UCL), tutorials (Spinning Up, CleanRL), blogs (Lilian Weng), podcasts and communities — a genuinely real resource list organized by learning stage.
Environment libraries (Gymnasium, MuJoCo, Brax, Isaac Gym/Lab), benchmark suites (Atari, Procgen, DM Control, Meta-World, Maze), datasets (D4RL, RL Unplugged), and the training toolchain (W&B, TensorBoard, Hydra, Optuna) — each profiled in one place.
Quick reference for RL jargon — MDP, Bellman equation, TD, MC, GAE, PPO, SAC, DQN, OOD, CTDE, RLHF, reward hacking, and 60+ more entries, each with a one-line definition plus a link to the page that covers it in depth.
The math behind RL — expectation and conditional expectation, Markov chains and stationary distributions, the dynamic programming principle, stochastic approximation and Robbins–Monro, convex optimization and KL divergence — every concept mapped to a concrete use in RL.
