PPO

PPO is, by a wide margin, the most-deployed on-policy deep RL algorithm of the last decade: continuous-control robotics, Dota 2 and StarCraft II agents, and — in a reincarnation few expected in 2017 — the reinforcement-learning-from-human-feedback (RLHF) stage of nearly every modern instruction-tuned language model. All of that success traces back to one design choice: replace TRPO's hard KL constraint with a soft, differentiable surrogate that a plain Adam optimizer can climb.

Two variants appear in the literature. PPO-Penalty keeps an unconstrained KL term with an adaptively tuned coefficient \(\beta\) — closer in spirit to a Lagrangian relaxation of TRPO's constraint. PPO-Clip, the version that won out in practice and the subject of this chapter, uses no KL term at all: it clips the probability ratio directly inside the objective. Unless stated otherwise, "PPO" in this course means PPO-Clip.

Recall Chapter 07's surrogate, defined relative to a fixed data-collecting policy \(\pi_{\theta_{\mathrm{old}}}\):

"CPI" stands for conservative policy iteration, the Kakade–Langford result that justifies this linearization. Left unconstrained, maximizing \(L^{\mathrm{CPI}}\) drives \(\rho_t\to\infty\) wherever \(\hat A_t>0\) and \(\rho_t\to 0\) wherever \(\hat A_t<0\) — an excessively large policy update that TRPO prevented with a hard trust-region constraint on mean KL, solved via natural gradients, conjugate gradients, and a backtracking line search.

That machinery is expensive: every update needs several Hessian-vector products (each a full backward pass through the KL Hessian), is awkward to combine with architectures that share parameters between policy and value heads, and does not trivially support minibatched, multi-epoch training the way ordinary deep-learning optimizers do. PPO's question: can a term inside the objective itself do the job the constraint was doing?

Two implementation details matter enough to state explicitly.

• Log-space computation. Networks output \(\log\pi_\theta(a_t\mid s_t)\) directly (log-softmax for discrete actions, Gaussian log-density for continuous ones). The ratio is formed as \(\rho_t=\exp\big(\log\pi_\theta(a_t\mid s_t)-\log\pi_{\theta_{\mathrm{old}}}(a_t\mid s_t)\big)\), which is numerically stable even when the raw probabilities are tiny.
• \(\log\pi_{\theta_{\mathrm{old}}}\) is a stored constant. It is computed once, at rollout time, and cached alongside \(s_t,a_t,\hat A_t\) in the batch. During the \(K\) epochs of gradient updates that follow, only \(\log\pi_\theta\) changes; \(\theta_{\mathrm{old}}\) never does, until the next rollout phase begins.

For continuous control, \(\pi_\theta(\cdot\mid s)\) is usually a diagonal Gaussian \(\mathcal N(\mu_\theta(s), \Sigma_\theta(s))\), and the ratio is the ratio of joint densities across action dimensions — a single scalar per timestep, exactly as in the discrete case.

PPO-Clip replaces (1) with the minimum of two candidate terms: the unclipped CPI term, and a version where the ratio has first been clamped to an interval around 1.

\(L^{\mathrm{CLIP}}\) is a pointwise minimum of two terms, hence \(L^{\mathrm{CLIP}}(\theta)\le L^{\mathrm{CPI}}(\theta)\) for every \(\theta\) and every sample: it is a pessimistic (lower) bound on the unclipped surrogate, in the same spirit as TRPO's improvement lower bound (Theorem 3.1 of Chapter 07). The two cases of \(\hat A_t\)'s sign behave very differently and are worth working through separately.

Say the clip binds on sample \(t\) if \(L^{\mathrm{CLIP}}\)'s value at \(t\) comes from the clipped branch rather than the raw \(\rho_t\hat A_t\) branch — equivalently, if \(\operatorname{clip}(\rho_t,1-\varepsilon,1+\varepsilon)\hat A_t < \rho_t\hat A_t\). Combining the two cases above:

The clip fraction — the empirical fraction of a minibatch on which (2) holds — is one of the most useful scalars to log during PPO training. It is reported by essentially every serious implementation (OpenAI Baselines, Stable Baselines3, RLlib, CleanRL) precisely because it is a cheap, direct readout of how hard the trust region is working.

• Clip fraction ≈ 0. Ratios never leave the band; the clip is inactive and PPO is behaving like unclipped CPI with a well-behaved step size — possibly the step size (or number of epochs) could be increased for faster learning.
• Clip fraction moderate (5–30%). Typical healthy PPO training; the clip is doing real work on a meaningful slice of the batch each epoch.
• Clip fraction very high (>50%). Most of the batch has already drifted outside the trust band; further epochs on this batch mostly hit the flat region and contribute little gradient signal — a sign to reduce epochs, reduce learning rate, or increase \(\varepsilon\) with caution.

Deep RL implementations of PPO almost always share a single network trunk between the policy head \(\pi_\theta\) and a value head \(V_\phi\) (often \(\phi\subset\theta\)), trained jointly. The full loss combines three terms.

maximized by gradient ascent (equivalently, minimize \(-L^{\mathrm{PPO}}\) with Adam). Typical coefficients: \(c_1\approx 0.5\), \(c_2\in[0, 0.01]\) — the entropy bonus is frequently annealed to zero or omitted entirely in RLHF settings where a KL-to-reference-policy penalty already discourages collapse (foreshadowing Chapter 09).

This is PPO's other major practical win over TRPO. Because \(L^{\mathrm{CLIP}}\) is an ordinary differentiable function of \(\theta\) — no constrained optimization required — nothing stops you from taking multiple SGD/Adam steps on the same rollout batch before collecting new data, the way a standard supervised-learning epoch loop would.

Typical \(K\) is \(3\)–\(10\); minibatch size divides the full batch \(N\times T\) into several chunks per epoch. Reusing data \(K\) times is exactly what the clip is protecting: without it, \(K\) full-strength CPI gradient steps on fixed data would very plausibly push some \(\rho_t\) far outside any reasonable trust region well before the epoch loop finishes, causing the next rollout to be collected under a badly-degraded policy.

The clip does not make this perfectly safe — flat gradient on a sample stops that sample's policy term from pushing further, but other samples in the same minibatch (via shared parameters) and the value/entropy terms keep updating \(\theta\), so \(\rho_t\) for a previously-unclipped sample can drift into the clipped region as training proceeds through the epochs. This is why clip fraction and (optionally) an approximate mean KL are tracked during the epoch loop, and why some implementations add an early-stopping rule:

PPO says nothing new about how \(\hat A_t\) is computed — it inherits Generalized Advantage Estimation from Chapter 06 wholesale:

\(\lambda=0\) recovers the one-step TD advantage (low variance, high bias from an imperfect \(V_\phi\)); \(\lambda=1\) recovers the Monte-Carlo advantage (unbiased, high variance). PPO papers and popular baselines default to \(\gamma\approx 0.99\), \(\lambda\approx 0.95\). Advantage batch normalization — subtract the batch mean, divide by the batch standard deviation, right before forming \(\rho_t\hat A_t\) — is close to universal in practice; it keeps the scale of \(\hat A_t\) comparable across iterations regardless of raw reward scale, which interacts directly with how meaningful a fixed \(\varepsilon\) is.

Contrast the line count and dependency footprint with Chapter 07's algorithm box: no Hessian-vector products, no CG solve, no line search — every line is a standard forward/backward pass or an elementwise tensor operation, which is exactly why PPO is the default "just works" on-policy baseline in virtually every RL library.

PPO trades TRPO's one delicate hyperparameter (\(\delta\), the KL budget) for a slightly longer list of milder ones. Representative ranges, drawn from the original paper and widely-used baselines:

• Clip range \(\varepsilon\). \(0.1\)–\(0.3\); \(0.2\) is the most common default. Often linearly annealed to \(0\) over training.
• Epochs \(K\). \(3\)–\(10\) for continuous control; smaller (even \(1\)–\(4\)) for large LLM policies where each epoch is expensive and off-policy drift is a bigger concern.
• Minibatch size. Divides the full rollout batch \(N\!\times\!T\); e.g. batch \(2048\)–\(32000\), minibatch \(64\)–\(4096\).
• Learning rate \(\alpha\). \(1\text{e-}4\)–\(3\text{e-}4\) (Adam), frequently linearly annealed to \(0\) by the end of training.
• Discount \(\gamma\), GAE \(\lambda\). \(\gamma\approx 0.99\), \(\lambda\approx 0.95\) — task-dependent; longer-horizon tasks want \(\gamma\) closer to \(1\).
• Value / entropy coefficients \(c_1,c_2\). \(c_1\approx 0.5\); \(c_2\) from \(0\) up to \(\sim 0.01\).
• Gradient norm clipping. Global norm clip around \(0.5\), a second, orthogonal stabilizer on top of the ratio clip.
• Rollout length \(T\) / parallel actors \(N\). \(T\) from tens to a few thousand steps; \(N\) from \(1\) to thousands of vectorized environments, trading wall-clock time for batch size.

A side-by-side of the two chapters' machinery:

• Trust-region mechanism. TRPO: hard KL constraint solved per-iteration via CG + line search. PPO: soft, differentiable ratio clip inside an ordinary loss.
• Per-update cost. TRPO needs several Hessian-vector products (extra backward passes) plus a line search with repeated KL/objective re-evaluation. PPO needs one forward/backward pass per minibatch, identical in shape to supervised training.
• Data reuse. TRPO's guarantee is tied to a single natural-gradient step per batch; PPO is explicitly designed for \(K\) epochs of reuse per batch.
• Guarantees. TRPO inherits a (idealized) monotonic-improvement argument from the KL trust region. PPO's clip is a heuristic surrogate — it has no equivalent monotonic-improvement proof, only the empirical observation that it behaves like a trust region in practice.
• Implementation complexity. TRPO requires Fisher-vector product code, CG, and line search machinery layered on top of autodiff. PPO is roughly 20–30 extra lines over vanilla actor-critic.
• Empirical performance. Comparable-to-better across the MuJoCo/Atari benchmarks in the original PPO paper, at a fraction of the compute — the reason PPO, not TRPO, became the field's default.

• \(\varepsilon\) too large. The trust band is wide enough that the clip rarely binds — PPO degenerates toward unconstrained CPI, reintroducing TRPO's original instability (large, destructive updates from stale data).
• \(\varepsilon\) too small, or \(K\) too small. Learning is needlessly slow: the policy is barely allowed to move per batch, wasting the expensive rollout data.
• Too many epochs \(K\) for a given \(\varepsilon\), \(\alpha\). Clip fraction creeps toward \(1\) partway through the epoch loop; remaining gradient steps are mostly wasted compute on flat-gradient samples, or — worse — parameters drift via the value/entropy terms while the clip masks the policy term, producing a policy that has moved further than the diagnostics suggest.
• Shared network, mis-tuned \(c_1\). If the value loss dominates the shared trunk's gradient, the policy head is starved of signal; conversely a too-small \(c_1\) leaves \(V_\phi\) poorly fit, which corrupts \(\hat A_t\) via GAE and indirectly destabilizes the "stable" clipped objective.
• Reward / advantage scale sensitivity. \(\varepsilon\) is a constraint on the ratio, not on \(\hat A_t\); without advantage normalization, a rescaled reward function changes how much a fixed clip band actually restricts learning, even though nothing "physical" about the task changed.
• Seed / implementation sensitivity. Henderson et al. (2018) and follow-up reproducibility studies documented that "PPO" results vary substantially across codebases due to under-documented implementation details (advantage normalization, reward scaling, orthogonal initialization, learning-rate annealing) that are not part of the equations above but materially affect outcomes — a caution for treating any single PPO result as definitive.
• Off-policy drift beyond one rollout. PPO is on-policy: reusing a batch across \(K\) epochs is already a mild controlled violation of that assumption. Attempting to reuse data across multiple rollout iterations (as in a replay buffer) breaks the \(\rho_t(\theta_{\mathrm{old}})=1\) anchoring that the clip's geometry relies on.

PPO's remaining expensive ingredient is the learned critic \(V_\phi(s)\): a second network, trained jointly, needed only to produce a low-variance baseline for \(\hat A_t\). For large language model policies — where a single forward pass is already costly and the "state" is an entire token prefix — training and serving a full value network purely for variance reduction is a significant overhead, and the value network is notoriously hard to fit well early in RLHF training.

Group Relative Policy Optimization (GRPO; DeepSeekMath, 2024) keeps PPO's ratio and clip exactly as derived in this chapter — Definition 4.1 is unchanged — but replaces the critic-based advantage with a group-relative one: sample a group of \(G\) completions for the same prompt, score each with a reward model (or verifiable reward), and set

broadcast to every token of completion \(i\) (or combined with a per-token or per-step variant). No value network, no GAE, no \(V_\phi\) loss term — the "group" itself supplies the baseline that \(V_\phi(s)\) supplied in PPO. Everything else — the log-ratio, the clip, the min, multi-epoch reuse, KL monitoring (often against a frozen reference policy rather than \(\pi_{\theta_{\mathrm{old}}}\)) — carries over directly. Chapter 09 derives (5) from first principles and works through why it is an unbiased, if higher-variance, drop-in replacement for a learned critic in the specific regime of sampling many completions per prompt.

Clip binds, \(\hat A>0\). Let \(\pi_{\mathrm{old}}(a|s)=0.20\), \(\pi_\theta(a|s)=0.30\), \(\hat A=1.5\), \(\varepsilon=0.2\). Then \(\rho=0.30/0.20=1.5\). Unclipped: \(\rho\hat A = 1.5\cdot1.5=2.25\). \(\operatorname{clip}(1.5,0.8,1.2)=1.2\), so clipped term \(=1.2\cdot1.5=1.8\). \(L^{\mathrm{CLIP}}=\min(2.25,1.8)=1.8\) — the clip binds (\(\rho=1.5>1+\varepsilon=1.2\)), and the gradient of this sample's contribution with respect to \(\rho\) (hence \(\theta\)) is zero at this point.

Clip inactive, \(\hat A<0\). Same policies, \(\hat A=-1.5\). Unclipped: \(1.5\cdot(-1.5)=-2.25\). Clipped: \(1.2\cdot(-1.5)=-1.8\). \(L^{\mathrm{CLIP}}=\min(-2.25,-1.8)=-2.25\) — the unclipped branch wins, because for \(\hat A<0\) the clip only ever binds when \(\rho<1-\varepsilon\), and here \(\rho=1.5\) is on the "no ceiling" side (Section 4's second case). The full gradient still flows — this update is being penalized as strongly as CPI would, exactly as intended.

Boundary check. With \(\varepsilon=0.2\), the flat value for \(\hat A>0\) is \((1+\varepsilon)\hat A=1.2\hat A\); for \(\hat A<0\) it is \((1-\varepsilon)\hat A=0.8\hat A\). At exactly \(\rho=1\pm\varepsilon\) the two branches meet by continuity of \(\operatorname{clip}\): \(L^{\mathrm{CLIP}}\) is continuous in \(\rho\) everywhere, though not differentiable exactly at the two kink points \(\rho=1-\varepsilon\) and \(\rho=1+\varepsilon\).

GRPO keeps every piece of PPO's clipped surrogate derived in this chapter — the ratio, the clip, the min, the multi-epoch loop — and removes only the learned critic, replacing \(\hat A_t\) with a group-relative, reward-model-based baseline suited to sampling many completions per prompt. That substitution is the algorithmic core of modern LLM RLHF and reasoning-model post-training, and is where this course's policy-optimization arc concludes.