09 · GRPO — Pearl

1. Overview

Chapter 08 built PPO around three ingredients: a policy \(\pi_\theta\), a learned critic \(V_\phi\) supplying advantages via GAE, and a clipped surrogate that keeps the policy near \(\pi_{\theta_{\mathrm{old}}}\). GRPO keeps the third ingredient almost unchanged and throws away the second. In its place: sample many completions per prompt and let the group itself estimate how good "average" is.

This is not a generic RL trick — it is specifically shaped by how large language models are post-trained. A prompt \(q\) is a state visited essentially once; a full completion \(o\) is, from the reward model's point of view, a single action. Fitting a separate neural critic to predict the value of a state you may never revisit is expensive and often inaccurate. Resampling the same state many times and averaging, on the other hand, is cheap: it costs a few extra forward passes through the policy you already have.

2. Motivation: the critic is the expensive part

In PPO for LLM RLHF, \(V_\phi\) is usually a second copy of the policy network (same transformer backbone, different head), trained jointly with \(\pi_\theta\). This has three costs that matter enormously at scale:

Memory and compute. A 70B-parameter critic roughly doubles activation memory and optimizer state versus policy-only training, and doubles the forward/backward passes per step.
Credit assignment is genuinely hard. Reward is usually given only at the end of a generated sequence (a single scalar for the whole response). \(V_\phi\) must learn to predict this terminal reward from every intermediate token — a sparse, high-variance regression target that is easy to overfit or underfit.
Bias from an imperfect critic propagates everywhere. GAE advantages are only as good as \(V_\phi\); a poorly calibrated critic silently corrupts every policy update, and diagnosing this in a multi-billion parameter system is hard.

GRPO's bet: for the "one action per prompt" structure of RLHF/RLVR fine-tuning, a Monte Carlo estimate from a handful of resampled completions is a good enough (and much cheaper, much more transparent) substitute for a learned baseline.

Remark · Not a general MDP claim

This trade only makes sense because the "episode" is short relative to how many times you can afford to resample the same state. It would be a poor fit for, say, a long-horizon robotics MDP where every state is visited once and resampling is physically expensive. Keep the LLM-specific setup in mind as you read the rest of this chapter.

3. Problem setup

Fix a prompt (context) \(q\), drawn from a prompt distribution. The current policy \(\pi_{\theta_{\mathrm{old}}}\) is frozen for a moment and used to draw a group of \(G\) independent completions:

(1) \[ o_1, o_2, \ldots, o_G \;\sim\; \pi_{\theta_{\mathrm{old}}}(\,\cdot \mid q\,). \]

Each completion is scored by a reward model or a rule-based verifier (exact-match checker for math/code — "RLVR"), producing one scalar per sample:

(2) \[ R_i \;=\; r(q, o_i), \qquad i = 1,\ldots,G. \]

That is the entire data-collection step. No bootstrapped targets, no discounted returns along a trajectory of states — one prompt, \(G\) parallel "rollouts," \(G\) scalar rewards. The laboratory below makes this concrete with a literal 4-armed bandit: \(q\) is fixed, the four completions are labeled A/B/C/D, and \(r(q,\cdot)\) is a fixed lookup table.

Definition 3.1 · Group

A group is the set \(\{(o_i, R_i)\}_{i=1}^G\) of completions and rewards sampled for a single prompt \(q\) under a single frozen \(\pi_{\theta_{\mathrm{old}}}\). All group statistics (mean, std) below are computed within this set — never pooled across different prompts.

4. The group-relative advantage

GRPO's central move is to standardize the rewards within the group instead of subtracting a learned baseline:

(3) \[ \hat A_i \;=\; \frac{R_i - \operatorname{mean}(R_1,\ldots,R_G)} {\operatorname{std}(R_1,\ldots,R_G) + \varepsilon}, \qquad \varepsilon \approx 10^{-8}. \]

Read this the same way you read any advantage: \(\hat A_i > 0\) means "better than what this prompt typically produces," \(\hat A_i < 0\) means "worse." Dividing by the group standard deviation additionally makes \(\hat A_i\) dimensionless and roughly unit-scale — it plays the same variance-control role as advantage normalization in PPO (Chapter 08), except the normalizing statistics are recomputed per prompt rather than pooled over a whole batch of unrelated states.

Analysis · A baseline you get for free

Subtracting \(\operatorname{mean}(R)\) is exactly a baseline subtraction — it does not change the expected policy gradient (Chapter 06's baseline lemma still applies) but reduces variance whenever completions within a group are correlated through the shared prompt. The baseline here is a sample mean over \(G\) draws rather than a learned function \(V_\phi(q)\); no gradient descent on a second network is required to obtain it.

Property 4.1 · Zero-signal groups

If every \(R_i\) in a group is equal, then \(\operatorname{std}(R)=0\) and \(\hat A_i = 0/\varepsilon \approx 0\) for every \(i\). The clipped surrogate (Section 6) then contributes exactly zero gradient for that prompt — the update is a no-op regardless of how the raw rewards compare to other prompts. Try this directly in the lab: sample a group after \(\pi\) has already collapsed onto one action.

5. Why this replaces \(V_\phi\), precisely

In the bandit view of a single-turn prompt, there are no intermediate states: \(q\) is the only state, \(o\) is the only action, and the "episode" ends after one action. The value function the critic would have learned is exactly

(4) \[ V^{\pi}(q) \;=\; \mathbb{E}_{o \sim \pi(\cdot\mid q)}\big[r(q,o)\big]. \]

The group mean is a Monte Carlo estimate of exactly this quantity:

(5) \[ \operatorname{mean}(R_1,\ldots,R_G) \;=\; \frac1G\sum_{i=1}^G R_i \;\xrightarrow[G\to\infty]{}\; V^{\pi_{\theta_{\mathrm{old}}}}(q). \]

So the group mean is a value estimate — just an unbiased, tabular, zero-parameter one, valid for exactly this \(q\), refreshed every time a new group is drawn. \(A(q,o) = R - V^{\pi}(q)\) is the textbook advantage; GRPO estimates it by Monte Carlo instead of by regression, and additionally rescales by the group std for numerical stability. No function approximator, no bootstrapping, no separate optimizer — the "critic" is a running mean over the current batch of samples.

Trade-off, stated plainly

This estimate is unbiased but high-variance for small \(G\) (typical values in practice: \(G \in [4, 64]\)), whereas a well-trained \(V_\phi\) can in principle have lower variance by pooling information across prompts. GRPO accepts more variance per update in exchange for removing an entire network, its training dynamics, and its failure modes.

6. Clipped surrogate + KL to a frozen reference

With \(\hat A_i\) in hand, GRPO reuses PPO's importance ratio and clip verbatim. For a full sequence, the ratio is a per-token product; token index \(t\) of completion \(i\) contributes

(6) \[ r_{i,t}(\theta) \;=\; \frac{\pi_\theta(o_{i,t}\mid q, o_{i,<t})} {\pi_{\theta_{\mathrm{old}}}(o_{i,t}\mid q, o_{i,<t})}. \]

Crucially, the same sequence-level advantage \(\hat A_i\) is broadcast to every token of completion \(o_i\) — there is no per-token value baseline, so every token in a good completion is credited equally. The clipped, per-token objective, averaged over the group and over tokens, is

(7) \[ \mathcal{J}_{\mathrm{clip}}(\theta) = \mathbb{E}_{q,\,\{o_i\}} \left[ \frac1G\sum_{i=1}^{G}\frac{1}{|o_i|}\sum_{t=1}^{|o_i|} \min\!\Big( r_{i,t}(\theta)\,\hat A_i,\; \operatorname{clip}\!\big(r_{i,t}(\theta),\,1-\varepsilon,\,1+\varepsilon\big)\,\hat A_i \Big) \right]. \]

GRPO's second departure from RLHF-style PPO: the KL penalty against a frozen reference policy \(\pi_{\mathrm{ref}}\) (typically the SFT checkpoint, or \(\pi_{\theta_{\mathrm{old}}}\) at the start of training) is added directly into the loss, not subtracted from the reward before computing advantages. DeepSeekMath uses an unbiased, always-nonnegative per-token estimator (the "k3" estimator) instead of the naive \(\log(\pi_\theta/\pi_{\mathrm{ref}})\):

(8) \[ \widehat{D}_{\mathrm{KL}}\big[\pi_\theta \,\|\, \pi_{\mathrm{ref}}\big] \;=\; \frac{\pi_{\mathrm{ref}}(o_{i,t}\mid\cdot)}{\pi_\theta(o_{i,t}\mid\cdot)} \;-\; \log\frac{\pi_{\mathrm{ref}}(o_{i,t}\mid\cdot)}{\pi_\theta(o_{i,t}\mid\cdot)} \;-\;1 \;\ge\;0. \]

The full GRPO objective is then

(9) \[ \mathcal{J}_{\mathrm{GRPO}}(\theta) \;=\; \mathcal{J}_{\mathrm{clip}}(\theta) \;-\; \beta\,\mathbb{E}\big[\widehat{D}_{\mathrm{KL}}[\pi_\theta\|\pi_{\mathrm{ref}}]\big]. \]

Remark · Two different reference points, on purpose

\(\pi_{\theta_{\mathrm{old}}}\) (the sampling/ratio policy) drifts every optimizer step or every few steps; \(\pi_{\mathrm{ref}}\) (the KL anchor) is fixed for the whole run, or updated much more slowly. The ratio clip controls local step size; the KL-to-reference term controls global drift from the model's original behavior (style, safety, calibration). The laboratory keeps both knobs — \(\varepsilon\) and \(\beta\) — separately adjustable.

7. Algorithm

Initialize π_θ from an SFT checkpoint; set π_ref ← π_θ (frozen)
For iteration k = 1, 2, …:
  θ_old ← θ                                  (freeze sampling policy)
  Sample a batch of prompts {q}
  For each prompt q:
    Sample group o_1..o_G ~ π_θ_old(·|q)
    Score R_i = r(q, o_i)                    (reward model / verifier)
    Compute Â_i = (R_i − mean(R)) / (std(R) + ε)
  For several inner epochs over the collected groups:
    Compute ratios r_{i,t}(θ) = π_θ / π_θ_old  (per token)
    L = mean over groups & tokens of min(r·Â, clip(r,1±ε)·Â)
    L ← L − β · D̂_KL[π_θ ‖ π_ref]
    θ ← θ + α ∇_θ L
  (no critic to fit — nothing else to update)

Compare this to TRPO/PPO's algorithm boxes (Chapters 07–08): the entire "fit \(V_\phi\) by regression" line is gone. Everything else — freeze \(\theta_{\mathrm{old}}\), compute ratios, clip, take several inner epochs — is structurally identical.

8. Comparison: PPO / RLOO / DPO

GRPO sits in a family of "how do we get an advantage signal for LLM RL without a full critic" answers. Three useful reference points:

PPO (RLHF-style)

Learned critic \(V_\phi\), GAE(\(\gamma,\lambda\)) advantages, KL penalty usually folded into the per-token reward. On-policy, needs a second network, most accurate credit assignment when \(V_\phi\) is well fit.

Â from regression · extra network · per-token reward-KL

RLOO

REINFORCE Leave-One-Out: baseline for sample \(i\) is the mean of the other \(G{-}1\) samples' rewards, \(b_i=\frac1{G-1}\sum_{j\ne i}R_j\). No clipping in the original formulation, no importance ratio (pure REINFORCE), no critic. Very close in spirit to GRPO but skips both standardization by std and the PPO-style clip.

Â = R_i − b_i · leave-one-out · no clip, no critic

DPO

Offline: no sampling loop, no reward model at training time. Uses fixed preference pairs (chosen \(\succ\) rejected) and a closed-form loss derived from the KL-regularized RL optimum via Bradley–Terry, collapsing RL into a classification-style objective on log-probability ratios.

no rollouts · no reward model at train time · offline pairs

Analysis · Where GRPO sits

GRPO is "RLOO plus PPO's clip plus explicit-in-loss KL," or equivalently "PPO with the critic replaced by group statistics." It stays on-policy and keeps a trust-region-flavored clip that RLOO lacks, while remaining far cheaper than PPO because it never trains \(V_\phi\). DPO is a different branch entirely — no online sampling at all — and is usually a separate, earlier or alternative stage rather than a drop-in replacement for GRPO.

9. Failure modes

Deleting the critic removes its failure modes but introduces new ones, most of them traceable directly to Equation (3):

Degenerate / zero-variance groups. As shown in Property 4.1, a group with identical rewards produces \(\hat A_i=0\) for everyone. This is common late in training, once \(\pi\) has nearly collapsed onto one high-reward completion per prompt — exactly when you might still want gradient signal to sharpen further or to escape a locally-good-but-suboptimal answer.
Length bias. Summing (or averaging) a per-token advantage over sequences of very different lengths interacts badly with how rewards are scored. If longer completions are systematically scored higher (or lower) by the reward model, the group-relative advantage inherits that bias, and the policy learns to change length rather than change quality — a widely reported GRPO/PPO-for-LLM pathology.
Reward hacking, amplified by small \(G\). With no critic to smooth over reward-model noise, a lucky high score for a low-quality completion in a small group can produce a large \(\hat A_i\) and a correspondingly large gradient. Larger \(G\) reduces this variance but costs proportionally more inference compute per prompt.
Std collapse / division sensitivity. When \(\operatorname{std}(R)\) is small but nonzero, dividing by \(\operatorname{std}(R)+\varepsilon\) can amplify noise into large, unreliable \(\hat A_i\) values — the opposite failure from the exact-zero case, equally worth watching for.

Practical mitigation

Common fixes in the literature: keep \(G\) reasonably large (16–64), monitor per-prompt reward variance and mask/down-weight zero-variance groups, decouple length from reward via explicit length penalties or length-conditioned normalization, and keep \(\beta\) large enough that \(\pi_\theta\) cannot drift far enough from \(\pi_{\mathrm{ref}}\) to fully exploit reward-model quirks.

10. Classic RL view: contextual bandit and token MDP, at once

GRPO is easiest to derive by squinting at LLM generation as a contextual bandit: state = prompt \(q\), action = whole completion \(o\), one-step episode, reward \(r(q,o)\). Every equation in Sections 3–5 is exactly the bandit-with-baseline story from Chapter 06, with the baseline computed by resampling instead of by regression.

But the completion is not really atomic — it is generated one token at a time, so the underlying process is a token-level MDP: state = (prompt, tokens so far), action = next token, and reward is typically zero everywhere except a single terminal bonus at end-of-sequence. GRPO's actual implementation (Equations 6–7) lives in this token MDP — the importance ratio and clip are computed per token, exactly as in PPO — but the advantage it plugs in is the bandit-level, whole-completion \(\hat A_i\), broadcast unchanged across every token of that completion.

This chapter's laboratory deliberately uses the pure bandit view (one action per sample, no token structure) so that every quantity — the sample, its reward, its advantage, its ratio — is visible on screen at once. Real GRPO is this exact same arithmetic, repeated once per token with a shared \(\hat A_i\).

Analysis · Why broadcasting Â works reasonably well

Broadcasting one sequence-level advantage to every token is a strong assumption — it says every token contributed equally to the eventual reward, which is rarely literally true. It is the same simplification classic REINFORCE makes when it uses the total return \(G_0\) as the learning signal at every timestep before any baseline or GAE is introduced (Chapter 06). GRPO essentially re-derives "REINFORCE with a resampled baseline," now for token sequences.

11. Laboratory · group-relative advantage on a 4-way bandit

One fixed "prompt," four "answers" \(\{A,B,C,D\}\) with a hidden-but-fixed reward table \(R = [0.2,\,0.9,\,0.4,\,0.1]\) — B is the best answer. Sample a group of \(G\) completions from the current softmax policy \(\pi_\theta\), watch the group mean/std and per-sample \(\hat A_i\), then take a GRPO step: a PPO-style clipped update using \(\hat A_i\) in place of a GAE advantage, plus a KL penalty pulling \(\pi_\theta\) back toward the frozen reference policy captured at reset. Push \(G\) down and \(\pi\) toward greedy to reproduce the zero-variance failure mode from Section 9 on purpose.

Sample → standardize → clip

Contextual bandit · no critic

Group & clip G (group size) 8 ε clip 0.20

KL to reference β KL 0.040

Actions

Policy π over {A,B,C,D}

Group samples (R, Â)

mean(R)—

std(R)—

Clip fraction—

KL(π‖π_ref)—

Entropy H(π)—

Ledger

Sample a group, then run a GRPO step.

12. Worked example

Group of four. Suppose a group of \(G=4\) returns rewards \(R = [0.9,\, 0.4,\, 0.9,\, 0.2]\) (two samples happened to land on the best answer B). Then \(\operatorname{mean}(R) = 0.6\) and \(\operatorname{std}(R) = \sqrt{\tfrac14\sum (R_i-0.6)^2} \approx 0.309\). Advantages: \(\hat A = [0.97,\,-0.65,\,0.97,\,-1.29]\) (using \(\varepsilon\approx0\)) — the two B-samples get a healthy positive push, the two worse answers get pushed down, and their magnitudes already reflect how far each reward sits from the group's own average.

One clipped term. Take the first sample: \(\hat A_1 = 0.97\). Suppose after one gradient sub-step the ratio for that sample is \(r_1(\theta) = 1.31\) and \(\varepsilon = 0.2\), so the clip band is \([0.8, 1.2]\). Unclipped term: \(1.31 \times 0.97 \approx 1.271\). Clipped term: \(1.2 \times 0.97 \approx 1.164\). The objective takes \(\min(1.271, 1.164) = 1.164\) — the clip is active, and because the unclipped branch was larger (would keep pushing the ratio further from \(1\)), its gradient contribution is zeroed for this sample this step.

Freeze θ_old; sample o_1..o_G ~ π_θ_old(·|q).
Score R_i = r(q,o_i); compute mean(R) and std(R) over the group.
Â_i = (R_i − mean(R)) / (std(R) + ε) — a per-prompt, zero-parameter baseline.
If std(R) ≈ 0 (degenerate group), every Â_i ≈ 0 — no update this round.
Otherwise: r_i(θ) = π_θ(o_i|q)/π_θ_old(o_i|q); form min(r·Â, clip(r,1±ε)·Â).
Subtract β·D̂_KL[π_θ‖π_ref]; ascend θ. No critic was fit anywhere in this loop.

13. What follows

GRPO closes out this course's tour of on-policy, critic-light policy optimization for structured/LLM-style tasks. SAC (soft actor-critic) turns back to continuous-control RL in the classic sense: off-policy, replay-buffer-based, with an explicit entropy-regularized objective and — unlike GRPO — a pair of learned critics back in the loop, this time trained for sample efficiency rather than trust-region stability.

← PPO

Up next

SAC

Off-policy, entropy-regularized actor-critic — the critic comes back.

Chapter 10 · soon