10 · Critique-then-Score

1. From verdict to critique

Chapter 09 let the judge speak. This chapter trains it to think before it speaks.

In Chapter 09 a generative reward model produced its judgment as tokens: sometimes a bare "Yes" or "No", sometimes a critique followed by a verdict. The critique there was a means to the verdict. It came from a prompt, or was imitated from example critiques, and nobody asked whether it was good reasoning.

A reasoning reward model makes the critique the centre of the design. It first generates a critique $c$: text that checks the response, quotes the parts that matter, and names errors. The score is computed afterwards, conditioned on that critique. Three questions follow, and they organise this chapter: what should read the critique (Section 3), how to train the critic so that its reasoning leads to correct scores (Sections 6 to 8), and how to spend more inference compute on the judge (Sections 9 and 10).

2. Definition

Let $p_\theta$ be a language model used as a critic. Given $(x,y)$ it samples a critique token by token:

(1) \[ \hat c\sim p_\theta(c\mid x,y). \]

Definition 10.1 · Critique-then-score reward model

A critique-then-score reward model is a pair $(p_\theta,g_\phi)$: a critic that generates $\hat c$ as in Equation (1), and a scorer that reads the prompt, the response and the critique together,

\[ r(x,y)=g_\phi\big(x,y,\hat c\big). \]

Because $\hat c$ is sampled, $r(x,y)$ is a random variable. As in Chapter 09 (its Equation 3), the quantity we would like is the average over critiques, estimated with $K$ samples:

(2) \[ \bar r(x,y)=\mathbb E_{c\sim p_\theta(\cdot\mid x,y)}\big[g_\phi(x,y,c)\big] \approx\frac{1}{K}\sum_{k=1}^{K}g_\phi\big(x,y,\hat c_k\big). \]

With $K=1$ and greedy decoding this is the usual single-critique judge. The new ingredient is the scorer $g_\phi$: it need not be a verdict token at all. Section 10 returns to larger $K$.

3. Two read-outs: scalar head or verdict

The scorer $g_\phi$ can take two forms. Both read the sequence $[x,y,\hat c]$; they differ in what sits on top of the backbone.

Scalar head on the critique

Take the hidden state at the last token of $[x,y,\hat c]$ and apply a linear head, exactly as in the scalar chapter. Train the head with the Bradley-Terry loss.

critique, then w⊤h

Verdict tokens after the critique

Keep the language-model head. After the critique the model writes a verdict, such as "Score: 7" or "[[A]]". Read the score from the token or its probability.

critique, then "Score: 7"

The scalar-head version is the design of Critique-out-Loud reward models: one backbone first generates a critique, then a reward head reads the prompt, response and critique. Its head is trained on pairs exactly as before, now with critiques in the input:

(3) \[ \mathcal L_{\text{head}} =-\log\sigma\Big(g_\phi\big(x,y_w,\hat c_w\big)-g_\phi\big(x,y_l,\hat c_l\big)\Big). \]

The verdict version stays entirely in text. When the verdict is a number on a scale, a smoother score is the expected value under the model's distribution over score tokens:

(4) \[ r(x,y)=\sum_{s=1}^{10}s\cdot p_\theta\big(s\mid x,y,\hat c\big). \]

Note · Train the head on the critic's own critiques

At deployment the head will read critiques the critic generated, not reference critiques written by a human or a stronger model. Training the head on self-generated $\hat c$, rather than on reference $c^*$, keeps its training inputs and its test inputs from the same distribution.

4. Why write the critique down

Take the prompt "A train leaves at 14:50 and the trip takes 1 h 25 min. When does it arrive, and what does a 170 km ticket cost at $0.12 per km?" The response says "It arrives at 16:15 and the ticket costs $24.00." A critic might write:

Part 1. 14:50 + 1 h = 15:50; 15:50 + 25 min = 16:15. Correct.
Part 2. 170 × 0.12 = 20.40. The response says 24.00. Incorrect.
Summary: one of two parts is wrong; the error is arithmetic.
Score: 4

Three things happened that a single forward pass does not do explicitly.

Decomposition. The judgment was split into two small checks. Each check is easier than "is the whole answer good?".
Error localisation. The critique names the wrong part and the correct value. A scalar of 0.3 says something is off; the critique says what.
Interpretability. A person can read the critique and audit the score. If the critique is wrong, it is wrong in a visible way.

Chapter 09 made the basic point that a verdict written after the evidence rests on work done in the open. The three properties above are what a reasoning reward model is designed to make reliable, rather than hoped for from a prompt.

5. What it costs

Chapter 09 counted the decoding cost of one judgment: about $K(L_c+1)$ generated tokens for $K$ critiques of length $L_c$. Scale that to an RL step. Suppose each critique is 400 tokens and we sample $K=8$ per response. With 512 prompts and 8 responses per prompt, the judge must generate

(5) \[ 512\times 8\times 8\times 400=13{,}107{,}200\ \text{tokens}. \]

That is about thirteen million sequentially decoded tokens per step, before the policy update. A scalar reward model generates none. Reasoning reward models make this worse in two ways specific to this chapter.

Longer critiques. A judge trained to reason (Section 8) tends to write longer traces than a prompted one, so $L_c$ grows.
Training the critic costs generation too. Self-training and RL (Sections 7 and 8) sample many critiques per training pair, before the judge is ever used.

The design question is therefore not whether a critique helps, but whether it helps more than spending the same compute elsewhere, for example on more policy samples.

6. Training the critic: supervised critiques

The direct approach is imitation. Collect reference critiques $c^*$ for many $(x,y)$, written by people or by a stronger model, and fine-tune the critic with the language-modelling loss on the critique tokens:

(6) \[ \mathcal L_{\text{SFT}}=-\sum_{t=1}^{|c^*|}\log p_\theta\big(c^*_t\mid x,y,c^*_{<t}\big). \]

Chapter 09 used the same loss on critique plus verdict. The limits are the same as for any imitation: the critic can only be as careful as its references, and good critiques of hard answers are expensive to write. The loss also rewards copying the style of a reference critique, whether or not the critic's own checks are right.

7. Self-taught evaluators

Human critiques are scarce. A self-taught evaluator builds its own training data. The trick is to create pairs whose preferred side is known by construction. Take an instruction $x$, write a slightly altered instruction $x'$, and let a model answer both. The answer to $x$ is preferred for $x$ because the answer to $x'$ solves a different problem. No human labels the pair.

The current judge then writes several critiques and verdicts for each pair. Only judgments that reach the known verdict $v^*$ are kept:

(7) \[ \mathcal D_{\text{new}}=\big\{(x,y_A,y_B,\hat c,\hat v)\;:\;\hat v=v^*\big\}. \]

1 · Take unlabeled instructions $x$.

↓

2 · Build a pair with a known winner, e.g. answer $x$ and a perturbed $x'$.

↓

3 · Sample critiques and verdicts from the current judge.

↓

4 · Keep judgments whose verdict matches the known winner (Equation 7).

↓

5 · Fine-tune the judge on the kept critiques with Equation (6).

↓

6 · Repeat with the improved judge.

Remark · Correct verdict, unchecked reasoning

The filter checks only the verdict. A critique that reaches the right verdict for a wrong reason passes the filter and is trained on. The same gap reappears in the next section.

8. Reinforcement learning on the verdict

Filtering keeps good judgments but never pushes the judge away from bad ones. Reinforcement learning does both. Treat the judge as a policy: given $(x,y_A,y_B)$ it generates a critique and then a verdict $\hat v\in\{A,B\}$. Preference data already contains the right answer $v^*$, so the reward is checkable by a rule:

(8) \[ R(\hat c,\hat v)=\begin{cases}1 & \hat v=v^*\\ 0 & \hat v\ne v^*\text{ or no parsable verdict}\end{cases} \]

The judge is then trained to maximise expected reward, for example with GRPO, sampling several judgments per pair and comparing their rewards:

(9) \[ \max_\theta\;\mathbb E_{(x,y_A,y_B,v^*)}\;\mathbb E_{(\hat c,\hat v)\sim p_\theta}\big[R(\hat c,\hat v)\big]. \]

This is the recipe of reasoning reward models such as RM-R1: preference labels are turned into a verifiable reward, and the critique is learned as whatever reasoning helps reach the right verdict. No reference critique is needed, only the label.

Key idea · A preference label is a verifiable answer

To the judge, "which response is better?" is a question with a known answer, like a maths problem with an answer key. That is why the RL methods used for reasoning on verifiable tasks transfer to training reward models.

Note · Position shortcuts under RL

Chapter 09 handled position bias at inference by asking both orders. Under RL the risk is sharper: if the preferred response is usually shown first, the judge earns most of the reward of Equation (8) by always answering $A$, with no reasoning at all. Presenting every pair in both orders, $(y_A,y_B)$ and $(y_B,y_A)$, takes that shortcut away.

9. Principles and rubrics

"Is this good?" is vague for open-ended prompts. A rubric makes it concrete. A principle-guided judge first writes the criteria that matter for this prompt, then critiques each response against them, then scores:

(10) \[ x,y_{1:n}\;\longrightarrow\;\underbrace{P}_{\text{principles}} \;\longrightarrow\;\underbrace{c}_{\text{critique}} \;\longrightarrow\;\underbrace{s_1,\ldots,s_n}_{\text{scores}}. \]

For a request to explain recursion to a beginner, the principles might be "technically correct", "uses one concrete example", and "avoids jargon or defines it". Each response is then checked against the same three items, which makes scores for different responses more comparable.

DeepSeek's work on generalist reward modelling uses this pattern: the judge generates its own principles and critiques and outputs a pointwise score for each response. Because principles are generated per prompt, the same judge can be used on maths, code and open-ended writing.

10. Inference-time scaling of the judge

A sampled critique is one opinion. Sample $K$ of them and combine. For pairwise verdicts the natural rule is a majority vote; for pointwise scores, such as the per-response scores of Section 9, it is a sum or average:

(11) \[ \hat v=\operatorname{mode}\big(\hat v_1,\ldots,\hat v_K\big), \qquad S(y_i)=\sum_{k=1}^{K}s_i^{(k)} . \]

Chapter 09 showed that averaging $K$ verdict probabilities shrinks their variance as $1/K$. Here the question is different: how often is the final decision right? Model each sampled verdict as correct with probability $p$, independently. With an odd number $N=K$ of votes, the majority is correct when at least $(N+1)/2$ votes are, so

(12) \[ A_N(p)=\sum_{k=(N+1)/2}^{N}\binom{N}{k}p^{k}(1-p)^{N-k}. \]

At $p=0.7$: $A_1=0.700$, $A_3=0.784$, $A_5\approx0.837$, $A_{15}\approx0.950$. At $p=0.4$ the same sums move the other way: $A_1=0.400$, $A_3=0.352$, $A_5\approx0.317$. The step from $N$ to $N+2$ has a closed form:

(13) \[ A_{N+2}(p)-A_N(p)=\binom{N}{\frac{N-1}{2}}\big(p(1-p)\big)^{\frac{N+1}{2}}\,(2p-1). \]

Proposition 10.1 · Voting amplifies the sign of $p-\tfrac12$

For independent votes and odd $N$, the sign of Equation (13) is the sign of $2p-1$. If $p>\tfrac12$, majority accuracy rises with every extra pair of votes and tends to 1; if $p<\tfrac12$ it falls toward 0. This is Condorcet's jury theorem. Voting sharpens a judge that is better than chance; it cannot rescue one that is worse.

Real critiques are not independent. They come from the same model and share its blind spots. A crude way to see the effect: with probability $\rho$ all $N$ critiques make the same judgment (one draw, correct with probability $p$), and otherwise they are independent. Then

(14) \[ A_N^{\rho}(p)=\rho\,p+(1-\rho)\,A_N(p), \]

which can never exceed $\rho p+(1-\rho)$. With $\rho=0.5$ and $p=0.7$, even infinitely many votes give at most $0.85$. Shared errors put a ceiling on what sampling buys. One practical response is to filter out low-quality samples before voting, for example with a second model that rates each critique.

11. Failure modes

The judge biases and failure modes of Chapter 09 all carry over. Training the reasoning, and scaling it, adds twists of its own.

Persuasive but wrong. A fluent critique can assert a false check ("the arithmetic is correct") with the same confidence as a true one. A scalar head trained on critiques (Section 3) learns to trust the critique's conclusion, so a confident error in the text becomes a confident error in the score.
Rationalising a pre-chosen verdict. The critique may justify a decision the model has effectively already made. A simple test: edit or remove the critique and see whether the score moves. If it does not, the critique is decoration, not evidence. Formats that put the verdict before the critique have this failure built in.
Right verdict, wrong reason. Filtering (Section 7) and RL (Section 8) reward the verdict only. A critique that reaches the right answer by a flawed argument is reinforced just as strongly as a sound one.
Length and verbosity bias. Judges tend to favour longer, more detailed-looking responses. A policy trained against such a judge learns to pad. Under RL the judge's own critiques can also grow longer without growing better.
Voting amplifies systematic errors. On a slice of prompts where the judge is usually wrong ($p<\tfrac12$), Proposition 10.1 says more votes make it more confidently wrong.
Cost. Millions of generated tokens per RL step (Equation 5), plus the generation spent training the critic.

12. Laboratory: voting over sampled critiques

Each sampled judgment is correct with probability $p$. Set $p$, the number of votes $N$, and the correlation $\rho$ between critiques. The lab computes the majority-vote accuracy exactly from Equations (12) and (14) and tabulates it for every odd $N$ up to 31. "Simulate" checks the formula by sampling.

Majority vote of N critiques

$A_N(p)=\sum_{k\ge (N+1)/2}\binom{N}{k}p^k(1-p)^{N-k}$

Single-judgment accuracy p 0.70

Sampled critiques (odd) N 5

Shared-error probability ρ 0.00

Check by sampling

One critique-

Majority of N-

Change vs N = 1-

Limit as N grows-

Accuracy by number of votes

Move a slider.

Try $p=0.7$, then $p=0.4$: the table climbs in one case and sinks in the other. At $p=0.5$ voting does nothing. Then set $\rho=0.5$ and watch the curve flatten below its ceiling.

13. Worked example, step by step

A reasoning judge compares two responses. On pairs like this one, a single sampled critique reaches the right verdict 70% of the time. We sample five critiques and take the majority. How accurate is the vote, and what if critiques share errors?

The five verdicts come back as $A,A,B,A,B$. The majority is $A$, three votes to two.
Majority of 5 is correct when at least 3 votes are correct: $A_5=\binom53p^3(1-p)^2+\binom54p^4(1-p)+p^5$.
With $p=0.7$: $10(0.343)(0.09)+5(0.2401)(0.3)+0.16807=0.3087+0.36015+0.16807=0.83692$.
Check with Equation (13): $A_3=0.784$, and $A_5-A_3=3\,(0.21)^2(0.4)=0.05292$, so $A_5=0.83692$.
If instead $p=0.4$, the same sum gives $A_5\approx0.317$: voting made a weak judge worse.
With shared errors $\rho=0.5$ and $p=0.7$: $A=0.5(0.7)+0.5(0.83692)\approx0.768$. Most of the gain from voting is gone.

14. What comes next

Reasoning reward models spend more computation on each judgment to make it better. The next chapter goes the other way. It drops the separate reward model entirely and shows that a policy trained with DPO already contains a reward, written in its own log-probabilities.

Key idea to remember

Generate a critique, then score with the critique in view. Train the critique by rewarding the verdict it leads to, and scale the judge by voting, which helps only when a single judgment beats chance and critiques do not share their mistakes.

← Generative Reward Model

Next chapter

Implicit reward model (DPO)

A reward read from the policy's own probabilities.

Continue →

Critique-then-score: reward models that reason first