The generative reward model: the verdict as text

A discriminative reward model maps an answer to a number. A generative reward model first says what it thinks of the answer, then gives the number.

Chapters 01 to 08 all share one design. A Transformer reads the prompt and response, and a small head on the last hidden state outputs one or more scores. Whatever checking the model does happens silently inside a single forward pass.

A generative reward model (GRM) drops the numeric head. It is a language model asked to evaluate: it reads the prompt and response, generates a critique \(c\), and ends with a verdict \(r\), such as "Yes", "No" or "8/10":

The critique is written in the same tokens the model uses for everything else. The model can therefore check arithmetic, restate the question or compare against a reference before it commits to a verdict.

A discriminative reward model is a function

The factorisation is just the order of generation: first the critique, then the verdict conditioned on it. The quantity we actually want is the verdict probability with the critique summed out:

The sum runs over every possible critique, so it cannot be computed exactly. The model may reach a verdict along many lines of reasoning; Equation 3 averages over them. Section 9 estimates it by sampling. The special case with an empty critique, \(c=\varnothing\), is a model that answers "Yes" or "No" immediately.

The last two rows are the main argument for the generative form. Verification becomes another text-generation task, the kind of task the model was pre-trained on, and its accuracy can be raised after training by spending more tokens.

Question: what is \(17\times 8\)? Candidate answer: \(17\times8=126\).

A discriminative RM must judge in one pass. It might output

A generative RM writes the check first:

and assigns \(P(\text{correct})=0.01\). The two numbers are similar. The difference is that the generative model's verdict rests on an intermediate result, 136, that it computed in the open.

The benefit grows with the difficulty of the check. Take the equation \(3(x+2)=15\), and a candidate solution whose first line is \(3x+2=15\). A classifier must spot the error from the surface of the text. A generative RM can redo the step:

Each line of the critique is an easy step that conditions the next. The verdict token is predicted after the evidence is on the page, not before.

RL and best-of-\(N\) need a number, not a word. The cleanest way to get one is to read it from the model's next-token distribution. Append a question \(I\) such as "Is the answer correct? Answer Yes or No." and take

where \(\ell\) are the logits at the verdict position. The right-hand form renormalises over the two allowed answers. If \(\ell_{\text{Yes}}=2.1\) and \(\ell_{\text{No}}=-0.4\), the score is \(\sigma(2.5)\approx0.924\). This is the approach of Generative Verifiers (GenRM): verification posed as next-token prediction, with the probability of "Yes" as the reward.

For a rating scale the same idea gives an expected score. If the model puts probability 0.2, 0.5 and 0.3 on the tokens "7", "8" and "9", then

Suppose the training set holds tuples \((x,y,c^\ast,r^\ast)\): a prompt, a response, a good critique and the correct verdict. Concatenate the target \(z=[c^\ast,r^\ast]\) and train with the ordinary language-model loss:

Unlike a scalar head, the model learns to generate evaluation text. With no critiques available, the same loss on a one-token target trains a direct verifier:

This is binary cross-entropy in disguise, but computed through the existing next-token head instead of a new one.

Where do good critiques \(c^\ast\) come from? Three common sources:

• Human-written critiques. Highest quality, most expensive.
• A stronger model's critiques, distilled into a smaller judge.
• Self-generated and filtered. Sample several critiques from the model itself and keep only those whose verdict matches the known label \(r^\ast\). This is rejection sampling: it needs only verdict labels, which are cheap when answers can be checked.

Because Equations 7 and 8 are the same loss as ordinary fine-tuning, judge data can be mixed with other instruction data in one training run.

The simplest generative RM needs no reward-model training at all. Take a strong instruction-tuned model and prompt it to evaluate. This is LLM-as-a-judge, widely used both for evaluation benchmarks and as a source of AI feedback for training. Two prompt formats dominate.

Pairwise judgments are usually more consistent, because comparing is easier than calibrating an absolute scale. Single-answer grades are cheaper for \(N\) candidates (\(N\) calls instead of up to \(\binom N2\)) and can be compared across prompts. The pairwise verdict probability plays the role of the Bradley–Terry probability of Chapter 03:

A third variant, reference-guided grading, supplies a correct solution in the prompt. It helps most on maths and code, where judges are otherwise easily misled by a confident but wrong answer.

A prompted judge inherits the habits of the model behind it. Three biases are well documented:

• Position bias. In pairwise prompts, the judge favours whichever response appears first (or second), independent of content.
• Verbosity bias. Longer, more detailed-looking answers are preferred even when the extra length adds nothing.
• Self-preference. A judge tends to rate text written by itself, or by its own model family, more highly.

Position bias has a simple fix: ask both orders and average. Suppose the judge prefers A with probability 0.70 when A is shown first, and with probability 0.40 when A is shown second. Then

and half the gap, \(\tfrac12(0.70-0.40)=0.15\), estimates how much the first slot alone is worth. When the two orders disagree on the winner, as here, many pipelines record a tie rather than trust either call.

A single sampled critique can go wrong: an arithmetic slip in the check itself, or a line of reasoning that misses the real error. Equation 3 says the verdict we want is an average over critiques. Monte-Carlo estimation gives it directly. Sample \(K\) critiques independently and average the verdict probabilities:

This is a form of inference-time compute: the same trained judge gives more reliable verdicts if we pay for more samples. The standard error shrinks as \(1/\sqrt K\), so going from 1 to 4 critiques halves it, and from 4 to 16 halves it again. A discriminative RM has no such dial; its one forward pass is all it can do.

Two caveats. First, averaging removes noise between critiques, not bias shared by all of them: if the judge systematically misses a kind of error, every critique misses it. Second, a common alternative is majority vote over the parsed verdicts. It is robust to an occasional overconfident critique but discards the confidence information of Section 5.

The cost difference is large. Both models first read \((x,y)\), a parallel prefill. A discriminative RM stops there. A generative RM then decodes token by token. With critiques of length \(L_c\), it generates about

With \(L_c=300\) and \(K=8\) that is about 2,400 sequential decoding steps for one judgment. In RL, where every rollout of every batch must be scored, this can exceed the cost of generating the rollouts themselves. The trade-off is simple: better verification potential, much more inference cost.

• Inference cost. Decoding critiques is orders of magnitude slower than one forward pass (Equation 12).
• Persuasive but wrong critiques. A fluent critique can contain its own errors, and readers, human or machine, tend to trust it.
• Unfaithful verdicts. The verdict token need not follow from the critique; the model may decide first and rationalise afterwards.
• Inherited biases. Position, verbosity and self-preference (Section 8) carry over from the underlying model.
• Prompt sensitivity. Small changes to the evaluation instruction or rating scale can shift scores noticeably.
• Exploitable through text. The policy writes the text the judge reads; under RL it can learn phrasings that sway the judge, including claims about its own correctness.

A judge evaluates a solution that is in fact correct. Five critiques are sampled, and after each one the probability of "Yes" is read off. Combine them and compare with relying on a single critique.

In this chapter the critique was a means to a verdict: prompted, or imitated from examples. The next chapter takes the reasoning seriously as the object of training. Critique-then-score reward models are trained to think at length before they score, so that more reasoning reliably buys better judgments.