01 · Scalar Reward Model

1. The basic problem

We want a function that looks at a prompt and an answer and returns a number saying how good that answer is.

Take a simple prompt:

(1) \[ x = \text{"Explain why the sky appears blue."} \]

Two candidate answers:

Response A Blue light is scattered more strongly than red light by molecules in the atmosphere.

Response B The sky reflects the blue color of the ocean.

A human reads both and says A is better: \(A \succ B\). Response A is correct physics. Response B sounds plausible but is wrong.

The scalar reward model's whole job is to turn that human judgment into a function \(r_\theta(x,y)\) that assigns a real number to any \((x,y)\) pair, such that:

(2) \[ r_\theta(x, A) > r_\theta(x, B) \]

Once we have that function, we can use it to pick good answers, filter bad ones, or steer a policy toward higher-scoring answers with RL. The rest of this chapter is about how \(r_\theta\) is built and trained.

One-sentence definition

A scalar reward model is a neural network that reads a prompt and a full response and outputs one real number: an overall quality score for that response, nothing more.

2. Architecture: transformer to scalar

The scalar RM reuses a pretrained transformer, the same kind of backbone used for language modeling. The only change is what sits on top of it.

Concatenate the prompt and the response into one sequence of tokens, \(x \oplus y\), and run it through the transformer. Take the hidden state at the final token, call it \(h_T\). Feed \(h_T\) into a small linear layer with a single output:

(3) \[ r_\theta(x, y) = w^\top h_T + b \]

\(w\) is a weight vector the same size as the hidden state, \(b\) is a single bias scalar. The language modeling head (the part that predicts the next token over a huge vocabulary) is thrown away and replaced by this one-output head. Everything else, the token embeddings and the transformer layers, is usually initialized from a pretrained or SFT-tuned checkpoint and then fine-tuned further.

Input: prompt and response tokens, concatenated: \(x \oplus y\)

↓

Transformer backbone (same architecture as the base language model)

↓

Take the hidden state at the last token: \(h_T\)

↓

Linear head: one neuron, weights \(w\), bias \(b\)

↓

Output: a single number, \(r_\theta(x,y)\)

   x = "Explain why the sky appears blue."
   y = "Blue light is scattered more..."

   [ x ⊕ y ]  (one token sequence)
        |
        v
   Transformer  (12, 24, ... layers)
        |
        v
   h_T          (hidden vector, last token)
        |
        v
   Linear head: w^T h_T + b
        |
        v
   r(x, y) = 2.7    <- one real number, done

Why the last token?

By the time the transformer has processed the whole prompt and the whole response, the hidden state at the final position has attended to everything before it. It is a reasonable summary of "what was just read," which is why it is a common choice for the pooling point. Some implementations use a special end-of-sequence token instead of the literal last token, or average over positions, but the idea is the same: compress the full \((x,y)\) pair into one vector, then map that vector to one number.

3. What is the final-token hidden state \(h_T\)?

\(h_T\) is not the last word itself. It is the transformer's numerical, context-aware representation of the whole sequence at the final position.

Suppose the input is:

The answer is 42.

After tokenization:

\[ [ t_1=\text{The},\quad t_2=\text{answer},\quad t_3=\text{is},\quad t_4=\text{42},\quad t_5=\langle\text{EOS}\rangle ] \]

Here \(T = 5\): there are five tokens. The transformer produces one hidden-state vector for every token:

\[ h_1,\; h_2,\; h_3,\; h_4,\; h_5 \]

Each \(h_t\) is a list of numbers representing what the model understands at that position. For example:

\[ h_5 = [0.21,\; -0.47,\; 0.83,\; \ldots] \]

Because the last token, usually an end-of-sequence token, can attend to all previous tokens, its hidden state contains information about the entire prompt and response:

Prompt + Response + <EOS>
                      |
                      v
                     h_T
       contextual representation of everything before it

Definition \[ \boxed{h_T=\text{the Transformer's internal representation at the final token}} \]

The reward head then converts this large vector into one number:

(3b) \[ r = w^\top h_T + b \]

For example, a 4096-dimensional hidden state collapses to a single score:

\[ h_T \in \mathbb{R}^{4096} \quad\longrightarrow\quad r = 3.7 \]

Complete prompt and answer

↓

Transformer creates contextual vectors

↓

Take vector at final / EOS token: \(h_T\)

↓

Linear reward head

↓

One reward score

Key point

\(h_T\) is not the final word itself. It is the model's numerical, context-aware representation of the whole sequence at the final position.

4. Training on winner and loser pairs

We do not have absolute quality labels like "this answer is a 7 out of 10." What we have is comparisons: for a prompt \(x\), a human (or another model) is shown two responses and picks the better one.

Call the chosen response the winner, \(y_w\), and the rejected one the loser, \(y_l\). Each training example is a triple \((x, y_w, y_l)\). Run both responses through the same reward model to get two scores:

(4) \[ r_w = r_\theta(x, y_w), \qquad r_l = r_\theta(x, y_l) \]

We want \(r_w > r_l\) most of the time. To turn that into a trainable probability, classical RLHF borrows the Bradley-Terry model, an old idea from ranking chess players and sports teams from pairwise match results. It says the probability that \(y_w\) is preferred over \(y_l\) is a sigmoid of the score difference:

(5) \[ P(y_w \succ y_l) = \sigma(r_w - r_l), \qquad \sigma(z) = \frac{1}{1 + e^{-z}} \]

The sigmoid function \(\sigma\) squashes any real number into the range \((0, 1)\), so this is a valid probability. If \(r_w\) is much bigger than \(r_l\), the model is very confident the winner is better. If the two scores are close, the model is unsure. If \(r_l\) is bigger, the model thinks the loser is actually better, which is the mistake we want training to correct.

Bradley-Terry model

Given two items with hidden "strength" scores \(r_1\) and \(r_2\), the probability item 1 beats item 2 in a head-to-head comparison is \(\sigma(r_1 - r_2)\). It only depends on the difference of the two strengths, never on either score alone. This one assumption is the entire statistical backbone of the classical RLHF reward model.

5. A numerical example

Suppose the reward model, after seeing our sky example, outputs:

(6) \[ r_w = r_\theta(x, A) = 3, \qquad r_l = r_\theta(x, B) = 1 \]

The difference is \(\Delta = r_w - r_l = 2\). Plugging into the sigmoid:

(7) \[ \sigma(2) = \frac{1}{1 + e^{-2}} \approx 0.881 \]

The model assigns about an 88.1% probability that A is preferred over B. That matches the human label, so this is a good prediction. Now look at what happens if the model has the ranking backwards.

Correct ranking

\(r_w = 3\), \(r_l = 1\), so \(\Delta = 2\).

\(P(y_w \succ y_l) = \sigma(2) \approx 0.881\)

Confident and right. Small loss.

Flipped ranking

\(r_w = 1\), \(r_l = 3\), so \(\Delta = -2\).

\(P(y_w \succ y_l) = \sigma(-2) \approx 0.119\)

Confident and wrong. Large loss.

In the flipped case, the model puts only 11.9% probability on the correct winner. It has effectively decided the loser is better. This is exactly the situation the loss function in the next section is built to punish hard.

6. The loss function

Training uses the negative log-likelihood of the Bradley-Terry probability. For one preference pair:

(8) \[ \mathcal{L}(\theta) = -\log \sigma(r_w - r_l) \]

This single line is the entire training objective of the classical scalar reward model. Averaged over a batch of preference pairs, it becomes the loss the optimizer minimizes. Three things to notice:

If \(r_w - r_l\) is large and positive, \(\sigma(r_w - r_l)\) is close to 1, \(\log\) of something close to 1 is close to 0, so the loss is small. The model is already doing well and gets a weak gradient signal.
If \(r_w - r_l\) is negative, \(\sigma(r_w - r_l)\) is small, \(-\log\) of a small number is large, so the loss is large. The model gets a strong gradient signal telling it to raise \(r_w\) and lower \(r_l\).
The loss is never satisfied by "close enough." It keeps rewarding a bigger margin between winner and loser, though the gradient shrinks as the margin grows, so most of the pressure is on pairs the model still gets wrong or is unsure about.

Using the numbers from the previous section: the correct ranking (\(\Delta = 2\)) gives loss \(-\log(0.881) \approx 0.127\), small. The flipped ranking (\(\Delta = -2\)) gives loss \(-\log(0.119) \approx 2.13\), about 17 times larger. That gap is what drives learning: wrong and confident pairs dominate the gradient.

7. The score is relative, not absolute

Here is a property that surprises people the first time they see it. Add the same constant \(c\) to both scores, and nothing about the training signal changes:

(9) \[ \sigma\big((r_w + c) - (r_l + c)\big) = \sigma(r_w - r_l) \]

The constant cancels inside the subtraction. The loss only ever sees the difference \(r_w - r_l\), never \(r_w\) or \(r_l\) by themselves.

Original scores

\(r_w = 3\), \(r_l = 1\)

\(\Delta = 2\), \(P \approx 0.881\), loss \(\approx 0.127\)

Shift both by +100

\(r_w = 103\), \(r_l = 101\)

\(\Delta = 2\), \(P \approx 0.881\), loss \(\approx 0.127\)

What this means in practice

A single score like \(r_\theta(x,y) = 2.7\) has no meaning by itself. There is no fixed zero point and no fixed unit. The number 2.7 only means something when compared against another score produced by the same reward model on a related response, usually for the same prompt.

This is why you cannot compare raw reward scores across two different reward models, or even meaningfully compare \(r_\theta(x_1, y)\) against \(r_\theta(x_2, y')\) for two unrelated prompts. Only differences within a comparable set carry information.

8. How is it used?

Once trained, \(r_\theta\) is used in two main ways.

A. Best-of-N selection

Generate several responses:

\[ y_1,\; y_2,\; \ldots,\; y_N \]

Score all of them:

\[ r_1,\; r_2,\; \ldots,\; r_N \]

Select the response with the maximum predicted reward:

(7) \[ y^\ast = \arg\max_{y_i}\, r_\theta(x, y_i) \]

Meaning

Choose the response with the maximum predicted reward. No policy training is required: this is pure selection at inference time.

B. Reinforcement learning

The policy generates a response:

\[ y \sim \pi_\phi(\cdot \mid x) \]

The reward model supplies:

\[ r_\theta(x, y) \]

PPO or another RL algorithm updates the policy so that higher-reward responses become more likely.

Policy generates answer

↓

Reward model assigns score

↓

RL increases probability of high-score answers

This was the central reward-model role in classical RLHF pipelines such as InstructGPT. Best-of-N gets more expensive as \(N\) grows, since every candidate needs a full generation and a reward-model pass. RL bakes the preference into the policy's weights, so inference stays cheap (one generation), but training is more complex and can overfit to the reward model's blind spots, which is the topic of the weaknesses section below.

9. What the model actually learns

It is tempting to think a reward model trained on "A is better than B" learns something like truth or correctness. It does not, not directly. It learns to predict what the labelers preferred, which is a pattern in the training data, not a definition of quality.

Most of the time these overlap: labelers usually prefer accurate, helpful, well-organized answers, so the reward model does end up rewarding those things too. But the reward model has no independent way to check if an answer is factually correct. It only knows what tended to get chosen in its training set.

A concrete failure mode

If labelers historically preferred longer, more confident-sounding answers, even when a shorter answer was equally correct, the reward model will learn to reward length and confidence. It will happily give a high score to a long, fluent, wrong answer, because from its point of view, "long and fluent" is what winners tend to look like.

10. Weaknesses of the scalar reward model

The scalar architecture is simple and it works, which is why it powered the first wave of RLHF systems. It also has real limits.

Four structural weaknesses
Weakness	What happens	Why it matters
No localization	One number covers the entire response.	Cannot say which sentence, step, or word caused a low score.
Inherited biases	Learns whatever patterns were in labeler choices.	Length, tone, and formatting biases get baked in as if they were quality.
Reward hacking	RL policy finds inputs that score high without being good.	Optimizing hard against an imperfect judge tends to find its blind spots.
Single dimension	Helpfulness, honesty, and safety are all mixed into one number.	No way to see or control the trade-off between competing goals.

No localization. If a math answer gets the final number wrong because of one bad step in the middle, the scalar RM can only say "this whole response scored low." It cannot point at the bad step. Process reward models, covered later in this course, exist specifically to fix this.

Biases from training data. If human raters (or an AI judge standing in for them) have a systematic preference, for example favoring longer answers, more hedging language, or answers that agree with the user, the reward model absorbs that preference as if it were genuine quality.

Reward hacking

When you run RL against a fixed, imperfect reward model, the policy is not optimizing for "actually good answers." It is optimizing for "answers that score high on \(r_\theta\)." Push hard enough and the policy finds the gap between the two: repeating certain phrases, padding length, or adopting a confident tone the RM associates with quality, all without actually improving the answer. This is Goodhart's law in action: once a measure becomes a target, it stops being a reliable measure.

Single dimension. Real answers trade off multiple goals: being maximally helpful can conflict with being maximally careful or safe. A single scalar collapses all of that into one number, so the RM (and the policy trained on it) has no explicit way to represent "very helpful but slightly risky" versus "very safe but less useful." Multi-head reward models, covered later in this course, address this directly.

11. Laboratory: Bradley-Terry playground

Move the two sliders to set the winner score \(r_w\) and the loser score \(r_l\). Watch the margin \(\Delta\), the predicted preference probability, and the loss update live.

Winner vs. loser scores

Bradley-Terry model

Winner score r_w 3.0

Loser score r_l 1.0

Quick examples

Δ = r_w − r_l2.00

P(winner) = σ(Δ)88.1%

Loss = −log σ(Δ)0.1269

What is happening

Move a slider to see the trace.

Try pushing \(r_l\) above \(r_w\). The probability drops below 50% and the loss climbs quickly. That is the reward model actively being wrong, and it is exactly the signal that drives the biggest gradient updates during training.

12. Worked example, step by step

Suppose after a training step the reward model gives \(r_w = 1.5\) for the winning response and \(r_l = 0.9\) for the losing one. Reveal the steps to compute the loss by hand.

Start with the two raw scores: \(r_w = 1.5\), \(r_l = 0.9\).
Compute the margin: \(\Delta = r_w - r_l = 1.5 - 0.9 = 0.6\).
Convert the margin to a probability: \(P = \sigma(0.6) = \dfrac{1}{1+e^{-0.6}} \approx 0.646\).
Compute the loss: \(\mathcal{L} = -\log(0.646) \approx 0.437\).
Interpret: the model already favors the winner (\(P > 0.5\)) but is not confident. Training will keep pushing \(r_w\) up and \(r_l\) down to widen the margin.

13. What comes next

The scalar discriminative reward model is the workhorse of classical RLHF: simple to train, cheap to run, and good enough to power systems like InstructGPT. Its main cost is exactly what we listed in the weaknesses section: one blurry number per response, prone to picking up the wrong patterns, and vulnerable to being gamed by RL.

Next we look at a different framing: pointwise binary verifiers, which trade "which response is better" for a sharper, easier question: "is this response correct, yes or no."

Key loss to remember

\[ \mathcal{L}(\theta) = -\log \sigma(r_\theta(x,y_w) - r_\theta(x,y_l)) \]

Everything in this chapter reduces to this one line: score the winner, score the loser, take the sigmoid of the difference, and push that probability toward 1 with a log loss.

Pointwise binary verifier

One question per answer: correct, or not.

Start →