Pearl · Rewards
Pipeline

Reward Models · Course overview

What is a reward model?

Abstract. A reward model (RM) is a learned judge. It looks at a prompt and a candidate answer (or a reasoning step) and returns a score, a preference, or a critique. In modern LLM and VLM training it sits between supervised fine-tuning and reinforcement learning: humans (or programs) teach the judge, then the judge teaches the policy. This page places the RM in the full training pipeline, contrasts it with SFT and RL, and maps the architectures we will study next.

1. What is a reward model?

The language model generates text. The reward model judges that text.

In the simplest form:

(1) \[ (x, y) \;\mapsto\; r_\theta(x,y) \]

Here \(x\) is the prompt, \(y\) is a response, and \(r_\theta\) is a number (or a probability, or a ranking, depending on the architecture).

The RM is not the agent that chats with users. It is the scoring function used to select answers, filter data, guide search, or drive RL.

One-sentence definition

A reward model is a learned preference or correctness function that turns human, programmatic, or synthetic feedback into a signal a training loop can use.

2. Where it fits in LLM / VLM training

Think of three big stages. The reward model lives mainly in the last one.

Pre-training · predict next token on huge corpora. Learn language and world knowledge. No RM yet.
↓
Supervised fine-tuning (SFT) · imitate high-quality demos. Learn the format of helpful answers. Still no RM required.
↓
Post-training with feedback · collect preferences or verifications, train an RM, then optimize the policy with that RM (RLHF, RLAIF, GRPO, …).

A common RLHF loop looks like this:

1 · Collect Humans (or another model) rank or label answers
2 · Train RM Fit a judge so preferred answers get higher scores
3 · Optimize policy PPO / GRPO / best-of-N use the judge as the reward
Not only chat models

The same pattern appears for VLMs (image + text), math tutors, code agents, and tool-using systems. Whenever “good behavior” is hard to write as a hard rule, a learned reward model is a practical substitute.

3. Reward model vs SFT vs RL

SFTReward modelRL (with RM)
Learns How to imitate demos How to score candidates How to act for higher score
Needs \((x,y^*)\) pairs Preferences or labels A reward signal (often the RM)
Output A policy \(\pi\) A judge \(r_\theta\) An improved policy \(\pi\)
Role Start from good behavior Define “better” Climb that definition

Short version:

  • SFT copies answers.
  • RM learns what “better” means.
  • RL pushes the generator toward higher RM scores.

You can use an RM without RL (best-of-\(N\), filtering, search). You cannot do classical RLHF-style preference optimization without some reward or preference signal. That signal is often a reward model.

Common confusion

People say “we did RLHF” when they only trained a preference model and reranked. Reranking uses the RM at inference. RLHF also updates the generator’s weights with that signal. Both are useful. They are not the same thing.

4. Why reward models are important

  1. Humans are slow. You cannot ask a person to score every RL sample. An RM amortizes human (or verifier) judgment into a fast network.
  2. Preferences are relative. People are better at “A beats B” than at giving absolute scores. RMs turn those comparisons into a usable number.
  3. RL needs a scalar (or dense) signal. PPO and friends expect a reward. The RM is how preference data becomes that reward.
  4. Process needs denser feedback. For math and agents, checking only the final answer wastes most of the trajectory. Process reward models score steps along the way.

5. How we classify reward models

“ORM versus PRM” is only one axis. A full taxonomy has four.

1 · What it outputs

One scalar, several objectives, a pairwise preference, a distribution, or a written critique.

score · preference · critique

2 · What it evaluates

Whole answer (ORM), each step (PRM), or future success of a prefix (value model).

outcome · process · value

3 · How it is trained

Binary classification, pairwise ranking, listwise ranking, regression, or generative supervision.

point · pair · list · gen

4 · Where labels come from

Humans, another AI, unit tests / compilers, environment reward, or rollouts.

human · AI · verifier · rollout

Example: a classical InstructGPT reward model is scalar + outcome + pairwise + human. A Math-Shepherd-style PRM is closer to scalar/process + rollout labels. A judge LLM that writes “A is better because…” is generative + pairwise.

6. Learning roadmap

We study architectures one by one, with worked numerical examples on every page.

#ArchitectureStatus
01Scalar discriminative RM (classical RLHF)Open
02Pointwise binary verifierOpen
03Direct pairwise comparatorOpen
04Listwise ranking RMOpen
05Outcome Reward Model (ORM)Open
06Process Reward Model (PRM)Open
07Value / future-success modelSoon
08Multi-head / multi-objective RMSoon
09Generative reward modelSoon
10Critique-then-score (reasoning RM)Soon
11Implicit reward modelSoon
12Personalized / conditional RMSoon
13Ensemble / uncertainty-aware RMSoon
14Hybrid learned + rule-based systemsSoon

Categories overlap on purpose. A PRM can be discriminative or generative. An ORM can be pointwise or pairwise. We separate ideas so each page can stay focused on one clear mechanism.

7. One running example

Prompt:

(2) \[ x=\text{“Explain why the sky appears blue.”} \]

Response A (better):

Blue light is scattered more strongly than red light by molecules in the atmosphere.

Response B (worse):

The sky reflects the blue color of the ocean.

A human marks \(A \succ B\). Different architectures will handle this pair differently:

  • Scalar RM: learn \(r(A)>r(B)\) via Bradley-Terry.
  • Pointwise: if we have absolute labels, score each as correct/incorrect (harder for open-ended prose).
  • Direct comparator: read \(A\) and \(B\) together and predict \(P(A\succ B)\).
  • Listwise: if we have four candidates, rank the whole list at once.

We reuse this sky example (and a math discount example for PRMs) across chapters so the differences stay concrete.

8. What follows

Next we build the classical scalar discriminative reward model from scratch: Transformer backbone, linear head, Bradley-Terry loss, and a full numerical walkthrough.

See also

  • Christiano et al., 2017; Stiennon et al., 2020; Ouyang et al., 2022 (InstructGPT).
  • Lightman et al., 2023 (PRM800K); Wang et al., 2024 (Math-Shepherd).
  • Pearl RL course (../ppo.html, ../grpo.html) for how RMs are consumed.

Next chapter

Scalar discriminative RM

One number per answer. Learned from \(A \succ B\).

Start →