Reward Models · Chapter 05

Outcome reward model (ORM)

Abstract. An ORM judges the complete solution only after it finishes. It maps a problem and a full answer to one final reward. It does not score individual reasoning steps. Labels are often automatic (answer match, unit tests, task success), which makes ORMs scalable, but credit assignment is weak: a lucky final answer can hide broken intermediate steps.

1. What an ORM does

An ORM judges the complete solution only after it finishes.

(1) \[ \text{Problem} + \text{Full solution} \;\longrightarrow\; \text{one final reward} \]

It does not score individual reasoning steps. The whole trajectory collapses into a single outcome signal.

2. Simple example

Problem:

What is \(15 \times 6\)?

Solution A

\[ 15 \times 6 = 90 \]

ORM output:

\[ r_A = 0.98 \]

Solution B

\[ 15 \times 6 = 80 \]

ORM output:

\[ r_B = 0.04 \]

The ORM says whether the overall answer is correct or good. It does not explain which line of work was responsible.

3. Architecture

Problem + complete reasoning + final answer
                    |
                    v
               Transformer
                    |
                    v
          Final-token hidden state h_T
                    |
                    v
               Reward head
                    |
                    v
             One outcome score

Mathematically:

(2) \[ r_\theta(x,y)=w^\top h_T+b \]

\(x\): problem
\(y\): complete solution
\(h_T\): final-token representation
\(r_\theta\): outcome reward

For correctness classification, pass the score through a sigmoid:

(3) \[ p_\theta(\text{correct}\mid x,y) = \sigma(w^\top h_T+b) \]

Relation to earlier chapters

Architecturally this looks like the scalar RM or the pointwise verifier. The ORM distinction is what is being judged: the finished outcome of a full solution, usually with an automatic correctness label, not a preference between two open-ended replies.

4. Training

Suppose the label is whether the final answer is right:

(4) \[ z= \begin{cases} 1, & \text{final answer correct}\\ 0, & \text{final answer incorrect} \end{cases} \]

Binary cross-entropy loss:

(5) \[ \mathcal{L} = -\big[z\log p+(1-z)\log(1-p)\big] \]

The model learns to give \(p \to 1\) for correct solutions and \(p \to 0\) for incorrect ones.

Loss at a glance

If \(z=1\), loss \(=-\log p\) (push \(p\) up). If \(z=0\), loss \(=-\log(1-p)\) (push \(p\) down).

5. Main weakness

Consider a solution with a broken middle step and a lucky finish:

Step 1: 2 + 3 = 6       ← wrong
Step 2: 6 - 1 = 5
Final answer: 5          ← accidentally correct

An ORM trained only on the final answer may reward this solution:

\[ r \approx 1 \]

even though the reasoning is invalid. It cannot identify where an error occurred. All credit (or blame) is attached to the whole trajectory at once.

6. Main advantage

ORM labels are easy to obtain.

For mathematics:

\[ \text{predicted answer} = \text{reference answer?} \]

For code:

\[ \text{did it pass the unit tests?} \]

For an agent:

\[ \text{was the task completed?} \]

Therefore, ORM training can use large amounts of automatically labelled data. That scalability is why ORMs appear so often in math and coding RL.

7. ORM versus PRM

ORM	PRM
One reward after completion	Reward after every step
Judges final outcome	Judges reasoning process
Easier to label	Harder to label
Sparse feedback	Dense feedback
Cannot localize errors	Can identify bad steps

For a three-step solution:

ORM

\[ [r_{\mathrm{final}}]=[0] \]

It only says the solution failed.

PRM

\[ [r_1,r_2,r_3]=[0.97,\,0.08,\,0.03] \]

It shows that the error probably began at Step 2.

8. Core understanding

Remember \[ \boxed{ \text{ORM evaluates whether the completed solution succeeded,} \text{ not how it reached the answer.} } \]

It is simple and scalable, but provides weak credit assignment because all reasoning steps receive only one final signal.

9. Laboratory · ORM outcome playground

Final logit → probability → BCE

Outcome label only

Logit \(a = w^\top h_T + b\) 3.90 Label \(z\) (final answer correct?) correct (1)

\(p=\sigma(a)\)-

Loss-

Adjust the outcome logit and label.

10. Worked example, step by step

Problem: \(15\times 6\). Solution A ends at 90. Label \(z=1\).
Run transformer on problem + full solution. Read \(h_T\).
Compute logit \(a=w^\top h_T+b\). Suppose \(a=3.9\).
\(p=\sigma(3.9)\approx 0.98\). Loss \(=-\log(0.98)\approx 0.02\).
Solution B ends at 80 with \(z=0\). If \(a=-3.2\), then \(p\approx 0.04\), small loss.
Lucky broken reasoning with correct final answer still gets \(z=1\). ORM may still output \(p\approx 1\).

11. What comes next

Process reward models fix the localization gap: they score each reasoning step, so a wrong middle step can be caught even when the final answer looks fine.

← Listwise

Up next

Process Reward Model

One score after every step, not only at the end.

Continue →