06 · Process Reward Model

1. First look at a PRM

A PRM evaluates every reasoning step, rather than only the final answer.

\[ \text{Problem} \rightarrow s_1 \rightarrow s_2 \rightarrow \cdots \rightarrow s_T \]

It produces a score for each step:

\[ r_1,\; r_2,\; \ldots,\; r_T \]

where $r_t$ indicates whether step $t$ is good or correct.

Example

Problem:

A shirt costs $80 with a 25% discount.

Reasoning:

$25\%$ of $80$ is $20$
Add $20$ to $80$
Final price is $100$

PRM scores:

\[ r_1=0.98,\qquad r_2=0.04,\qquad r_3=0.02 \]

It detects that Step 2 is where the reasoning became wrong. The correct step is:

\[ 80-20=60 \]

Architecture (intuition)

After each reasoning step, the model reads the entire reasoning prefix $x,s_1,\ldots,s_t$ and produces a score:

\[ r_t = f_\theta(x,\, s_{\leq t}) \]

$x$: problem
$s_t$: current reasoning step
$s_{\leq t}$: all steps up to $t$
$r_t$: predicted step quality

Usually:

\[ r_t = \sigma(w^\top h_t + b) \]

where $h_t$ is the hidden state at the end of step $t$.

Training (intuition)

Each step receives a label:

\[ z_t= \begin{cases} 1, & \text{correct step}\\ 0, & \text{incorrect step} \end{cases} \]

Loss:

\[ \mathcal{L} = -\sum_{t=1}^{T} \big[ z_t\log r_t + (1-z_t)\log(1-r_t) \big] \]

So the PRM learns to give high scores to correct steps and low scores to incorrect ones.

How it improves reasoning

During search, suppose the model generates three possible next steps:

\[ s_A:0.92,\qquad s_B:0.17,\qquad s_C:0.75 \]

The system continues with $A$ or $C$ and removes $B$. Thus a PRM can:

detect mistakes early
guide beam search or tree search
select better complete solutions
provide dense rewards during RL

PRM versus ORM

ORM	PRM
Scores complete solution	Scores every step
One final reward	Multiple intermediate rewards
Easier to label	Harder to label
Cannot locate errors	Can locate errors
Sparse feedback	Dense feedback

Core difference \[ \boxed{\text{ORM: Did the solution succeed?}} \qquad \boxed{\text{PRM: Was each reasoning step good?}} \]

The rest of this chapter makes the same ideas rigorous: exact definitions, what $q_t$ really means, training details, aggregation, search, and limits.

2. Exact mathematical definition

We now make the same idea precise. We focus first on the classical discriminative PRM. Generative PRMs come later.

Suppose a problem is $x$ and the model generates a solution with $T$ reasoning steps:

\[ s_1,\; s_2,\; \ldots,\; s_T \]

A PRM evaluates every reasoning prefix:

(1) \[ (x,s_1),\quad (x,s_1,s_2),\quad \ldots,\quad (x,s_1,\ldots,s_T) \]

At step $t$, it predicts:

(2) \[ q_t = f_\theta(x,\, s_{\leq t}) \]

$s_{\leq t}=(s_1,\ldots,s_t)$: all steps up to $t$
$f_\theta$: PRM with parameters $\theta$
$q_t\in[0,1]$: predicted quality of step $t$ (or of the prefix)

So the output is a vector, not one final reward:

\[ [q_1,\, q_2,\, \ldots,\, q_T] \]

3. What exactly does $q_t$ mean?

This is the most important subtlety. Two different meanings are both often called PRM scores.

A. Local-correctness PRM

(3) \[ q_t = P(s_t\text{ is correct}\mid x,\, s_{\lt t}) \]

It asks: given all previous reasoning, is the current step logically correct?

Example: $q_3=0.08$ means Step 3 is probably incorrect.

B. Prefix-value PRM

(4) \[ V_t = P(\text{eventual correct answer}\mid x,\, s_{\leq t}) \]

It asks: from this partial solution, how likely are we to eventually solve the problem?

A step may be technically correct but have low value. Example: “Let us check one billion possibilities manually.” That may be valid, yet $V_t\approx 0$ because the strategy is unlikely to finish.

Key distinction \[ \boxed{\text{step correctness}\neq\text{future success probability}} \]

In practice

Classical human-supervised PRMs such as PRM800K primarily label whether each step is correct and reasonable. Automatically labelled methods such as Math-Shepherd often estimate whether a prefix can lead to a correct final answer through sampled continuations, which is closer to a value estimate.

4. PRM architecture

Suppose the solution is formatted with explicit step endings:

Problem: ...

Step 1: ...
<STEP>

Step 2: ...
<STEP>

Step 3: ...
<STEP>

The transformer produces a hidden vector at every token. At the end of Step $t$, we take the hidden state $h_t$. This vector represents $x+s_1+\cdots+s_t$.

The reward head converts it into logits:

(5) \[ \ell_t = W h_t + b \]

For binary labels we may have two logits $\ell_t^{+}$ and $\ell_t^{-}$, then:

(6) \[ q_t = P(\text{positive}) = \frac{e^{\ell_t^{+}}}{e^{\ell_t^{+}}+e^{\ell_t^{-}}} \]

Problem + steps

↓

Transformer

↓

Hidden state at each step ending

↓

Classification head

↓

$q_1, q_2, \ldots, q_T$

PRM800K detail

The original PRM800K work trained the model to predict a correctness token after the final token of every step; one forward pass could score the entire solution. Human annotators used positive, negative, and neutral labels.

5. Training data

A PRM dataset contains solutions with a label for every step:

(7) \[ \mathcal{D} = \Big\{ \big( x^{(i)},\, s^{(i)}_1,\ldots,s^{(i)}_{T_i},\, y^{(i)}_1,\ldots,y^{(i)}_{T_i} \big) \Big\}_{i=1}^{N} \]

$N$: number of solutions
$y_t=1$: step is correct
$y_t=0$: step is incorrect

Example:

Step 1 → positive
Step 2 → positive
Step 3 → negative

\[ [y_1,y_2,y_3]=[1,1,0] \]

PRM800K contained roughly 800,000 human step labels across about 75,000 solutions, with annotators marking steps positive, negative, or neutral.

6. Training loss

For binary labels, the loss for Step $t$ is ordinary BCE:

(8) \[ \mathcal{L}_t = -\big[ y_t\log q_t + (1-y_t)\log(1-q_t) \big] \]

The total loss averages over steps:

PRM loss \[ \boxed{ \mathcal{L}_{\mathrm{PRM}} = -\frac{1}{T} \sum_{t=1}^{T} \big[ y_t\log q_t + (1-y_t)\log(1-q_t) \big] } \]

This is simply binary cross-entropy applied at every step.

7. Complete mathematical example

Problem:

An item costs $120. It receives a 25% discount, then 10% VAT is applied to the discounted price.

Candidate reasoning:

Step 1 $25\%$ of $120 = 0.25\times 120 = 30$

Correct: $y_1=1$

Step 2 $120-30=90$

Correct: $y_2=1$

Step 3 $10\%$ VAT $=0.10\times 120=12$

Incorrect: VAT should use $90$, not $120$. So $y_3=0$.

Step 4 $90+12=102$

Depends on the previous error, so also invalid: $y_4=0$.

True answer:

\[ 90 + 0.10(90) = 99 \]

PRM prediction

Suppose the PRM outputs:

\[ [q_1,q_2,q_3]=[0.97,\,0.92,\,0.15] \]

Masking after first error

The training scheme may stop supervising after the first incorrect step, so Step 4 can be masked out. The original PRM800K setup deliberately supervised only up to the first incorrect step in many comparisons.

The loss is:

(9) \[ \mathcal{L} = -\frac{1}{3} \big[ \log(0.97) + \log(0.92) + \log(1-0.15) \big] = -\frac{1}{3} \big[ \log(0.97) + \log(0.92) + \log(0.85) \big] \]

Approximately $\mathcal{L}\approx 0.092$. This is small because:

correct Steps 1 and 2 received high probabilities
incorrect Step 3 received a low probability

8. Automatically creating PRM labels

Human annotation is expensive. An automatic method can use rollouts. Suppose the prefix is $z_t=(x,s_1,\ldots,s_t)$. Generate $M$ possible continuations $c^{(1)},\ldots,c^{(M)}$. For each continuation, check the final answer:

(10) \[ R^{(m)} = \begin{cases} 1, & \text{correct final answer}\\ 0, & \text{incorrect final answer} \end{cases} \]

Then estimate the prefix value:

(11) \[ \hat{V}_t = \frac{1}{M}\sum_{m=1}^{M} R^{(m)} \]

Example: five continuations give labels $[1,1,0,1,0]$, so

\[ \hat{V}_t = \frac{1+1+0+1+0}{5} = 0.6 \]

Interpretation: sixty percent of continuations from this prefix reached the correct answer. This does not prove the current step is correct. A later continuation may repair an earlier error, or reach the right answer accidentally.

Math-Shepherd and after

Math-Shepherd introduced this rollout-style automatic supervision. Later studies found that Monte Carlo labels can be less reliable than high-quality human or judge-based process labels because they measure recoverability rather than pure local correctness. (arXiv:2312.08935)

9. How do we score a complete solution?

A PRM gives multiple scores $q_1,\ldots,q_T$. We need one score to compare complete solutions.

Product aggregation

(12) \[ S_{\mathrm{product}} = \prod_{t=1}^{T} q_t \]

This approximates the probability that every step is correct. The original PRM800K experiments used the product of step-correctness probabilities for complete-solution ranking.

Solution A

[0.98, 0.96, 0.15]

\[ S_A = 0.98\times 0.96\times 0.15 \approx 0.141 \]

Solution B

[0.88, 0.85, 0.82]

\[ S_B = 0.88\times 0.85\times 0.82 \approx 0.613 \]

Although A begins strongly, its final step is suspicious. The product selects B.

Minimum aggregation

(13) \[ S_{\min}=\min_t q_t \]

This asks: what is the weakest step? For Solution A, $S_{\min}=0.15$. Useful because one major logical error can invalidate an entire proof.

Length problem

Products penalize long solutions. Ten steps each scoring $0.9$:

\[ 0.9^{10}\approx 0.349 \qquad\text{vs}\qquad 0.9^{3}\approx 0.729 \]

So an alternative is average log probability:

(14) \[ S_{\log} = \frac{1}{T}\sum_{t=1}^{T}\log q_t \]

or its geometric mean:

(15) \[ S_{\mathrm{geo}} = \Big(\prod_{t=1}^{T} q_t\Big)^{1/T} \]

10. PRM-guided search

Suppose the current reasoning prefix is $z_t$. The generator proposes $K$ possible next steps $a_1,\ldots,a_K$. The PRM evaluates $q_j=f_\theta(z_t,a_j)$.

Example: $q_1=0.91$, $q_2=0.18$, $q_3=0.76$. Keep Steps 1 and 3; reject Step 2.

A beam-search version maintains $B$ promising prefixes:

1. Generate K next steps from every prefix
2. Score each new step with the PRM
3. Add score to the cumulative branch score
4. Keep the best B branches
5. Repeat until final answers are produced

For a local-correctness PRM, a cumulative branch score can be:

(16) \[ C_t = \sum_{i=1}^{t}\log q_i = \log\prod_{i=1}^{t} q_i \]

PRMs are commonly evaluated through best-of-$N$ reranking and guided search, where they select promising solutions or branches from multiple generated candidates.

11. PRM as an RL reward

An ORM provides one reward $R_T$. A PRM can provide dense rewards $r_1,\ldots,r_T$.

For example $r_t=\log q_t$. The policy is encouraged to generate steps with higher PRM scores.

For a value-style PRM, a cleaner shaping reward is:

(17) \[ r_t = V_t - V_{t-1} \]

Then the telescoping sum is:

\[ \sum_{t=1}^{T} r_t = \sum_{t=1}^{T}(V_t-V_{t-1}) = V_T - V_0 \]

So the model is rewarded when a step increases the estimated probability of final success. Math-Shepherd demonstrated both PRM-based reranking and stepwise PPO training, although PRM-based RL remains vulnerable to inaccurate rewards and reward hacking. (arXiv:2312.08935)

12. What PRMs actually improve

Error localization

Instead of $R_{\mathrm{final}}=0$, we obtain $[0.96,\,0.91,\,0.07,\,0.03]$. The first likely error is Step 3.

Better credit assignment

The model knows which steps were useful and which caused failure.

Better search: incorrect branches can be removed before spending computation completing them.

Process supervision was shown to outperform outcome supervision in the original PRM800K best-of-$N$ mathematical reasoning experiments, especially as the number of candidate solutions increased.

13. Critical limitations

A PRM score is not proof of correctness:

\[ q_t=0.99 \;\not\Rightarrow\; s_t\text{ is certainly correct} \]

#	Problem	Why it hurts
1	Step granularity	One “step” may contain several logical operations.
2	Error propagation	After the first error, later steps are hard to label.
3	Recovery ambiguity	A model may correct an earlier mistake.
4	Style bias	Polished or verbose reasoning may look more convincing.
5	Outcome leakage	The PRM may judge mostly from the final answer.
6	Distribution shift	A math PRM may fail on harder or unfamiliar problems.
7	Reward hacking	A generator may learn reasoning that looks good to the verifier.

Benchmarks

ProcessBench and PRMBench found substantial weaknesses in current PRMs, particularly in generalizing to difficult problems and identifying fine-grained reasoning errors. (arXiv:2412.06559)

14. Current evolution: generative PRMs

Classical PRM

step → score

Generative PRM

step → verification reasoning → verdict

The step divides both sides by 4, but the coefficient is 3.
Therefore, this transformation is invalid.

Verdict: Incorrect

Systems such as THINKPRM and GenPRM generate verification reasoning before assigning a judgment, allowing verifier computation to scale at inference time. They are more computationally expensive than direct classifiers but reported stronger process evaluation in their experiments. (arXiv:2504.16828)

15. Final understanding

Remember \[ \boxed{ \text{PRM: } (x,s_1,\ldots,s_t) \rightarrow \text{quality of the reasoning process at step }t } \]

A PRM is not simply an ORM repeated several times. Its central purpose is to learn where reasoning remains valid, where it breaks, and which partial paths are worth continuing.

16. Laboratory · step scores and aggregation

Three-step PRM scores

Product · min · geo · loss

$q_1$ 0.97 $q_2$ 0.92 $q_3$ 0.15 Labels $y$ for loss (1=correct, 0=wrong on step 3) y = [1, 1, 0]

$S_{\mathrm{product}}$-

$S_{\min}$-

$S_{\mathrm{geo}}$-

$\mathcal{L}$-

Drag step scores to compare aggregations.

17. Worked loss, step by step

Labels for the discount path: $y=[1,1,0]$ (mask after first error).
PRM scores: $q=[0.97,0.92,0.15]$.
Term 1: $-\log(0.97)\approx 0.0305$.
Term 2: $-\log(0.92)\approx 0.0834$.
Term 3: $-\log(0.85)\approx 0.1625$ because $1-q_3=0.85$.
Average: $\mathcal{L}\approx(0.0305+0.0834+0.1625)/3\approx 0.092$.
Product score $0.97\times 0.92\times 0.15\approx 0.134$. Weakest step is $0.15$.

18. What comes next

Next chapters move along other axes of the taxonomy: value models that estimate future success of a prefix, multi-objective heads, and generative or critique-then-score verifiers.

Up next

Value / future-success model

Score a prefix by how likely it is to finish correctly.

Chapter 07 · soon

Process reward model (PRM)