The value model: future success of a prefix

A PRM judges the step. A value model judges the position the step leaves you in.

In Chapter 06 the process reward model scored each step \(s_t\) by asking whether it is correct given what came before. That is a local question. When we use a model to guide generation, the question we really care about is global: if we continue from here, will we finish correctly?

A value model answers that global question directly. It looks at a reasoning prefix and returns a number between 0 and 1, read as the chance that a continuation from this prefix ends in a correct answer.

Write the problem as \(x\) and the first \(t\) reasoning steps as \(s_1,\ldots,s_t\). The prefix, or reasoning state, is

Architecturally it is the same Transformer-plus-head as earlier chapters. The backbone reads the prefix, we take the hidden state at the last token, and a linear head followed by a sigmoid turns it into a probability:

Nothing in the architecture says "value." The meaning comes entirely from the training target, which is the subject of Section 5.

The two often agree. A wrong step usually makes success unlikely, so both scores drop together. They come apart in two places: a correct step that leads somewhere hard, and a slightly sloppy step from which recovery is still easy.

Problem: find \(23\times 17\). Two candidate first steps are generated.

Branch A. Split the multiplier:

Branch B. "Add 23 to itself seventeen times." This is also mathematically valid.

A step-level PRM checks local correctness, so both steps score high:

A value model asks how often each branch actually finishes correctly. Branch B needs sixteen additions, and every one is a chance to slip, so its value is lower:

No human can label "probability of eventual success" directly. We estimate it by simulation. From the prefix \(z_t\), sample \(M\) full continuations with a completion policy \(\pi\), check each final answer, and record

With \(M=5\) and outcomes \([1,1,0,1,0]\), the estimate is \(\hat V=\tfrac{3}{5}=0.6\). Correctness only has to be checkable at the end (an answer key, a unit test, a proof checker), which is far cheaper than labelling every step by hand.

The estimate is noisy. Each \(R_m\) is a Bernoulli draw with mean \(V\), so

At \(V=0.6\) and \(M=5\) the standard error is about \(0.22\): the label could easily have come out as 0.4 or 0.8. Quadrupling \(M\) only halves the error. Rollout budget is the main cost of value labels.

Collect many prefixes (every step boundary of many sampled solutions), attach a rollout estimate to each, and regress:

Because \(\hat V\) is a probability, binary cross-entropy with a soft target is an equally natural choice, and it keeps gradients informative near 0 and 1:

Treat reasoning as an episodic MDP: the state is the prefix \(z_t\), an action appends a step, and the only reward is 1 at the end if the answer is correct. With no discounting, the state-value function of the completion policy \(\pi\) is

So the value model is exactly \(V^\pi\) from the value functions chapter of the RL notes, and the rollout estimate of Section 5 is Monte-Carlo policy evaluation. Two consequences follow.

• Values depend on the policy. A strong solver and a weak solver assign different values to the same prefix. A value model is always the value under some completion policy, usually the model being trained.
• Bootstrapping is possible. Instead of full rollouts, train toward the next state's prediction, \(V(z_t)\leftarrow V(z_{t+1})\), as in TD learning. This is cheaper but inherits TD's bias.

The main use of a value model is deciding which partial solutions to keep exploring. Suppose the solver proposes three next steps and the value model scores the resulting prefixes:

With a beam of width 2 we continue A and C and prune B. Repeating this at every step is value-guided beam search. The same scores can serve as leaf evaluations in tree search such as MCTS, where they replace expensive full rollouts.

A value model also gives a principled per-step reward. The step \(s_t\) moved us from \(z_{t-1}\) to \(z_t\); how much did it help?

This is an advantage in the RL sense. A step that raises the chance of success earns a positive reward, a step that wastes effort earns about zero, and a step that damages the solution earns a negative one. Rewarding progress rather than raw correctness avoids paying the solver for correct but useless steps, like Branch B in Section 4.

• Rollout cost. Each label needs \(M\) full generations; good labels need large \(M\).
• Noisy targets. With small \(M\), labels scatter widely around the true value (Equation 7).
• Policy dependence. As the solver improves, old value labels go stale and must be refreshed.
• Checkable answers only. Rollout labels need an automatic success test, which open-ended writing lacks.
• Exploitable in search. Search pushes toward prefixes the value model overrates, so its errors are amplified, not averaged out.

A solver is at prefix \(z_1\) with \(V(z_1)=0.50\). It proposes three next steps. Five rollouts from each extended prefix give the outcomes below. Choose two branches to keep and compute the progress reward of each.

The value model completes the "what is judged" axis: whole answers (ORM), single steps (PRM), and future success of a prefix (value). The next chapters change a different axis, what the model outputs, starting with reward models that return several scores at once.