The implicit reward model: a reward hidden in the policy

Classical RLHF learns a reward, then optimises a policy against it. DPO skips the first stage.

This is not a heuristic shortcut. Under the same assumptions as classical RLHF, the two routes target the same optimal policy. Sections 3 to 6 show why.

Let \(\pi_\theta\) be the policy being trained and \(\pi_{\text{ref}}\) a frozen reference model, usually the supervised fine-tuned model that training starts from. Fix a scale \(\beta>0\).

The reward is positive when the trained policy has made \(y\) more likely than the reference did, negative when it has made \(y\) less likely, and zero before training starts, when \(\pi_\theta=\pi_{\text{ref}}\). It measures how far training moved the probability of \(y\), not how probable \(y\) is.

For example, with \(\beta=0.1\), a response whose log-ratio is \(2\) gets \(\hat r=0.2\), and one whose log-ratio is \(0.5\) gets \(\hat r=0.05\). The first is preferred.

Start from the objective that PPO-based RLHF optimises. Given a reward \(r(x,y)\), maximise expected reward while staying close to the reference model:

The KL term stops the policy from drifting into text the reward model has never seen and can be fooled by. Larger \(\beta\) means a tighter leash. The objective splits over prompts, so from here on we fix one prompt \(x\).

Define the normaliser and a candidate policy:

Write the KL as an expectation and collect terms. Since \(r=\beta\log\big(\pi^*Z/\pi_{\text{ref}}\big)\) by Equation (2),

The last term does not depend on \(\pi\). The KL term is never negative and is zero only when \(\pi=\pi^*\).

This closed form is of no direct use for sampling: \(Z(x)\) sums over every possible response, which is astronomically many sequences. Its value is what it says about the relation between \(r\) and \(\pi^*\).

Take logs in Equation (2) and solve for \(r\):

Read in this direction, the equation says that a policy determines a reward. Any policy \(\pi_\theta\) is the optimal policy of Equation (1) for the reward \(\beta\log(\pi_\theta/\pi_{\text{ref}})+\beta\log Z(x)\). The first term is the implicit reward of Definition 11.1. The second is a problem: it is intractable.

Preference data only ever compares two responses to the same prompt. Under the Bradley-Terry model of the scalar chapter, the probability of a preference depends only on the reward difference. Substitute Equation (4) for both rewards:

The two copies of \(\beta\log Z(x)\) are equal, because both responses share the prompt \(x\), and they subtract to zero. What remains involves only the policy and the reference, both of which we can evaluate.

Replace the unknown \(\pi^*\) in Equation (5) by the trainable \(\pi_\theta\) and take the negative log-likelihood of the observed preferences. Writing \(\hat r_w=\hat r_\theta(x,y_w)\) and \(\hat r_l=\hat r_\theta(x,y_l)\):

This is the Bradley-Terry loss of the scalar chapter, with the reward network replaced by the implicit reward. Fitting a reward model and fitting the policy have become the same act.

With the numbers from Section 2, \(\hat r_w=0.2\) and \(\hat r_l=0.05\): the margin is \(0.15\), \(\sigma(0.15)\approx0.537\), and the loss is about \(0.621\), only a little below the starting value \(\log 2\approx0.693\).

Use \(\frac{d}{dz}\big[-\log\sigma(z)\big]=-\sigma(-z)\), and note that \(\pi_{\text{ref}}\) is frozen, so \(\nabla_\theta\hat r=\beta\nabla_\theta\log\pi_\theta\):

The bracket is the direction: make the winner more likely and the loser less likely. The weight sets the step size. It is the probability the implicit reward model currently assigns to the wrong ordering.

Without the weight, every pair would push equally hard, and pairs that are already separated would keep being pushed apart. The weight concentrates learning on the pairs the model still gets wrong.

Once trained, \(\pi_\theta\) and \(\pi_{\text{ref}}\) together form a reward model. To pick the best of \(n\) candidate responses, score each by its implicit reward:

Scoring costs two forward passes per response, one through each model, and no generation. Benchmarks of reward models such as RewardBench evaluate DPO-trained models in exactly this way, next to explicit classifiers.

Two cautions carry over from Section 6. The score is defined only up to the per-prompt constant \(\beta\log Z(x)\), so comparing implicit rewards across different prompts is not meaningful. And the score is a sum over tokens, so responses of very different lengths can differ in score partly because of length.

A language model factorises over tokens, so the log-ratio does too. For a response \(y=(y_1,\ldots,y_T)\):

Each term is a per-token credit: how much training raised or lowered the probability of that token in its context. Summing up to position \(t\) gives a running total for the prefix,

where \(t_k\) is the last token of reasoning step \(k\) and \(\rho_k\) is the reward of that step. This is an implicit PRM: step-level rewards with no step-level labels. The model is trained only on whole-response signals, either preference pairs with Equation (6) or outcome labels \(o\in\{0,1\}\) with a cross-entropy loss on the implicit reward:

• Needs the reference model. Every score needs \(\pi_{\text{ref}}\) as well as \(\pi_\theta\): two models in memory, or cached reference log-probabilities for fixed data.
• Offline data only. The pairs were usually sampled from another model. The policy never sees its own outputs judged, unlike PPO. Iterative or online DPO samples fresh pairs from the current policy and labels them with a judge to close this gap.
• Poor calibration off-distribution. The implicit reward is shaped only on the training pairs. On responses unlike them, the log-ratio reflects whatever incidental drift fine-tuning caused, and can rank such responses badly.
• Overfitting to confident labels. If a pair is always preferred, the Bradley-Terry optimum wants an infinite margin. The loss keeps pushing the log-ratios apart, and the KL leash of Equation (1) is never enforced directly.
• Both likelihoods can fall. The loss depends only on the margin, so it can be lowered by making both responses less likely, as long as the loser falls faster (see the note below).

For one prompt, the policy and the reference give these sequence log-probabilities: \(\log\pi_\theta(y_w)=-42.0\), \(\log\pi_{\text{ref}}(y_w)=-45.0\), \(\log\pi_\theta(y_l)=-40.0\), \(\log\pi_{\text{ref}}(y_l)=-38.0\). Take \(\beta=0.1\). Compute the implicit rewards, the loss and the gradient weight.

So far every reward model, explicit or implicit, has scored responses for an average user. The next chapter conditions the reward on who is asking, or on a stated preference, so that the same response can earn different rewards in different contexts.