The conditional reward model: good for whom?

A standard reward model assumes one universal ordering of answers. Real users disagree.

The scalar reward model of Chapter 01 is a function of the prompt and the answer only:

Take the prompt "Explain transformers." and two answers: \(y_1\) is a short plain-language explanation, \(y_2\) is a long mathematical one with attention equations. User A wants concise answers. User B wants the mathematics. A reward that knows who is asking would score them like this:

The answers are the same. The rewards differ because the context differs, \(c_A\neq c_B\). Equation (1) has no slot for that difference, so it must give both users the same pair of numbers. At least one of them will be served badly.

Preference datasets mix labels from many annotators. If who labelled a pair is not recorded, the annotator's taste is hidden context: it shapes the label, but the model never sees it. Suppose a fraction \(\pi_g\) of comparisons comes from group \(g\), and that group prefers \(y_2\) over \(y_1\) with probability \(P_g(y_2\succ y_1)\). The pooled data then shows

A single Bradley-Terry model fitted to plenty of these comparisons matches the pooled probability, so its margin is

Now take two equal groups with opposite, strong views: group A prefers \(y_1\) with probability 0.9, group B prefers \(y_2\) with probability 0.9. The pooled probability is \(0.5\cdot 0.1+0.5\cdot 0.9=0.5\), so the single model's margin is exactly 0. It reports that the two answers are equally good. No annotator holds that view.

With more than two answers, a majority can erase a minority entirely. Add a medium-length answer \(y_3\). Group A (60% of labels) ranks \(y_1\succ y_3\succ y_2\); group B (40%) ranks \(y_2\succ y_3\succ y_1\). Give each answer 2 points for first place, 1 for second, 0 for third, weighted by group share (a Borda count):

The pooled ordering is \(y_1\succ y_3\succ y_2\), which is exactly group A's. Siththaranjan et al. show that standard preference learning under hidden context aggregates in this Borda-like way. A policy optimised against that reward gives \(y_1\) to everyone, and 40% of users receive their least favourite answer.

The fix is to make the hidden context visible. Give the reward model a third input \(c\) that describes the user, task or criterion the answer is judged against.

The question changes from "Is this answer good?" to "Is this answer good for this user, task or condition?" If the model ignores \(c\), it collapses back to Equation (1). So the universal reward model is the special case of a constant context.

The definition says nothing about the form of \(c\). Four kinds are common.

• User profile. An identifier, a learned user embedding, or stated preferences such as "prefers short answers" or "is a beginner".
• Written rubric or system prompt. A natural-language instruction for how to judge: "Reward answers that cite sources; penalise speculation." The same model can then be reused with new rubrics.
• Task or domain. A label such as code, medical, or creative writing. A terse answer is good code review and poor storytelling.
• Attribute targets. A vector of desired levels, for example helpfulness 4, verbosity 1, on a 0 to 4 scale.

Attribute targets come from attribute-conditioned training such as SteerLM. There an attribute model rates responses on dimensions like helpfulness and verbosity, and the language model is fine-tuned to produce a response given the target values it should have. Read as a reward, one simple choice is to score how close the predicted attributes \(a_\phi(x,y)\) are to the target \(c\):

A user who asks for verbosity 1 then rewards a short answer, and a user who asks for verbosity 4 rewards a long one, with the same attribute model underneath.

Encode the prompt and response with the Transformer as before, and the context as a vector \(e_c\):

The simplest idea is to concatenate them, \([h_T;e_c]\), and apply a linear head. This does not work, and it is worth seeing why:

The context has to interact with the response. Three ways do this.

• A nonlinear head on the concatenation, such as a small MLP: \(r=g_\theta(h_T,e_c)\).
• A context-dependent weight, so the context decides which features of the response matter:

• Context in the text. Write \(c\) into the input before the response, \(r_\theta(c\oplus x,y)\), where \(\oplus\) joins strings. Attention then mixes context and response at every layer.

Each preference record now carries its context: \((x,c,y_w,y_l)\). The loss is the Bradley-Terry loss of Chapter 01, evaluated with the record's own context:

The model learns "for this context, which answer is preferred?" The same pair \((y_1,y_2)\) can appear twice with opposite labels and different contexts, and that is no longer label noise. It is signal about how \(c\) changes preference.

Often we do not have a profile, only a few comparisons the user has made. We can treat the user's taste as a latent vector \(z\) and infer it from those labels. Let \(D_u=\{(x_i,y_{w,i},y_{l,i})\}_{i=1}^{n}\) be user \(u\)'s labelled pairs. An encoder maps them to a distribution over \(z\):

Encoder and reward model are trained together in the style of a variational autoencoder. The reward must explain the user's held-out comparisons, and a KL term keeps \(q_\psi\) close to a prior \(p(z)\):

This is the idea behind variational preference learning (Poddar et al.). At test time a new user supplies a handful of comparisons, the encoder infers \(z\), and the reward model is personalized with no further training. With no comparisons the posterior falls back to the prior, which acts like the average user.

A different route avoids one big conditional model. Train separate networks for a few basic preferences (concise, detailed, formal, playful), all fine-tuned from the same starting weights, and combine them per user. Two ways to combine:

The first mixes outputs, which costs one forward pass per basic model. The second mixes weights, giving a single network. Rewarded soups (Ramé et al.) interpolate the weights of networks fine-tuned on different rewards; personalized soups (Jang et al.) merge policies trained for separate preference dimensions after training. Weight interpolation works well in practice when the models share a pretrained starting point, but it is an empirical finding, not a guarantee.

• Few labels per user. Most users label a few pairs or none. A per-user model with millions of parameters overfits; the personal part must be small and the shared part large.
• Cold start. A new user has no labels. The model must fall back to a sensible default, such as the prior in Section 7 or a written profile.
• Privacy. Profiles and labelled histories are personal data. Storing them, and training on them, raises consent and leakage questions that a universal reward model does not.

Evaluation must also change. A single held-out accuracy can hide a model that serves the majority and ignores everyone else. Useful checks:

• Per-group accuracy, and the worst group's accuracy, not only the average.
• Context-swap test. Score each test pair with a wrong context. If accuracy barely drops, the model is not using \(c\).
• Accuracy against labels from new users, to measure cold-start behaviour.

Personalization rewards what a user prefers. Users often prefer hearing that they are right. A reward model conditioned on "this user believes X" can learn to reward answers that agree with X, whether or not X is true. Preference-trained models are already known to drift toward such sycophancy; conditioning on the user makes the target easier to hit.

One safeguard is to split the reward so that context only touches the parts that should vary between people:

The core term judges accuracy and safety and does not see \(c\). The preference term judges length, tone and format. Because the preference term is bounded by \(B\), it can change a pairwise margin by at most \(2B\). If one answer leads on the core term by more than \(2B\), no context can reverse that.

• Sparse personal data. Each user contributes few labels, so personal parameters are poorly estimated.
• Context must be available. At RL or inference time the right \(c\) must be known and accurate; a wrong profile gives a confidently wrong reward.
• Silent collapse. If shared signal dominates training, the model can learn to ignore \(c\) and still look accurate on average.
• Unseen contexts. A rubric or user type absent from training may be read poorly, especially with learned embeddings.
• Sycophancy and echo chambers. Rewarding what a user wants to hear can reward inaccuracy and reinforce existing views.
• Privacy cost. Personalization requires collecting and storing information about people.

Use the rewards of Equation (2). Half the labels come from user A, half from user B, each labelling according to their own Bradley-Terry probabilities. Compare the training loss of the conditional model with that of the best single model.

A conditional reward model admits that "good" depends on who is asking. The next chapter admits something else: even for a fixed context, a reward model can be unsure. An ensemble of reward models reports both a score and how far to trust it.