The multi-objective reward model: one score per quality

A scalar reward says how good an answer is. It cannot say in which way.

The scalar reward model of Chapter 01 is trained on preferences that mix many concerns. A labeller who prefers answer A over answer B may have been thinking about accuracy, tone, length or risk. The model learns one number that blends all of these with whatever weights the labellers happened to use.

That blend is fixed at training time and invisible afterwards. If a score of 0.8 comes back, we cannot tell whether the answer was accurate but rude or polite but wrong. And if we later decide safety should count more, there is no knob to turn. A multi-objective reward model keeps the qualities apart so we can see them and weigh them ourselves.

With four qualities the output might look like this:

Read it coordinate by coordinate: the answer is very likely correct (0.95), helpful (0.88) and well written (0.91), but its safety score (0.60) is low enough to deserve a second look. A single scalar would have averaged that warning away.

The name "multi-head" describes the usual implementation: one network with several output heads. "Multi-objective" describes the use: the scores are later treated as separate objectives to balance.

One Transformer reads the prompt and response. As in earlier chapters, we take the hidden state at the final token,

and attach \(K\) linear heads instead of one:

Why share one backbone instead of training \(K\) separate reward models?

• Shared understanding. Judging correctness, helpfulness and safety all start from the same work: understanding what was asked and what was said. The backbone does that once.
• Cost. One forward pass yields all \(K\) scores. With \(d=4096\), four heads add only \(4\times4097=16{,}388\) parameters to a backbone with billions.
• Data efficiency. Attribute labels are scarce. A head with few labels borrows features learned from the others.

The price is interference. If two objectives want opposite features, the backbone must compromise, and one head can degrade another. When objectives conflict strongly, separate models are sometimes the safer choice; Safe RLHF (Section 9) trains its reward and cost models separately.

Each head needs labels for its own attribute. The total loss is a weighted sum of per-head losses:

Here \(\alpha_k\) balances the heads during training, and \(m_k\in\{0,1\}\) masks heads whose label is missing for this example, so datasets that rate only some attributes can still be mixed. The loss weights \(\alpha_k\) are a training choice. They are not the same as the weights \(\lambda_k\) used later to combine the scores (Section 5).

The per-head loss depends on the label type. Three are common.

Binary labels (correct or not, safe or not): read the head as a logit, \(p_k=\sigma(r_k)\), and use BCE,

Ratings on a scale, such as 0 to 4: regress the head onto the rating,

Per-attribute preferences ("A is safer than B"): apply Bradley–Terry to that head alone,

Best-of-\(N\) selection and PPO both need one number per response. The simplest way to get it is a weighted sum, called linear scalarisation:

The weights \(\lambda_k\) say how much each quality matters. Take three heads with weights \(0.5,\,0.3,\,0.2\) and scores \(\mathbf r=[0.9,\,0.8,\,0.5]\):

Only the ratios of the weights matter for ranking, so fixing their sum to 1 loses nothing. The full pipeline from text to training signal is short:

Two answers to the same prompt, scored on (correctness, helpfulness, safety):

Which is better depends on what we value. With weights \((0.5,0.3,0.2)\):

B wins. With weights \((0.8,0.2,0)\), which ignore safety, \(R_A=0.934\) and \(R_B=0.890\), and A wins. The general rule is the difference

A is ahead only on correctness, and only by 0.08; it is behind on safety by 0.55. Any meaningful safety weight hands the win to B. A scalar reward model would have made this decision silently, using weights nobody chose. The multi-objective model makes the decision explicit, and Equation 11 says exactly where it flips.

Weights only mean what they appear to mean if the heads live on comparable scales. They often do not. A correctness head trained with BCE (Equation 5) and read as a probability may vary by about 0.1 across responses to a prompt. A helpfulness head trained with Bradley–Terry (Equation 7) produces unbounded logits that may vary by 2.0.

With equal weights of 0.5, one typical step in helpfulness moves \(R\) by \(0.5\times2.0=1.0\), while one typical step in correctness moves it by \(0.5\times0.1=0.05\). Helpfulness effectively counts twenty times as much, even though the weights say "equal."

The fix is to standardise each head on a reference set of responses before combining:

where \(\mu_k\) and \(\sigma_k\) are the mean and standard deviation of head \(k\) on that set. Now a weight of 0.5 means the same thing for every head: half a standard deviation of \(R\) per standard deviation of that quality.

Without any weights at all, some comparisons are already settled. Response \(y\) dominates \(y'\) if it is at least as good on every head and strictly better on at least one:

The responses that no other response dominates form the Pareto front. Every response off the front is a bad choice under every reasonable preference. For example \(C=[0.85,0.80,0.90]\) is dominated by \(B=[0.90,0.85,0.95]\), so C never wins for any non-negative weights. A and B are both on the front: neither dominates the other, and choosing between them requires a value judgement.

A fixed weighting is that value judgement. It picks one point on the front, the one that touches the highest line \(\sum_k\lambda_k r_k=\text{const}\). Changing the weights slides the choice along the front.

This is a real limitation. If you want "reasonably good at everything" rather than "excellent at one thing," a weighted sum may never select it. Non-linear combinations, such as maximising the worst normalised head, can reach such points.

A weight treats safety as something that can be traded at a fixed rate: a little more correctness pays for a little less safety. For many safety requirements that is the wrong model. We do not want "less harm is worth more helpfulness"; we want "harm must stay below a limit, and within that limit, be as helpful as possible."

Write the safety head as a cost \(c(x,y)\), higher for more harmful responses, and state the goal as a constrained problem:

The standard tool is the Lagrangian. Introduce a multiplier \(\mu\ge0\) and train the policy on the shaped reward

while adjusting \(\mu\) by gradient ascent on the constraint violation:

Take budget \(d=0.05\), step \(\eta=0.5\), and start at \(\mu=0\). A batch with average cost 0.20 raises \(\mu\) to \(0.5\times0.15=0.075\). The next batch, now safer, averages 0.12 and raises \(\mu\) to \(0.075+0.5\times0.07=0.110\). Once a batch averages 0.04, below budget, \(\mu\) eases back to \(0.110-0.5\times0.01=0.105\). The price of harm rises while the limit is broken and stops rising once it holds.

Safe RLHF follows this recipe: it trains a reward model for helpfulness and a separate cost model for harmlessness on separately collected preferences, then optimises the policy with the Lagrangian method. A cruder variant is a hard gate at selection time: discard any response whose safety score is below a threshold, then rank the rest by the other heads.

One fixed \(\lambda\) is a compromise across all prompts. But "fix this Python bug" should care mostly about correctness, while "write a short poem about autumn" should care more about style and helpfulness. The natural extension is to let the weights depend on the prompt:

Here \(h_x\) is a representation of the prompt alone and \(g_\phi\) is a small gating network. The softmax keeps the weights non-negative and summing to one. The gate sees only the prompt, so the same weights apply to every candidate response for that prompt, and the ranking among candidates stays a fair comparison.

ArmoRM is a published example of this design. Its heads are fitted on multi-attribute absolute ratings; a mixture-of-experts style gate is then trained, with the heads held fixed, so that the combined score agrees with pairwise human preferences under a Bradley–Terry loss. The gate learns from preference data which attributes matter for which kind of prompt.

• The value judgement moves, it does not vanish. Someone still chooses \(\lambda\), \(d\) or the gate's training data; the choice is only made visible.
• Scale sensitivity. Unnormalised heads silently reweight the objectives (Proposition 8.1), and the normalising statistics drift as the policy changes.
• Missing points. Linear weights cannot select Pareto-optimal answers in dents of the front (Proposition 8.2).
• Correlated heads. Heads trained on overlapping data leak into each other; helpfulness ratings, for instance, often correlate with length.
• Costly labels. Per-attribute ratings take more annotator effort, and annotators disagree more on fine-grained attributes than on overall preference.
• More surfaces to hack. Under RL, the policy can push up the easiest head to fool and collect reward through its weight.

Use the answers of Section 6, \(\mathbf r_A=[0.98,0.75,0.40]\) and \(\mathbf r_B=[0.90,0.85,0.95]\). Find which answer wins under several policies and the exact weight at which the winner flips.

Every model so far, scalar or vector, reads the response and emits numbers from a head in a single forward pass. The next chapter changes how the judgment is produced: the reward model writes an evaluation in text, and the score is read from the tokens it generates.