Listwise reward model

Pointwise: one answer. Pairwise: two. Listwise: a list.

Given \(y_1,\ldots,y_K\), predict an order such as \(y_3\succ y_1\succ y_4\succ y_2\).

Pairwise labels can cycle: \(A>B\), \(B>C\), \(C>A\). A listwise model sees the set and learns global structure. One human ranking of four answers implies up to six pairwise relations.

Often each candidate still gets a scalar. The difference is the training objective over the whole list. Example: \(s_C=2.6\), \(s_A=1.8\), \(s_D=1.1\), \(s_B=0.3\) implies \(C>A>D>B\).

For \(s_A=3\), \(s_B=2\), \(s_C=1\): roughly \(P(A)=0.665\), \(P(B)=0.245\), \(P(C)=0.090\).

First factor: pick A among all three. Second: pick B among the remainder. If the model ranks B first when A should win, loss is large.

Candidates can share one context so the model sees interactions. Or an LLM can generate C > A > B with a short critique. That generative listwise judge is powerful and expensive. People often distill it into a scalar RM.

• Better global ranking signal.
• Richer annotation per human effort.
• Hard negatives among near-tied good answers.
• Aligned with best-of-N selection.

• Context cost scales like \(K\times L\) tokens.
• Position bias in the list.
• Middle ranks are noisy for large \(K\).
• Forced total orders when ties are real.
• Set dependence: quality looks different beside weak vs strong peers.

Reranking, dataset filtering, synthetic preference data, distillation into scalar RMs, and RL when scalar scores \(s_i\) are available.

Next we switch axes: from ranking form to evaluation granularity. Outcome reward models score the finished solution. Process reward models score the steps.