Architecture 03 · Joint comparison
Direct pairwise comparator
1. Idea
Scalar RMs score A and B separately, then compare. A direct comparator asks the joint question: is A better than B?
2. Architecture
3. Example
Prompt: explain gradient descent simply.
A: Repeatedly move parameters to reduce loss.
B: Randomly change parameters until accuracy improves.
Comparator may output \(P(A\succ B)=0.96\).
4. Training
5. vs scalar pairwise RM
Scalar RM
\(r_A\), \(r_B\) independent. \(P=\sigma(r_A-r_B)\).
has standalone scores
Direct comparator
\(P=\sigma(g(x,A,B))\). Can depend on interactions. May have no \(r_A\).
joint judgment
6. Why direct can help
Absolute clarity is hard. Relative clarity is easier: "Compared with B, is A clearer?" Humans annotate that way too. The model can notice specific contrasts (missing detail, better format, same correctness but different clarity).
7. Order and bias
Ideally \(P(A\succ B)+P(B\succ A)=1\). Position bias can break this. Train and evaluate with both orders, then average.
8. Transitivity
Scalar scores force transitive rankings. Direct comparators can create cycles \(A\succ B\succ C\succ A\). That can match messy human taste. It makes a single global ranking harder.
9. Selecting among many
No \(\arg\max r_i\). Use a tournament, round-robin (\(\frac{N(N-1)}{2}\) pairs), or win-rate scores. For \(N=10\), round-robin needs 45 comparisons.
10. LLM judges
Many "LLM-as-a-judge" prompts are generative direct comparators: they read A and B and output A, B, or Tie (often with a critique first).
11. Weaknesses
- Expensive for large \(N\).
- No natural scalar for PPO.
- Position, length, and style biases.
- Possible preference cycles.
For RL, people often distill the comparator into a scalar RM, or compare against a fixed baseline.