Architecture 02 · Pointwise classification
Pointwise binary verifier
1. Idea
Previously we learned from \(A \succ B\). Now we learn from a single pair \((x,y)\) with label \(z\in\{0,1\}\).
\(z=1\) means acceptable. \(z=0\) means not.
2. Architecture
3. Example
Question: what is \(12\times 8\)?
Candidate A: \(96\) with \(z=1\). Candidate B: \(86\) with \(z=0\).
Model might predict \(p_A=0.97\), \(p_B=0.08\). No pairing required.
4. Loss
If \(z=1\), loss \(=-\log p\) (push \(p\to 1\)). If \(z=0\), loss \(=-\log(1-p)\) (push \(p\to 0\)).
Example: correct answer with \(p=0.9\) gives loss \(\approx 0.105\). Same answer with \(p=0.1\) gives loss \(\approx 2.3\).
\(\mathcal{L}=-[z\log p+(1-z)\log(1-p)]\)
5. vs pairwise scalar RM
| Pairwise scalar | Pointwise verifier | |
|---|---|---|
| Training input | Two candidates | One candidate |
| Label | Winner / loser | Correct / incorrect |
| Meaning | Relative preference | Class probability |
| Needs absolute criterion? | Not always | Yes |
| Best for | Open-ended quality | Math, code, checks |
Pairwise can prefer "bad" over "worse." Pointwise asks whether the answer meets a bar.
6. Where labels come from
- Humans: correct/incorrect, safe/unsafe.
- Programs: final answer match, unit tests, task success.
Programmatic labels are cheap and objective, but often only check the final result.
7. ORM or PRM?
Pointwise is an output style, not a location. Apply it to a full solution (ORM) or after every step (PRM):
8. Uses
- Best-of-N: pick \(\arg\max_i p(x,y_i)\).
- Filtering: reject when \(p<\tau\) (for example \(\tau=0.8\)).
- Search: keep high-\(p\) branches.
- RL: reward \(p\) or the raw logit \(a\) (logits saturate less).
9. Weaknesses
- Needs an explicit criterion (hard for creative writing).
- Correct final answer can hide bad reasoning.
- Class imbalance: always predicting "wrong" can look accurate.
- Overconfidence: \(p=0.99\) is not a guarantee.
- Distribution shift breaks verifiers trained on easier data.
10. Laboratory · BCE playground
Logit, label, loss
Binary cross-entropy live11. Worked
- Take one (x, y) with label z in {0,1}.
- Run Transformer; read final state; compute logit a.
- p = sigma(a).
- Loss = -[z log p + (1-z) log(1-p)].
- Use p for filtering, best-of-N, search, or RL.