Hybrid rewards: rules where you can, a model where you must

A rule checks. A learned reward model judges.

Every chapter so far has trained a network to imitate human or AI judgements. That is necessary when quality is a matter of judgement. But for many prompts the right answer is known in advance. "What is \(17\times 23\)?" has one correct answer, and a program can compare the model's answer with it.

A program has two advantages over a learned model. It does not drift as the policy changes, and it cannot be talked into a high score by confident prose. It also has one large disadvantage: it only sees what it was written to check. A hybrid reward uses each kind of signal for the part of the job it does well.

The most common rule is exact match. Extract the final answer from the response, normalise it (strip spaces, simplify fractions, unify number formats), and compare it with the key \(a^\star(x)\):

For code, the rule runs the program. With \(K\) unit tests, the fraction that pass gives a reward with partial credit:

Other rules in common use:

• Compiler or parser. 1 if the code compiles or the JSON parses, 0 otherwise.
• Format check. A regular expression confirms that the reasoning sits inside <think> tags and the answer inside \(\boxed{\cdot}\).
• Instruction constraints. "Answer in fewer than 100 words" or "use exactly three bullet points" can be counted.
• Length limit. A penalty that grows once the response exceeds a token budget.

A rule is only as good as its extractor. If the answer is \(\tfrac12\) and the response writes \(0.5\), a careless normaliser marks a correct response wrong.

Reinforcement learning with verifiable rewards (RLVR) replaces the learned reward model with a verifier on prompts where one exists. The Tülu 3 recipe applied it to math problems with known answers and to instructions with checkable constraints. The verifier gives a constant reward \(\alpha>0\) when the check passes and zero otherwise, and the usual KL penalty keeps the policy near its reference:

Compared with standard RLHF, where the reward is a scalar RM, only the reward has changed. The optimiser (PPO, GRPO or another policy-gradient method) is the same. What changes is the failure mode: a verifier cannot be flattered, so the policy can only raise its reward by being right more often or by finding a hole in the verifier.

DeepSeek-R1-Zero was trained with RL from a base model using only rule-based rewards. Its authors chose not to use a neural reward model for this stage, citing the risk of reward hacking during long RL runs. The reward has two parts:

• Accuracy. For math, the final answer must appear in a required form (for example a box) and match the key. For coding problems, a compiler runs predefined tests.
• Format. The reasoning must be placed between <think> and </think> tags.

The two parts are coupled. The accuracy rule can only read an answer it can find, so the format rule is partly a precondition for the accuracy rule. A model that solves the problem but forgets the box gets nothing for its correct answer.

For DeepSeek-R1 the authors also observed that the chain of thought mixed languages. They added a language-consistency reward, the proportion of words in the reasoning that are in the target language, and summed it with the accuracy reward:

In the final RL stage of DeepSeek-R1, reasoning prompts kept their rule-based rewards while general prompts were scored by learned reward models for helpfulness and harmlessness. That is a hybrid reward of the routing kind (Section 6). The optimiser behind these runs is the GRPO method from the RL notes.

Rules run out in four recognisable ways.

• Open-ended quality. "Explain recursion to a beginner" has no answer key. A rule can count words but cannot tell a good explanation from a bad one.
• Partial credit. A proof with one gap, or code that is right but slow, is neither fully right nor fully wrong. Exact match gives 0 to both.
• Style. Two correct answers can differ greatly in readability. A correctness rule scores them the same.
• Safety nuance. A keyword filter can block a word, but it cannot tell a dangerous request from a historian's question that uses the same word.

In each case a learned model, trained on judgements, supplies the missing signal. The rule still has a job: it anchors the parts that can be checked.

Four choices of \(\Phi\) cover most systems:

• Routing. Use the rule when the prompt is verifiable, the learned RM otherwise.
• Additive shaping. Add weighted rule rewards to the RM score (Section 7).
• Gating. If a hard rule fails, the response gets a floor reward whatever the RM says (Section 8).
• Penalties. Subtract costs for length, language mixing or other unwanted traits (Section 8).

Routing needs a set \(\mathcal V\) of prompts that come with a verifier:

In a training loop the patterns are usually stacked:

The additive form keeps every signal and weights it:

The weights only mean something if the terms share a scale. A Bradley-Terry RM outputs unbounded logits, while a rule outputs 0 or 1. Suppose a confident wrong answer gets an RM logit of 4.0 and a plain correct answer gets 2.5. With \(w_{\text{RM}}=w_{\text{acc}}=1\), the totals are \(4.0+0=4.0\) and \(2.5+1=3.5\): the wrong answer wins. Squash the RM score first,

and the totals become \(0.982\) and \(1.924\). The correct answer now wins. Other choices, such as standardising RM scores within a batch, serve the same purpose. Once scales are fixed, the weights can be chosen so that correctness cannot be outvoted.

Within each group, correct or incorrect, the RM and the other terms still decide the order. This is a lexicographic preference written as a sum: be right first, then be good.

A gate turns a rule into a hard requirement. Let \(g(x,y)\in\{0,1\}\) be 1 when all hard rules pass (the output parses, the answer tag is present, no banned content). The gated reward is

Penalties are softer. They subtract a cost that grows with the unwanted trait. A length penalty with budget \(L_{\max}\) and strength \(\lambda\) is

With \(L_{\max}=1000\) tokens and \(\lambda=0.5\), a 1400-token response pays \(0.5\times 400/1000=0.2\). A language-mixing penalty can be written as \(-(1-r_{\text{lang}})\) using Equation 5. Penalties target known biases of learned RMs: most preference-trained RMs favour longer answers, and an additive length penalty is a direct counterweight.

Safety behaviour is too nuanced for a regular expression, yet designers often know exactly what they want: a refusal should be brief, should not lecture the user, and should not contain the requested harmful content. Two lines of work turn such written rules into rewards by asking a language model to apply them.

In the additive form, a proposition-based safety reward looks like

These are not rule-based in the program sense, because the grader is itself a model. They sit between the two families: the criteria are fixed and written down like rules, while the checking is done by a model. This makes the policy auditable (each rule can be read and changed) without requiring a regex for every nuance. The generative judges of Chapter 09 are the natural graders.

A policy optimises whatever the reward measures, including its blind spots. Rules and learned RMs have different blind spots, which is the main argument for using both.

Multiple choice shows how a small change to a rule removes an incentive. With four options and reward 1 for right, 0 for wrong, a random guess earns \(0.25\) in expectation, more than saying "I don't know". Give \(-\tfrac13\) for a wrong answer instead:

• Scale and weight tuning. The weights in Equation 7 interact with RM calibration and must be retuned when the RM changes.
• False negatives from verifiers. Correct answers in unexpected forms (\(0.5\) vs \(\tfrac12\), a reordered list) are marked wrong and the policy is pushed away from them.
• Narrow coverage. Verifiable prompts are mostly math, code and constrained instructions. A model trained mostly on them can drift on everything else.
• Routing errors. A prompt sent to the wrong branch gets a meaningless reward, for instance an open question checked against a single answer key.
• Incomplete written rules. Safety propositions cover the cases their authors thought of; new failure types need new propositions.
• Sparse signal. A 0/1 verifier gives no gradient on prompts the policy never solves, or always solves. Group-relative methods see zero advantage on such prompts.

A GRPO group samples four responses to one math prompt. The reward is accuracy plus format, \(r=w_{\text{acc}}r_{\text{acc}}+w_{\text{fmt}}r_{\text{fmt}}\). Response 1 is right and formatted, 2 is right without tags, 3 is wrong with tags, 4 is wrong without tags. Compute the group-normalised advantages \(\hat A_i=(r_i-\bar r)/\operatorname{std}(r)\) for two weightings.

Rules give a clean outcome signal on verifiable tasks, but they only judge the end of a solution. The last chapter returns to rewards inside the reasoning: the many kinds of process reward model, what each one's step score actually means, and how those scores are used in search.