Unlearnable, or Unmeasured? On the Reliability of Difficulty Labels in RLVR

Motivation

Prompt difficulty has become an operational quantity in RLVR. Methods use a model's empirical success rate to decide which prompts to replay, filter, route, reweight, or give extra training effort, and difficulty is also used to probe the limits of RLVR itself. Most strikingly, Chen et al. (2026) identified hard prompts that occasionally succeed yet barely improve, called them unlearnable, and connected their behavior to lower within-group gradient similarity.

All of these uses depend on identifying hard prompts reliably. Difficulty is estimated from sampled responses, so the same prompt can land on either side of a threshold across repeated evaluations even when the model is unchanged. We treat the aggregation rule and the rollout budget as part of the measurement, and ask when a prompt-level trainability label is stable enough to support a scientific claim.

Setup

We reproduce the unlearnability study of Chen et al. (2026) and keep its published cohorts, threshold, and verifier.

Component	Configuration
Model	Qwen2.5-0.5B trained with GRPO, G = 8 rollouts per prompt, five independent seeds.
Data	1,023 prompts from MATH; the original answer verifier, unmodified.
Cohorts	The published easy, learnable, and unlearnable groups; difficulty threshold τ = 0.1. Du denotes the set produced by the published unlearnability construction.
Set membership	N = 128 evaluation rollouts per prompt; independent sets compared by Jaccard similarity.
Gradients	N = 200 initial-policy rollouts; each prompt-level gradient averages its correct rollouts.
Uncertainty	95% bootstrap intervals, resampling prompts at the level appropriate to each statistic.

Our run is not a byte-for-byte rerun. The key difference is a 1,024-token generation cap instead of the original 5,120, which we test directly on the evaluation side; the full constant-by-constant audit is below.

“Unlearnable” Prompts Do Learn, Slowly

The Aggregation Rule Changes the Set

To separate training variation from the aggregation rule, we fix a common candidate pool: evaluating all 1,023 prompts 128 times on one reference checkpoint leaves 292 prompts at or below τ = 0.1. From these same candidates:

• the published rule, which drops a prompt marked no_reward in any of the five runs, keeps 74;
• dropping a prompt only if it is no_reward in all five runs keeps 228;
• each training seed on its own keeps 145–150 (mean 147.8);
• requiring zero correct responses in the 128-response evaluation keeps 181.

What fails is the estimator, not the target. For a fixed policy with true success probability p(x), the set D_τ = {x : p(x) ≤ τ} is well defined. But requiring a prompt to be labeled difficult in every independent run means that, as runs are added, only prompts almost guaranteed that label survive, pushing the retained set toward p(x) = 0. The no_reward exclusion then removes exactly those prompts. The procedure does not converge to D_τ; it converges to a degenerate outcome of the aggregation rule. Pooling the same rollouts into one deeper estimate avoids this.

“No reward was observed in training” is not the same as “unsolvable”: of the 218 prompts removed by the no_reward criterion, 116 are solved at least once in a later N = 128 evaluation.

Most of the Disagreement Is Evaluation Noise

To remove training variation entirely, we evaluate the same checkpoint twice, changing only the sampled rollouts. At N = 32 the two evaluations agree at Jaccard 0.798, versus 0.751 across independently trained seeds: resampling one fixed model reproduces about 81% of the apparent cross-seed disagreement. A binomial model of the threshold decision predicts the same-model agreement to within 0.004.

A longer generation cap does not explain the movement either. Re-evaluating all prompts with 5,120 instead of 1,024 tokens cuts truncation from 9.6% to 3.2%, yet the two sets agree at 0.898, no less than two passes at the same cap (0.877).

Table: Key robustness controls.

How Many Rollouts Does a Difficulty Label Need?

Spend the budget deeply, not in shallow passes. With the same 128 rollouts per prompt, one deep estimate thresholded once agrees across repeats at Jaccard 0.877, while four blocks of 32 intersected agree at only 0.791 (difference +0.086 [+0.038, +0.135]). Both rules use the same effective threshold, ⌊0.1 × 128⌋/128 = ⌊0.1 × 32⌋/32 = 0.09375, so the gain comes from allocation, not from extra compute.

Predicting reproducibility. With N rollouts, a prompt with success probability p is flagged with probability q_N(p) = Pr(K ≤ ⌊τN⌋), where K ~ Binomial(N, p). We estimate the distribution F of latent success probabilities from the pooled N = 128 evaluation with the Kiefer–Wolfowitz nonparametric MLE, which accounts for the noise in observed pass rates, and predict the agreement of two independent evaluations:

Table: Predicted Jaccard agreement at representative budgets (deep, single-pass rule).

The Budget Model Predicts Unmeasured Budgets

Table: Retrospective and cross-model consistency checks. Cross-model rows reuse the pass their distribution was estimated from, so they are consistency checks rather than independent validation; the Qwen2.5-1.5B row (greyed) is an excluded diagnostic.

The Gradient Gap Shrinks When Sample Count Is Matched

The gradient explanation compares how similar per-prompt gradients are within each group, but each prompt-level gradient averages its correct rollouts, and hard prompts have far fewer: in the N = 200 cohort, a median of 8 for D_u versus 65 for easy prompts.

Every matched level (K = 4, 6, 8, 11) lies below the unmatched ratio and above one, so matching weakens the separation substantially without eliminating it. Matching also selects: at K = 11 only 28 of the 71 D_u prompts remain, and they are the easier ones, so this shows sensitivity to sample count, not a corrected full-cohort estimate.

Two Simple Optimization Explanations Do Not Hold

Content starvation, where rewarded trajectories carry weak learning signal, remains possible; we cannot test it because the training-time responses were not stored. The cause of the slow-learning effect is still unresolved.

An Unstable Label Does Not Necessarily Mean Unstable Training

Does a noisy difficulty rule make difficulty-aware RLVR ineffective? We train five seeds under two neighboring constructions that remove very different amounts of usable training signal, 5.0% versus 27.6% after GRPO's zero-variance filter, alongside size-matched random-removal controls. The resulting models differ by less than one held-out accuracy point. The experiment cannot resolve effects that small, so this is not evidence of equivalence, but it bounds our claim: a large change in the reported set need not produce an equally large change in training. Our conclusion is about measurement and inference, not a claim that every method using noisy difficulty signals must fail.

Replication Audit

We audited the effective configuration from the actual training and evaluation paths: of 47 effective constants, 30 match the original and the 17 below differ. The generation cap matters most, since it affects hard prompts more than easy ones; its evaluation-side effect is tested above, but our training-dependent magnitudes should not be read as an exact reproduction of the original configuration. All runs used a single NVIDIA RTX 3090 Ti (24 GB).

BibTeX

@inproceedings{chakma2026unlearnable,
    title={Unlearnable, or Unmeasured? On the Reliability of Difficulty Labels in RLVR},
    author={Chakma, Chandak and Sakib, Syed Nazmus and Haque, Nafiul and
            Arman, Shifat E.},
    booktitle={NeurIPS 2026 Workshop: Transitioning from Pre-Training to Post-Training},
    year={2026}
}