Checking Whether Task Selection Favors Invalid Examples

An improvement loop that concentrates on difficult tasks may also concentrate on tasks that cannot be answered as written. Disagreement between model answers can signal either a reasoning challenge or a defect in the question.

Questioning the Questions: Sustaining Self-Evolution in Reasoning Models, by Jinyuan Li and colleagues, examines mathematical model training. In their R-Zero analysis, selecting generated questions by answer consistency increased the proportion judged invalid.

One comparison held 766 valid questions fixed across two datasets of 1,273 questions. The other 507 questions remained invalid in one dataset and were replaced with model-repaired versions in the other. Training Qwen3-4B-Base produced final averages of 46.54 and 49.91, respectively, across seven mathematics benchmarks.

Validity judgments and repairs depended on GPT-5.6-Sol. Repairs aimed to preserve difficulty, but the paper does not establish exact difficulty equivalence or uncertainty across independent training runs. Its R-Quest method adds validity and novelty feedback to training; it changes model weights rather than an agent's harness.

Inspect what the selector concentrates

Consider a hypothetical municipal-service agent arranging bulky-waste collection. A generated evaluation case asks it to book the resident's requested day, omits that day and then grades against a particular date.

Repeated failures could make the case look valuable for improving the agent. Yet the agent might be correct to ask for the missing day. Changing its instructions to guess a date could raise the test score while making the service less reliable.

An incomplete request can also be a valid test of clarification. The expected outcome must identify that purpose. The flaw is the mismatch between the information supplied and the behavior the evaluator demands.

Our benchmark-validity note separates a difficult test from a sound one. This paper adds a selection question: does the rule used to choose improvement tasks increase the share of flawed cases?

Keep task repair separate from agent repair

For WTK, a failure selected for investigation should retain the request, available evidence and expected outcome. That record helps distinguish an agent mistake from a missing input or an incorrect evaluation requirement.

Our reference-guidance note holds supporting material steady when assessing an update. Here, changing the case itself needs a separate record, so a repaired test does not look like an improvement to the agent.

The relevant factory question is what enters the improvement process and why. People can define valid outcomes and acceptable clarification behavior, while automated selection identifies cases for review. A high failure rate can prioritize investigation without deciding which component needs to change.

Source: Jinyuan Li and colleagues, Questioning the Questions: Sustaining Self-Evolution in Reasoning Models, arXiv:2610.04299v2, October 6, 2026. The municipal-service example and WTK interpretation are ours.