Distinguishing Blocked Work from Unfinished AI Agent Tasks

An unfinished agent task does not tell us whether another attempt can help. A reviewer may accurately recognize completed work while struggling to distinguish a missing prerequisite from an action the agent has not discovered yet.

Imagine an assistant preparing a report from a required spreadsheet. If the file exists in an approved folder but the assistant has not found it, another permitted search might help. If the required file is unavailable and no valid replacement exists, searching longer cannot supply the original data. Both runs are unfinished, but they need different responses.

What OSWorld-Pro evaluates

In OSWorld-Pro, Zhilin Wang and colleagues at NVIDIA evaluate computer-use agents through intermediate steps, not only final deliverables. Their September 21 preprint contains 305 Linux desktop tasks, 2,814 subgoals and 67,264 human step-label annotations.

Annotators identify the subgoal an action targets, whether the environment permits completion, whether the action makes progress and whether it completes that subgoal. Reviewers first annotate independently, then reconcile disagreements. This makes the distinction concrete: a subgoal can remain incomplete even when completion is still possible.

Completion agreement does not establish feasibility agreement

For the paper's GPT-5.6-Sol judge at maximum reasoning effort, Table 1 reports a completion Macro-F1 score of 94.3 and a feasibility score of 61.9. Macro-F1 summarizes agreement with reference labels across classes; these scores are not percentages of whole tasks completed successfully.

The measurements also have prerequisites. Feasibility is scored only when the targeted subgoals are correctly identified. Progress is scored when feasibility is correct, and completion when progress is correct. Errors in earlier judgments are therefore outside the later conditional score. A strong completion score cannot describe the full sequence of decisions.

The reasoning comparison adds a caution. With no reasoning, the same named judge scored 67.8 on completion and 67.5 on feasibility. At maximum effort, completion agreement improved while feasibility agreement declined. That is a descriptive result under this setup, not a reason to reduce reasoning generally or evidence that reasoning caused unsafe actions.

Where the study leaves uncertainty

The authors acknowledge that an undiscovered menu option can look unavailable until the agent finds it. Feasibility labels involve interpretation, even with human review. Their final judge comparisons use one run per configuration and one model family, without per-label uncertainty estimates.

The study judges recorded trajectories retrospectively. It does not test a live stopping policy, permission enforcement or WTK. Its separate actor benchmark counts completion of subgoals deemed feasible by humans; that does not demonstrate that an agent or judge independently recognizes blocked work.

What this means for WTK oversight

WTK's catalog describes agent and team packages. Compilation prepares a package for a target runtime; qualification evaluates a specified version in that environment. Our question is whether the evidence supporting an evaluator names the decision it can actually support: completed, still feasible, blocked or unresolved.

Our blocked-work note concerns keeping work inside approved authority. OSWorld-Pro adds a prior question: how well can the reviewer identify the condition that calls for a stop or reassessment? Our checker-coverage note similarly separates a working evaluator from evidence that it catches the relevant mistakes.

Higher-level human oversight does not require approving every click. It does require dependable exception information. In the report example, a useful escalation would identify the required file, the permitted locations checked, the partial work retained and what remains uncertain. It should not merely say that the agent tried many times.

The question for WTK is whether its evaluation records distinguish recoverable unfinished work from genuine blockers, including mistakes about prerequisites. Feasibility advice does not grant new permissions or authorize a substitute deliverable. WTK has not established this distinction experimentally; changes to the model, environment, evidence input or label definitions require renewed assessment.

Source

Zhilin Wang and colleagues, NVIDIA, OSWorld-Pro: Process-based Evaluation for Computer Use Agents, arXiv:2609.24890v1, September 21, 2026, especially Section 4, Table 1 and Appendices D and G. The WTK oversight question is our interpretation, following our research method.