Testing Agent Evidence Checks Against Valid Alternatives

An agent can obtain the evidence it needs through more than one legitimate path. A checker that recognizes only one path can reject correct work. A checker that accepts matching values without checking whom or what they describe can approve the wrong action.

From Evidence to Action: How Tool-Using Agents Fail, an October 6 preprint by Hongzhan Lin and colleagues, makes both problems concrete. Its evaluator audit asks whether alternative ways of establishing a requirement are accepted, while evidence for the wrong entity or an outdated state is rejected.

Check the requirement, not one route to it

Consider a hypothetical enterprise access agent renewing a contractor's access to a reporting system. Assume the organization's policy permits renewal only when current approval covers that contractor, system and renewal period.

The agent might read the approval directly from the access system, or discover the request through a service desk and retrieve its linked approval record. If both permitted paths establish the same required facts, the checker should recognize both.

An approved record for another contractor would not qualify, even if its status also says approved. Nor would last month's approval establish permission for a new period. Accepting either could extend access beyond the declared policy; rejecting a valid alternative could send routine work into unnecessary manual review.

The useful distinction is between how the agent found the evidence and what that evidence establishes.

The researchers had to repair their checker

The paper introduces SAFEACTBENCH: 656 synthetic cases across six operational domains, spanning action judgments, investigated non-action, single actions and dependent workflows. A deterministic evaluator checks recorded observations, action arguments, dependencies and resulting state against each case's requirements.

That makes the evaluation inspectable, but the checker still needs testing. All 656 author reference solutions passed. That establishes consistency with those solutions; it does not establish that every legitimate alternative will pass.

In an initial audit, the evaluator rejected four valid alternatives: two using alternative retrieval, one obtaining equivalent facts through different tools, and one combining support from multiple observations. The authors traced these false rejections to dynamic date handling and repaired it before the final benchmark evaluation.

Table 8 then reports acceptance of 32 out of 32 alternative retrieval variants, alongside equivalent-fact and combined-evidence variants. It accepted none of the 20 wrong-entity or 30 stale-evidence controls.

These are results from the authors' synthetic audit family after repair, not a separate held-out audit. They demonstrate why both kinds of control matter without establishing complete coverage. The recorded evidence also shows what was available before an action, not what the model internally relied on.

What this sharpens for WTK

For WTK catalog packages and compiled target runtimes, the question becomes more specific: can qualification distinguish an acceptable alternative evidence path from an entity or state substitution?

Before relying on a target checker, we would need the package's actual evidence requirement, its permitted sources, the proposed action and the observations available beforehand. Qualification evidence should make clear which of those conditions the check establishes. A matching approval value alone is insufficient, and an observation cannot widen authority to act.

This supports higher-level human oversight. People define acceptable evidence, delegated limits and exceptions; routine investigation can follow different valid paths within them. The checker should preserve those requirements without forcing a person to approve every variation in how the agent satisfies them.

Source

Hongzhan Lin, Shidong Cao, Ziyang Luo, Wenhao Chai, Mong-Li Lee and Wynne Hsu, From Evidence to Action: How Tool-Using Agents Fail, arXiv:2610.07753v1, October 6, 2026. Sections 3–4 and Appendix B.3, especially Table 8, support this interpretation. The access-renewal example and WTK question are ours.