Your Harness Is Changing Your Agent's Mind

A new paper holds the task, the environment, and the base model constant, changes only the harness the agent runs inside, and finds that the agent's beliefs move. Terminal success mostly survives. What the agent believed on the way there does not.

The catch: the scaffolding is an experimental variable

Measuring Harness-Induced Belief Divergence in Multi-Step LLM Agents pins this on ordinary harness features, not exotic ones: "blocked actions, compressed repairs, selective verification, and cost-aware evidence pruning often preserve terminal success while changing the beliefs that drive later decisions." Their conclusion is blunt: "harness design is an experimental variable in agent evaluation, not an implementation detail."

Read that twice if you build agents. The pass/fail signal at the end is exactly the part that survives a harness swap. The trajectory underneath it is the part that quietly moves.

Why this matters to WTK

How WTK research views this: it is the sharpest challenge our sweep has produced against our own harness portability claim. Our position is that governance rides inside the agent artifact, so the artifact runs safely on whatever deployment target you point it at, while the qualification evidence gets produced on our harness. This paper says an artifact can qualify green with us and then diverge mid-trajectory somewhere else while still passing its fixtures. The qualification record would not notice.

There is a second hit. If terminal success is harness dependent, then our bar for general availability (at least 80% goal accomplishment, 100% on honesty and safety, measured on held-out work) is only valid per harness. It is not a property of the artifact unless it is measured per target. That is a scoping constraint on how we may state the bar, not a refutation of it.

How WTK proposes to solve it: the cheap version first. Their belief-elicitation protocol is heavyweight and research-grade, and we are deliberately not building one. The narrow, testable version of their claim is to run one existing fixture set under two harnesses and check whether the terminal verdicts agree. If they agree, the concern is bounded. If they diverge, the qualification record needs to say which harness produced it.

The honest summary: we already knew harnesses differ. What we do not currently do is stamp qualification evidence with the harness that made it. That is one field, not a research program. The paper's value is the justification for the field, not its method.

Also on the radar

• Safety you can copy is safety you can strip. Safeguards Based on Copyable Context Cannot Provide Reliable Safety for LLMs proves a worst-case floor on attacker assistance whenever the evidence about downstream use is copyable, and states it as a trilemma: "Useful Capability, Reliable Safety, and Open Access cannot coexist." An agent artifact is copyable by construction. Our read, and this one lands in our favor: it does not touch the grounding floor or the honesty checks, because those are properties of the answer and they travel fine. What cannot travel is access control for dual-use capability, which depends on knowing who is calling. We sign packages for integrity, not to gate who may run them. So the output here is a scoping edit to what we claim, not a feature: artifact-portable governance covers answer validity, not caller authorization. We are not building a credential layer off a paper.
• Stop asking the same question twice. CAPA benchmarks cross-session ambiguity adaptation with 600 sessions across 60 balanced user-ambiguity cells, 300 held-out evaluation sessions, and 12 models, scored on executable success, first-turn success, and turns-to-completion. Our goal intake is fail-closed and effectively stateless, so it re-interviews the same operator about the same recurring preferences forever. The metric is worth more to us than the method: measure turns-to-completion on repeat goals before building any personalization, and if the number is small, drop the idea. One hard rule if we ever do build it: history may propose a resolution for the operator to confirm, never satisfy the completeness check on its own. Fail-closed intake that resolves ambiguity from memory is just fail-open with extra steps.

The takeaway

A test result is a claim about a system, and the harness is part of that system. If you are shipping an agent that will run somewhere other than where you tested it, either measure it there too or say plainly that you did not. We would rather add a field recording where the evidence came from than keep a check mark that quietly means less than it looks like.

See you at the next catch.