Evaluating Agent Deployments Across Runtime Configurations

An agent can have a defined goal and approved permissions, yet perform differently depending on its execution setup. For WTK, the practical question is whether the selected model and runtime configuration can complete the intended work within those boundaries.

The runtime supplies capabilities such as tool execution and context handling. Some behavior is configurable; some is fixed by the provider. Our evaluation needs to reflect what we can actually deploy and control.

What the research demonstrates

An Empirical Study of Harness Design for Coding Agents examines planning, tool interfaces, and context management within one custom coding harness. It does not compare commercial deployment runtimes.

One result illustrates the tradeoff. Under the study’s 128k context configuration, its persistent planning mechanism increased Nemotron-3 30B’s SWE-Bench Verified success from 13.6% to 25.2%, while mean cost rose from $0.02 to $0.09 per task.

That is evidence about a particular planning implementation, model, and task set, not a universal recommendation. Planning was evaluated under one context-management configuration, and each setting ran once per task. The tool-interface comparison also changed instructions, state tracking, and diagnostics alongside tool availability.

The lesson we draw is to evaluate configuration choices under their intended conditions, rather than assume that a useful component transfers unchanged. That deployment lesson is WTK’s interpretation of the study.

Start with requirements, then inspect the runtime

Consider a hypothetical coding agent authorized to modify a designated repository and run its tests, but not deploy the result.

Before choosing settings, we need to establish what the task requires. Can the runtime execute the necessary tools? Can it restrict writes to the approved location? What happens when the working history exceeds the available context? Which of those behaviors can we configure, and which must we accept or work around?

A useful deployment review separates:

• Required capabilities: what the agent needs to complete the task.
• Available controls: what the runtime exposes and enforces.
• Unresolved gaps: what is unsupported, unknown, or not yet verified.

A prompt asking the agent to stay inside a directory is not equivalent to an enforced filesystem restriction. Likewise, a runtime’s context-management feature does not establish that it preserves everything a particular task needs.

These are separate questions about security and task performance. Passing one does not settle the other.

Evaluate the configuration we can actually use

For WTK, the paper motivates evaluating supported deployment choices, not recreating the researchers’ harness or assuming access to provider internals.

Our runtime qualification note explains why a portable package does not establish equivalent behavior across environments. Our deployment-review guidance adds the need to verify actual access before allowing work.

Where a runtime exposes a relevant setting, a later comparison could evaluate an approved alternative against the current configuration. Where behavior is fixed, that constraint belongs in the evaluation record. We should not describe an inaccessible internal mechanism as a parameter WTK can tune.

Before proposing that comparison, we need an inventory of the selected target’s supported settings and enforced restrictions. Any experiment would require its own specification and approval. This note does not authorize one.

What remains unknown

The paper does not establish which configuration will work best for WTK, validate our security boundaries, or evaluate our agent teams. Its results concern selected coding tasks and an author-controlled harness.

Our takeaway is narrower: define the requirements, inspect the runtime’s capabilities, and evaluate the configuration that will actually execute the work. Keep unsupported requirements visible rather than treating a configuration declaration as proof.

Source

Run-Ze Fan and colleagues, An Empirical Study of Harness Design for Coding Agents, September 17, 2026. The deployment example and WTK recommendations are our interpretation, not results reported by the authors.