Evals Made the Agent Concrete

Yesterday's progress did not come from adding another layer to the architecture. It came from becoming more precise about what an agent must actually produce.

The goals gave the agent a reason to exist. The evaluations and catalog requirements turned that purpose into something we could build, inspect, compile, and eventually qualify.

The observation

A goal such as "verify that a runtime has a bounded filesystem sandbox" sounds clear until we try to evaluate it.

Then the unanswered questions appear.

Must the agent use a tool, or can it answer from prior knowledge? Which tool is permitted? What result counts as success? What happens when the tool is unavailable? Can the agent reveal the returned paths? Is a plausible answer enough, or must the runtime produce evidence that the tool actually ran?

Writing the evaluation forces those questions into the open.

The evaluation is not merely a score applied after the agent has been built. It helps define the observable behavior the builder must produce.

The catalog requirements add another layer of useful pressure. If an agent is meant to be reusable, the catalog needs more than a name and description. It needs enough structure to explain:

• Why the agent exists.
• What capabilities it requires.
• What authority it has.
• What inputs and outputs it accepts.
• Which evaluations apply.
• Which execution forms have been tested.
• What evidence supports its current status.
• Which limitations remain.
• What a user must configure before running it.

Those fields made the desired agent artifact much clearer in my mind.

Why it matters for agent systems

A loosely defined agent can produce something that looks successful while operating outside the intended bounds.

It may return the expected answer without using the required source. It may use an undeclared capability. It may satisfy the visible task while violating an authority limit. It may produce an artifact that appears complete but cannot be evaluated consistently.

More detailed prompts alone do not solve this. Prompts can guide behavior, but they are not a complete engineering contract.

The useful structure comes from connecting purpose to observable outcomes:

Purpose defines why the agent exists.
Contracts bound what it may do.
Evaluations define acceptable outcomes.
Receipts record what happened.
Qualification limits the resulting claim.
The catalog makes the governed artifact discoverable and reusable.

This structure should constrain consequential boundaries without prescribing every reasoning step. The agent still needs room to interpret information, choose an approach, and adapt within its authority.

What this means for WTK

WTK is becoming less about generating agent instructions and more about manufacturing governed agent packages.

The builder should not stop when it produces a prompt, contract, or target configuration. It should be able to show how the original purpose became a package, how the package became a target projection, what the runtime actually did, and which evidence permits a bounded qualification claim.

The catalog then becomes more than a directory of available agents. It becomes a structured record of purpose, capability, compatibility, evidence, limitations, and target-specific readiness.

This is also why qualification cannot automatically transfer between harnesses. The canonical purpose and package may remain the same, but each compiled execution form must earn its own evidence.

The boundary

More fields do not automatically create more trust.

A large schema can create the appearance of rigor while adding little practical value. Evaluations can also become too narrow, rewarding agents for reproducing a fixture instead of accomplishing the broader goal.

The structure must remain connected to meaningful behavior. Every required field should help build the agent, govern its execution, evaluate an outcome, explain a limitation, or support a trustworthy reuse decision.

If a field does none of those things, it may only be paperwork.

Still unknown

We still need to learn how much structure is enough.

Which evaluation requirements generalize across many kinds of agents? Which catalog fields help users make better decisions? Which controls materially improve outcomes, and which merely increase the cost of building?

We also need to test whether increasingly capable agents can retain useful freedom inside these boundaries. A governed agent should not become a rigid workflow simply because it is easier to evaluate.

The progress yesterday was that the desired outcome became more concrete. WTK now has a clearer path from why an agent should exist to what it must produce before anyone should rely on it.