Evaluating AI Agents Without Leaking Prior Task Outcomes
Advice from an earlier run can help an agent, but it cannot make the same task an independent test.
Full articleAI Agent Engineering Journal
Research records, newest first.Notes, findings, failures, and factory logs remain one chronological record of what WTK is learning and changing.
Showing 12 of 102 publications
Research area descriptionsTesting Whether AI Agent Checks Catch Important Errors
A deterministic verdict can be real and still leave important faults untouched.
Full articleDefining Authority for Automated Agent Approvals
An automated reviewer needs a defined decision scope, not permission to rewrite the rules.
Full articleMeasuring Agent Research Progress Beyond Activity Counts
More agent work is useful only when we can tell what it accomplished and what people had to correct.
Full articleChecking Agent Summaries Before Work Resumes
A continuation summary can preserve a mistake as well as useful progress.
Full articleChoosing Between Longer Agent Runs and Multiple Attempts
A fixed token budget can fund one longer attempt or several shorter ones. The useful choice depends on the task and how the result is selected.
Full articleEvaluating Agent Deployments Across Runtime Configurations
Evaluate the capabilities and settings available in the environment where the agent will actually run.
Full articleWhen an AI Agent Cannot Finish Within Its Permissions
Verify the environment before activation, and keep blocked work inside its approved scope.
Full articleChecking Shared Services for Unauthorized Agent Communication
Separate workspaces do not establish isolation if agents can exchange information through a shared service.
Full articleA Warning Is Not Containment
Recognizing a hostile message is only the first check in a persistent agent system.
Full articleWhen AI Agent Tests Pass but the Acceptance Criteria Miss the Goal
Check whether the evaluator made a mistake or the requirements left something important out.
Full articleTesting AI Agent Monitoring and Harm Prevention
Measure what a monitor detects separately from what an intervention prevents.
Full articleTesting AI Agent Safety Harnesses with Fresh-Context Review
Test whether safety checks still work when a request changes but its intent does not.
Full articleA Tool Result Is Not a New Permission
Runtime data can fill in a task without expanding its authority.
Full articleGoverning Improvement in Agents and Models
What should remain under independent control when the system proposes its own changes?
Full articleDoes the Running Agent Enforce Its Configuration?
A rule can reach the running agent without constraining what it does.
Full articleChecking Agent Permissions After Memory, Prompt, or Code Changes
An update can misuse existing access without changing permission settings.
Full articleTesting Agent Harness Changes on Held-Out Tasks
A harness change needs evaluation on tasks that were not used to develop the change.
Full articleA Loop That Forgets Can Be Patiently Fooled
A patient attack can cross the boundary that a single-run safety check never sees.
Full articleA Model Binding Is Not a Language Baseline
One deployed model can present different caution and candor signals across language contexts.
Full articleNo Tools Has to Survive the Factory
A supplied-text-only team acquired unnecessary capability requirements during planning, before its agents could execute.
Full articleWho is responsible when an AI agent acts?
An enterprise approach to agent authority, team accountability, and governed improvement.
Full articleA Revised Goal Cannot Inherit Yesterday's Success
Later user trials exposed a completion path that needed to distinguish a revised goal from its earlier successful package.
Full articleA Retry Should Not Erase the First Attempt
Hidden transport retries can separate the work an agent performs from the attempts its factory records.
Full articleA Clean Receipt Does Not Make a Stable Judge
A hash can prove the same request was sent, not that a shared evaluator will make the same decision.
Full articleA Truce Does Not Erase the Damage
An agent team can settle its disagreement after crossing the boundary that should have stopped it.
Full articleA Valid Control Path Is Not a Valid Outcome
Typed routing can stop a broken trace, but it cannot prove that the agent should have acted or reached the right result.
Full articleA Stronger Critic Can Make Tool Use Less Reliable
A reviewer model is part of the runtime, so it needs evidence that it improves the target it can block.
Full articleA Judge Cannot Promote Its Own Homework
A label made by an improvement loop is still a hypothesis until independent evidence checks it.
Full articleProposed Evidence Checks for Agent Teams
A proposal to account for every required team role before assessing the executed workflow.
Full articleA Polished Panel Is Not Permission to Act
Evidence can look authoritative and still fail to justify an irreversible action.
Full articleRouting Has to Pay for the Decision to Route
A smarter assignment is not cheaper when its own inspection cost is hidden.
Full articleAn Independent Reviewer Still Needs the Evidence
Separate authority is not enough when a reviewer cannot validate the observations it receives.
Full articleA Smaller Skill Still Has to Preserve the Contract
Compression is useful only when rare obligations remain executable and testable.
Full articleA Plausible Answer Can Hide a Failed Search
Tool access does not prove that an agent found the record its answer requires.
Full articleA Model Handoff Also Transfers a State Interface
Changing the receiver is only half the experiment when context crosses with it.
Full articleA Bounded Rating Scale Can Manufacture an Effect
A preregistered comparison can still be unable to identify the claim it reports.
Full articleA Repair Cannot Borrow Its Test
A repair can respond to a failed review without copying the withheld case that exposed it.
Full articleA Retrieval Refresh Can Change the Answer
A green utility score can still hide a changed answer regime.
Full articleA Better Claim Needs Its Own Definition
An improvement score is only meaningful when its intended outcome is declared before the comparison begins.
Full articleA Valid Target Needs More Than Valid Syntax
A target can look complete on paper and still miss the relationship that makes it work.
Full articleA Replay Cannot Rewrite the First Verdict
A second look can test retained evidence without quietly replacing the decision it was meant to examine.
Full articleA Valid Error Is Not a Passing Answer
A response can match a schema and still fail the obligation that the fixture was meant to test.
Full articleA Secret Is Not a Tool Setting
A connection can name the credential it needs without becoming the place that governs the credential itself.
Full articleA Catalog Claim Needs the Source It Will Run
Reviewing durable package source is not enough when the executable form also depends on compiled inputs.
Full articleThis Pilot Could Not Isolate a Memory Effect
Every run held unsafe promotion, but ambiguous scoring kept the memory comparison inconclusive.
Full articleA Score Gain Needs a Frozen Twin
Repeated tests of an unchanged system help distinguish measured improvement from score variation.
Full articleA File Tool Has to Stop at the Boundary
Read-only describes an operation. A package grant must also say where that operation may reach.
Full articleRelevant Memory Can Still Be the Wrong Context
Correct history can still steer an agent away from the goal it has now.
Full articleA Whole-Task Skill Can Hurt the Next Task
A skill induced from an entire trajectory can drop the no-memory baseline. A skill induced from a shared subtask can raise it.
Full articleA Registry Miss Can Be a Language Miss
A complete catalog still fails when a user's words never reach the capability it already has.
Full articleA Judge Can Become Code Without Becoming Truth
PAJAMA makes parts of model judgment inspectable and repeatable, but the resulting programs still need evidence boundaries of their own.
Full articleA Gate at the End Is Not a Gate Along the Way
A final action check cannot repair a workflow that already went off course.
Full articleSelection Is Not Verification
More candidate compute helps only when the final choice is tied to the outcome that matters.
Full articleAn Auditor Remembers What You Fixed
A verifier's repair history can change its threshold before the next task even begins.
Full articleA Gate Failure Has to Name Its Own Layer
A blocked candidate is not automatically evidence that the candidate is the problem.
Full articleA Gate Should Hunt for Its Weakest Claim
More answer samples are not always the best use of a verification budget.
Full articleA Final Score Cannot Tell You What the Agent Learned
A retained lesson should have to beat a matched run that starts fresh.
Full articleA Memory Index Has to Earn Its Rewrite
A fancier memory layer should have to beat a controllable path back to the source.
Full articleA Precise Score Can Still Measure the Wrong Thing
A score only speaks for the behavior its cases can actually distinguish.
Full articleA Matched Score Can Still Miss the Failure
The command may be right before the path to execution makes it wrong.
Full articleTwo Passing Roles Can Still Fail Together
A team is not reliable just because each role passed alone.
Full articleThe Repair Loop Needs an Exit Ramp
When a failure survives an instruction change, the system should change repair layers instead of polishing the same guess again.
Full articleThe Receipt Was Used. The Abstention Still Failed.
One model binding followed changed observations in every positive case, then failed the strict evidence-absence contract in every negative case.
Full articleThe Receipt Cannot Prove the Model Looked
A tool observation can be present in the trajectory without doing any work in the answer.
Full articleThe Evidence Stayed. Its Authority Did Not.
A package edit preserved the old record for inspection while preventing stale construction and approval evidence from authorizing the revised package.
Full articleOne Work Thread Should Not Become Two Histories
Exploration, package construction, testing, and repair should remain one traceable journey without turning the conversation into source truth.
Full articleFive Failures Needed Five Different Repair Decisions
A fixed comparison found that typed escalation routed every persistent failure correctly while instruction-only repair repeated the same move.
Full articleAn Installed Tool Is Not Necessarily the Right Tool
In four fixed cases, WTK's capability ladder preserved fit, cost consent, discovery, and construction boundaries that an installed-first policy missed.
Full articleAn Edit Has to Retire Its Old Evidence
Changing governed package source should create a new accountable revision, not let yesterday's proof follow it forward.
Full articleA Portable Package Does Not Qualify Its Runtime
Portability is not permission to assume that every harness enforces the same safety boundary.
Full articleA Safe Target Has to Survive Its Own History
A passing static test says little about what a governed agent does after its environment changes.
Full articleContext Needs a Reason to Be Trusted
A model that ignores bad context is not trustworthy if it also ignores the good kind.
Full articleMemory Has to Show Its Source
A useful memory layer still needs a receipt for where its advice came from and when it should apply.
Full articleA Skill Graph Has to Beat the Baseline
More relations can add explanation without adding one more useful skill.
Full articleA Requirement Needs a Job
The right amount of structure is not more fields; it is a clear purpose for every constraint.
Full articleA Capability Claim Has to Reach an Executable Path
A registry should not promise an action that no compiled target can actually perform.
Full articleA Benchmark Cannot Grade Its Own Blind Spots
A passing score is only useful when the test can tell a correct solution from a lucky one.
Full articleThe Leak Can Happen Before the Answer
A careful final response cannot undo sensitive data an agent already collected.
Full articleA Controller Has to Earn Its Keep
If a reasoning agent can run the search loop, explicit control needs a measurable job.
Full articleThe Tool Contract Is Not the Tool Call
The same declared tool can behave differently when the runtime changes how an agent calls it.
Full articleA Green Check Can Go Stale
A tool connection is not ready merely because it was once configured or once passed a check.
Full articleBetter Evidence Can Still Produce a Worse Decision
More grounded explanation does not guarantee that an agent will honor its hard constraints.
Full articleAn Answer Can Be Correct and Still Miss the Required Tool
For a governed tool requirement, an answer is only a partial result until the runtime shows the tool was actually used.
Full articleWhen a Required Tool Is Missing, Stop Guessing
A missing evidence capability should trigger an explicit decision, not a quieter unsupported answer.
Full articleEvals Made the Agent Concrete
An agent becomes buildable when its purpose is translated into observable outcomes, bounded behavior, and evidence requirements.
Full articleA Revised Goal Must Keep Its History
Changing the work should create a new accountable version, not let old evidence answer a new question.
Full articleA Release Claim Has to Remember the Misses
A deployment claim is only as honest as the complete evidence set it keeps.
Full articleA Judge Can Only Ground What the Evidence Makes Answerable
Better judges, better holdouts, and better repairs all depend on knowing exactly what evidence is available.
Full articleNo Evidence Path, No Confident Answer
WTK now turns a missing required evidence path into a recorded limitation instead of accepting unsupported confident prose.
Full articleMore Thinking Just Bought a More Expensive Guess
When the structure of the world shifted underneath them, agents fell back on exhaustive search instead of deduction, and extra reasoning budget made that failure cost more rather than less.
Full articleFeedback Can Guide Improvement Without Rewriting the Verdict
The August 5 development record separates session-bound feedback from evaluation results; safe autonomous promotion remains unproven.
Full articleA Package Build Is Not a Deployment Claim
WTK now requires evidence from the compiled execution form before a catalog entry can claim deployable readiness.
Full articleYour Harness Is Changing Your Agent's Mind
Fix the task, fix the model, swap only the harness, and the agent still passes while thinking differently. That is a problem for anyone who reads a green test run as a property of the agent.
Full articleMaking Evaluation Evidence Harder to Game
WTK separated evidence by purpose so a convincing run cannot quietly become a qualification claim.
Full articleGovern the Boundaries, Not the Reasoning
Flexible reasoning becomes dependable when purpose, authority, evidence, and acceptable outcomes remain governed.
Full articleA Known Workflow Needed Fewer Model Decisions
In a matched 24-run experiment, a governed fixed workflow completed every case while using fewer provider calls, tool decisions, and tokens.
Full articleHow One Misleading Source Affected Research Agents
In the cited study, one misleading document increased false conclusions under the tested research-agent conditions.
Full articleThe Agents Did All the Engineering and None of the Science
In two case studies, frontier agents received six days and substantial compute, completed the engineering, and made no substantial progress on the research questions. A second model and scaffold reproduced the failure pattern.
Full article"Held-Out" Doesn't Mean Clean
In one multimodal fact-checking study, up to 29% of post-cutoff claims remained potentially contaminated, and the effect changed system rankings. That challenges how comparative value is measured, including in WTK.
Full articleVerify Once, Answer Forever?
Replaying verified answers requires checking whether the original verification still applies.
Full articleThe Best Controls Might Be the Ones You Can't Ship
A new paper puts agent decision-making inside the model's hidden states. That's where WTK requires a tighter boundary, and we can now say why.
Full articleOlder records remain available through the archive, research areas, and RSS.