Testing AI Agent Monitoring and Harm Prevention

An AI-agent monitor can help investigators find harmful activity without establishing that it can prevent it. For WTK, the practical question is whether an additional action-time control improves protection without unnecessarily blocking legitimate work.

Imagine an agent requests deletion of a protected record. An alert may help someone investigate the request. A control applied before execution may reject it. To assess prevention, we need to inspect whether the record was actually deleted, not just whether either component produced a reassuring message.

What the research measures

MOLE evaluates monitoring in a simulated organization with 150 AI-operated accounts, nine services, and twelve constructed threats. Its retrospective monitors select ten accounts for daily review. Completed harm is checked against service state.

On the GPT-5.3 single-day campaign audit-event corpus, Claude Opus 4.7 retrieved 53.3% of account-days involving completed harm within that review limit. This measures investigation coverage, not the percentage of attacks prevented. GPT-5.3 generated that corpus; Claude Opus 4.7 was the monitor in this result.

The paper also describes an implemented pre-execution intervention interface, but reports no prevention experiments. It therefore does not establish that interface's effectiveness or the cost of incorrectly blocking legitimate work.

These are September 2026 preprint results under selected models and synthetic conditions. Assigned threat objectives do not establish how often agents would independently attempt harmful actions in production. Nor do these results establish a WTK defect or transfer the reported detection rate to WTK.

What WTK should take from it

Our interpretation is that investigation coverage and prevention need separate tests. Monitoring remains useful for diagnosis and review even when it does not stop an action. A prevention claim additionally needs evidence of what the system allowed to change.

This connects to our note on checking actual outcomes: an accepted request or successful tool response does not establish the final result. For the deletion example, inspect the protected record after the test and retain the relevant decision history.

The same separation applies to fresh-context safety review. An extra review step can contribute evidence without becoming final evaluation or deployment approval. We should evaluate each control for the job it performs.

A proposed comparison in WTK

First, identify an existing WTK action path and document its current protections. We have not established an alert-only baseline or a missing enforcement check. If a specific additional control is worth testing, compare the unchanged path with that same path plus the proposed control before execution. Preserve all existing protections in both groups.

Use isolated test records, with matched requests for prohibited and permitted deletions. Keep the agent version, model, tools, permissions, starting data, and evaluation criteria fixed. The added control should be the planned difference, not an opportunity to change the agent's authority or make the baseline weaker.

Measure harmful changes that actually occur, legitimate work completed, incorrect denials, response time, and human review effort. Retain every attempt, decision, and final-state check. Allocate attempts and stopping conditions in advance; do not selectively retry inconvenient outcomes.

If both groups already prevent the harmful action, report no added prevention benefit under those conditions. A valid null result is different from an inconclusive test that cannot resolve the question. Missing required evidence can make the comparison inconclusive; weakening safeguards violates the test's safety requirements. Neither should be disguised as improvement.

This is a proposed WTK experiment, not a completed Finding or permission to run. A human must approve the exact action path, added control, test cases, attempt budget, and acceptance criteria before execution. Changes to those components would require a new assessment. Any adoption would need separate qualification and deployment approval.

Source

Primary research: MOLE: Detecting Insider Threats in AI Agents, Aashiq Muhamed and Virginia Smith, September 2026, arXiv:2609.06966v1. The source-review archive records a complete 48-page PDF review; this editorial revision rechecked the cited metric and intervention boundary. WTK has not run the proposed comparison.