What AI Trace Search Can Leave Outside the Review

Searching agent conversations by behavior could help reviewers find failures that ordinary tags miss. The question for WTK is what a reviewer can conclude when that search returns nothing, especially when the search model never received the decisive tool result.

LangChain's October 13 engineering account describes AI Search in LangSmith. Structured filters select candidate sessions, then a decision model checks their trajectories, an ordered view that removes repeated messages. The original article describes how that view is shortened before the decision model reads it.

What the search model receives

The search uses an 8K token budget. User messages and replies to the user take priority; intermediate messages, tool results and system messages use the remaining space. A message that partly fits is trimmed, and later material is dropped. The authors explicitly identify ignored tool errors as a question that needs tool evidence. Question-specific context selection and a second pass with more context are future directions.

Consider a hypothetical coding session. The user asks an agent to fix a failing test. The agent reads several large files, runs the tests and reports success, while the test command actually returns an error. A dialogue-focused search might retain the request and reassuring reply while omitting the command's result. That would leave the reviewer without the information needed to distinguish completed work from a claim of completion.

The review question for WTK

Our exploration of bounded semantic decisions already asks whether a narrow model judgment can help choose what to inspect. Trace search adds a specific constraint: the selector's answer depends on which parts of the execution survived the input budget. Preserving the original record is useful only if the reviewer can return to the omitted evidence when the question requires it.

For security review, the same distinction matters when searching for disclosure of private information. A missing match cannot establish that no disclosure occurred if the relevant output was excluded.

WTK's catalog defines agent and team packages that are compiled for target runtimes. Humans could define which evidence is required for package evaluation and reserve consequential exceptions for review. A future search aid could direct attention during review while leaving the actual judgment tied to retained execution evidence. It would need to make the searched scope and omitted material recoverable so an empty match list is not treated as clearance.

We have not implemented or evaluated that integration. Under WTK's research method, the useful contribution here is a review question: did the search have access to the evidence that could answer this particular concern? A changed context policy, parser or decision model would require reviewing that boundary again.

Source: Arjun Natarajan, Eugene Yurtsev and Suraj Vathsa, LangChain, Search agent traces by behavior with AI Search in LangSmith, October 13, 2026.