Evaluating Agent Plugins Within the Workflows They Change

A plugin can improve one agent setup without improving every workflow that installs it. When a platform adds an extension for memory, tools or instructions, the useful question is not only whether the component works, but how the complete workflow uses it.

Imagine a terminal extension that highlights failed tests before shortening their output. It helps a coding agent notice errors. Now combine it with a memory extension that repeatedly returns an older failure after the test has passed. Each component may work as designed while their combination distracts the agent. This is a hypothetical interaction, not an observed WTK failure.

How PluginRSI separates the decisions

In PluginRSI, Yaorui Shi and colleagues represent an agent harness as executable plugins plus a workflow that coordinates them. The harness is the software around a fixed model, including its roles, tools, skills and memory. The workflow determines when components run and how their outputs are used.

The search first changes one plugin while keeping the workflow and other plugins fixed. It compares that candidate with the current setup on the same small task set. A variant with a better score enters a reusable library.

Then a separate stage selects plugin versions and revises their coordination. That complete assembly is evaluated on the full validation set before it can replace the current harness. Retention in the library does not automatically make a component part of the accepted workflow. The policy and proposer models remain fixed during search; this is harness improvement, not model-weight training.

Local gains can include regressions

The screening sets contain 20 tasks, deliberately balanced between ten previously solved and ten unsolved. They are diagnostic samples, not representative estimates of deployment success.

One terminal-feedback mutation increases resolved tasks from 10 to 14. The authors report five repaired failures and one regression. The net gain therefore includes a task the previous setup handled better, and the original outcomes used in this comparison are cached.

The larger comparison also matters. In Table 4, the complete PluginRSI method reaches a 69% held-out resolve rate with Kimi-K3, compared with 59% when harness recomposition is removed. Adding a plugin library to Meta-Harness alone changes its held-out result from 63% to 64% with Kimi-K3, while lowering results with the other two tested solvers. These are descriptive results on 100 SWE-bench tasks, not proof that any particular plugin interaction caused the difference.

What the evidence does not transfer

The study uses fixed benchmark tasks and September 2026 model endpoints, without reported multi-seed uncertainty for these rates. Held-out coding tasks share repositories with development tasks. Isolated mutations can also miss improvements requiring several components to change together.

Its separate reuse experiment optimizes on the former held-out tasks. That shows adaptation on those tasks, not a fresh independent final test. Our held-out harness evaluation note explains why that distinction changes the claim.

Nor does this benchmark establish the security of platform extensions. The authors explicitly call for code review and appropriate execution permissions beyond benchmark environments. A higher task score cannot authorize additional access.

The WTK question for reusable components

WTK's catalog describes agent and team packages. Compilation prepares a package for a target runtime; qualification evaluates a specified version in that environment. A reusable component can be worth retaining without qualifying every compiled package that includes it.

Our interpretation is to keep the two evidence records distinct. Component evidence would identify its version, tested surrounding setup, local benefit and regressions. Assembly evidence would identify the selected versions, coordination, target runtime and completed goal, including required constraints.

This extends our workflow-revision note. That note concerns changes during execution; PluginRSI separates component discovery from acceptance of a newly assembled harness. For a team, the same distinction asks whether a helper improved one role or the complete handoff and outcome.

Higher-level oversight could review those bounded changes and exceptions rather than every component invocation. Whether WTK's current evidence supports that review remains untested. Changes to components, coordination, model, permissions or target require renewed assessment; a library entry is not deployment approval.

Source

Yaorui Shi and colleagues, PluginRSI: Recursive Improvement of Agent Harnesses with Reusable Plugins, arXiv:2609.32423v1, submitted September 26, 2026; PDF dated September 29. Sections 3-4 and Appendices D, E and K support this interpretation under WTK's research method.