Evaluating What Each Agent Contributes to a Team Outcome

A coding team finishes a change and its tests pass. That tells us something about the result. It does not tell us whether the tester checked the intended behavior, whether the reviewer saw those results, or whether either agent influenced the work.

For WTK, this raises a practical question: what evidence would let us understand each role's contribution without confusing participation with value?

What AgentWorld measures

AgentWorld, by Raphael Shu and colleagues, studies collaboration in a multiplayer game sandbox. Its benchmark contains 100 human-annotated tasks and 100 model-generated variants. Teams have different skills and resources, communicate through explicit messages, and work within round limits.

The paper separates task completion from an assessment of which actions contributed. Authored task verifiers check outcomes. A separate model judge works backward from successful actions to construct a dependency graph: which earlier actions or messages helped make the result possible?

The authors call their measure Causal Collaboration Effectiveness. It counts the fraction of actions selected by that analysis for each task, with failed tasks assigned zero, then averages across tasks.

That provides a way to inspect a team's activity beyond its final score. It does not independently establish that every selected action was necessary or that every excluded action was useless.

Where the evidence stops

The contribution graph depends on model judgments. The paper reports agreement on 69 of 84 judgments in a human comparison covering 11 tasks, using one human annotator. That is a useful check, but limited validation for a measure described as causal.

The score also treats actions equally. An inexpensive message, an expensive operation and a preventive check can have different practical value. Because failures receive zero, the aggregate score mixes task success with the judged contribution of actions. It should not be read as a direct percentage of wasted work.

These are game-sandbox results, not evidence that the same scoring method works for production coding teams. We have not tested it in WTK.

A concrete question for a coding team

Consider an implementer, a tester and a reviewer.

The implementer changes a function. The tester finds a failing case and sends it back. The implementer fixes the defect. The reviewer then checks the revised change against the test evidence.

A useful record would connect the failing case, the correction and the review. Three completion messages alone would not explain that sequence. Conversely, a reviewer who finds no defect may still have performed a required safety check. Their value should not depend on producing another code change.

Our earlier proposed evidence checks for agent teams ask whether required roles and handoffs are accounted for. This paper sharpens a further question: what did those handoffs contribute to the outcome?

What WTK should learn from this

For a catalog-defined team compiled for a target runtime, we would want to distinguish three questions:

• Did the team achieve the defined goal?
• Did required roles and checks run with the right inputs?
• What evidence explains how their work affected the result?

The runtime would need to expose enough evidence to investigate those questions. A reconstructed dependency map could guide review, but it would not authorize removing a role, bypassing a check or promoting a changed team.

This also extends our distinction between research activity and demonstrated progress. More messages or more agents are not sufficient evidence of improvement. Neither is a smaller action count automatically better.

The next research question is whether contribution review can identify missed handoffs or unnecessary repetition while preserving required checks. Any comparison would need a defined WTK target, retained failures and separate approval. For now, AgentWorld supplies a method to examine and a limitation to keep visible, not a validated WTK control.

Source and review boundary

Raphael Shu and colleagues, AgentWorld: Benchmarking Long-Horizon Collaboration of Multi-agent LLMs, arXiv:2609.31590v1, September 25, 2026. We reviewed the complete primary PDF, including its evaluation method and appendices. This is an external-research interpretation under the WTK research method, not a completed WTK experiment.