A Skill Graph Has to Beat the Baseline

A skill graph can make an agent's options look wonderfully organized and still fail to find anything its ordinary retrieval missed. That is a useful speed bump for builders: relation structure needs to earn its context and maintenance cost.

The signal

Comparative Approaches to Agent Retrieval over Large Skill Libraries compares a hybrid lexical-plus-dense ranker with a typed workflow graph for choosing and ordering skills. The study uses a corpus of 690 skills and 117 realistic, non-echoing queries.

The hybrid ranker put the right skill in its top five for 73.5% plus or minus 8.0% of queries. Under the paper's intended, token-matched comparison, replacing additional ranked results with graph neighbours was 11.2 points worse, with p = 0.0007. That is the contradiction worth keeping: a structure meant to improve selection can lose to a smaller, direct retrieval baseline.

The evidence

The authors offer a mechanism, not just a disappointing score. Their graph edges were drawn from the same embedding neighbourhood already searched by the ranker, so 98.6% of typed edges connected skills that retrieval had already surfaced together. Among the queries the ranker missed, 73% were not reachable through the graph at all.

The paper also reports that author-written queries inflated hit@5 by up to 44 points. That matters because a friendly evaluation can make extra architecture look useful before it encounters the messy requests a real operator would write.

For WTK, the point is not that skill relationships are useless. Explicit dependencies, authority boundaries, and workflow semantics can still be valuable because they make governed behavior inspectable. The narrower question is whether graph-mediated selection improves reach, ordering, or a declared safety property beyond a competent local baseline.

The boundary

This is one research corpus, one graph construction, and 117 queries. It does not evaluate WTK’s registries, show that all skill graphs harm retrieval, or prove anything about tool authority, target-runtime qualification, or production outcomes.

Nor does the paper show that its ranker is the right baseline for every library. Its reachability explanation depends on graph candidates coming from the same embedding neighbourhood as the ranker. A graph built from independent evidence or a different retrieval problem could behave differently.

The builder impact

Our take: do not add a relation layer because a diagram of it feels smarter. Before it becomes part of capability routing, compare it with a token-matched local retrieval baseline on independently authored, non-echoing queries. Measure whether the layer reaches candidates the baseline cannot, improves the order in which agents use them, or supplies a separately stated governance benefit.

Keep the evaluation honest about what the graph is for. A relation that explains why one already-retrieved tool may follow another is not automatically a relation that improves discovery. Those are different claims and should have different evidence.

The WTK test

WTK can turn this into a bounded qualification fixture without inventing a new subsystem. Hold a task set, skill library, model, token budget, and authority declarations fixed. Compare local retrieval with graph-mediated selection, retain every planned result, and record retrieval reach, ordering outcome, and any declared safety check separately.

The fixture should include independently authored requests and mark whether each expected skill was already reachable by the baseline. If the graph cannot demonstrate incremental value under those conditions, WTK should retain the simpler routing path and preserve graph semantics only where they have an explicit governed purpose.

Still unknown

We do not yet know which relationship sources can extend reach rather than echo retrieval, how graph structure affects multi-step sequencing, or whether it provides a safety benefit that a ranker cannot. The next step is a controlled, provenance-bound comparison, not a claim that WTK has solved capability routing.