How Source Labels Change AI Agent Selections

An AI agent can understand a request and still favor a familiar website when choosing results. That matters when the website's name starts standing in for evidence that an item actually meets the user's requirements.

Imagine a travel assistant comparing two hotel snippets. Both mention the requested location and breakfast, but neither confirms the price. Choosing one because its booking service seems cheaper adds an assumption the snippets do not support. The domain may help identify where information came from; it does not supply the missing price.

What the source labels changed

In Source Preference in the Wild, Jonghyun Song and colleagues evaluated 12 models on shopping, accommodation and scholarly search. They compared items judged to satisfy the same requirements and controlled display position. Every model favored some sources and avoided others in each domain.

Their stronger intervention kept item content fixed and swapped displayed source identities, presenting both item orders. Selection favored the preferred label in every model and domain, although three accommodation comparisons lacked statistical significance. This supports an effect of displayed source identity under those controls, rather than attributing every selection difference to the label.

A separate comparison considered items differing by one satisfied requirement. Among comparisons with exactly one item selected, the median inversion rate was 68% when the less satisfying item came from a preferred source and the better item from a dispreferred source, versus 2% with those source roles reversed. These are conditional comparison medians, not percentages of all tasks that failed.

Preference and credibility answer different questions

For WTK, the useful distinction is between knowing the source, checking its claim and seeing whether the chosen result fulfills the goal. Our earlier note on misleading sources explains why traceability cannot establish source truth. This paper adds a different question: did a source's identity influence selection independently of the displayed requirements?

Removing all source information would also remove useful provenance. We think the aim should be to distinguish a justified reason to rely on a source from an unsupported assumption about its content. Equal requirement scores do not establish equal credibility, and reduced source preference does not itself demonstrate factual accuracy or better completed work.

What WTK needs to learn

WTK's catalog describes agent and team packages. Compilation connects a package definition to a particular target runtime, with its supported capabilities and controls. A research assistant described there could be expected to preserve source identity and distinguish confirmed facts from missing information. Whether its actual model and search configuration satisfy that expectation remains a question for evidence from that deployment.

This follows a different step from checking whether an agent understands the user's goal. Here, the request can remain fixed while the label on a candidate changes. Which source cues are necessary for verification? When does the agent infer an unstated fact from a domain? What evidence distinguishes that inference from a legitimate credibility judgment?

The study's boundary

The October 2 preprint used a text-only search environment. Agents saw titles, snippets and URLs, could not open pages, and ended by selecting results. It did not test purchases, bookings or WTK. Requirement labels came from an LLM judge with sampled human validation; the measured effects depend on those labels and the selected models.

The paper's prompt interventions do not supply a general cure. Supplying equal qualifying prices reduced preference in five shopping models. A general source-price disclaimer barely changed it; an instruction favoring the dispreferred retailer reduced preference by replacing an assumption, not verifying prices.

We have not replicated these results. Changes to the model, search provider, snippets, prompt or target runtime could change their magnitude. The learning question is whether WTK can make the reasons for selection inspectable without mistaking familiarity for verified evidence. No experiment or new capability is staged here.

Source

Jonghyun Song, Haewon Park, Jeonghoon Shim, Woojung Song and Yohan Jo, Source Preference in the Wild: How LLM Agents Favor Items by Source, and How to Reduce It, arXiv:2610.03195v1, October 2, 2026, especially sections 2-5 and 7 and appendices B, D, H and K. The study's results are external evidence; the WTK questions are our interpretation under our research method.