Why AI Agents Get It Wrong: Blame Retrieval, Not the Model

Andrew Maryasov, AI consultant, founder of Auspex and Grow2.ai. When an agent gives a wrong answer, a business’s first reaction is to get a smarter model. A Stanford and Together AI study on 2,100 questions showed something else: over 70% of errors arise at the retrieval step, not the reasoning step. When the model finds the right source, it almost always extracts the fact from it correctly. The whole difficulty is landing on the right document in the first place.

Two terms up front, because they come up a lot below. Retrieval is the step where the system looks up documents and hands them to the agent before it starts answering. RAG is the whole setup: “first find it in the knowledge base, then answer from what you found.”

TL;DR (in 30 seconds)

• An agent’s error most often happens before generation. The agent was handed the wrong document and diligently did its job on it.
• Stanford and Together AI ran six commercial chatbots through 2,100 questions on fresh BBC news over 14 days. In the authors’ words: retrieval, not reasoning, failures drive over 70% of all errors.
• The breakdown of causes: retrieval failure — 38.8%; a source “on the same topic, but not about that” — 32.7%.
• The best systems score over 90% on multiple-choice questions. In free-response form accuracy drops by 11–13 percentage points, and by 16–17 across the cohort.
• Accuracy is lowest in Hindi: according to the study, 79% on average versus 89–91% in the other languages, in the same multiple-choice format. The cause isn’t the model’s language ability but a skewed index: on Hindi queries, models cite English Wikipedia more often than any Hindi-language news outlet.
• Slip a single false premise into the query and accuracy, according to the same paper, falls from 88–96% to 19–70%, even when the system was explicitly allowed to answer “insufficient information.” The most vulnerable model took a fabricated fact on faith 64% of the time.
• The practical takeaway: before you pay for a more expensive model, spend the same budget on retrieval. It pays off better.

2,100 questions, six chatbots, fourteen days

In May 2026 Mirac Suzgun’s group at Stanford, together with Together AI, posted the paper “Evaluating Commercial AI Chatbots as News Intermediaries” (arXiv:2605.22785). In July The Batch covered it under the title Web Retrieval Flusters LLMs.

The design is simple and transparent. For fourteen days in a row, from 9 to 22 February 2026, the authors took fresh BBC News stories from six regional services and generated questions from them about events that had happened just a few hours earlier. That made 2,100 questions. Then six chatbots with web search turned on tried to answer them; the cohort in the study was Gemini 3 Flash and Pro, Grok 4, Claude 4.5 Sonnet, GPT‑5 and GPT‑4o mini.

Testing on fresh news closes the main loophole. The model couldn’t have known the answer from its training data: the event happened after training. The only route to a correct answer is to find the right story and extract the fact from it. That is exactly the setup any agent of yours works in on corporate data, except that instead of the BBC it’s your contract database.

The first result looks reassuring. On questions with five answer options, the study’s results were: Gemini 3 Flash at 95.6%, Grok 4 at 95.0%, Gemini 3 Pro at 93.7%, Claude 4.5 Sonnet at 90.4%. GPT‑5 scored 85%, GPT‑4o mini 69%.

But then the authors removed the answer options and asked the models to answer in their own words. The best systems lost 11–13 percentage points; across the whole cohort the drop was 16–17. So a noticeable share of those “ninety-something percent” rested on guessing the right option from a list — roughly what a schoolkid does on a test when they only half know the material.

The key number: over 70% of errors are retrieval

Next the authors classified the causes of the wrong answers. Each failure was sorted into categories: didn’t find it, found the wrong thing, didn’t understand, mixed up the timeline.

The paper’s abstract puts it this way: retrieval, not reasoning, failures drive over 70% of all errors. When a model manages to retrieve a correct source, in the vast majority of cases it extracts the right answer from it. The whole difficulty is landing on the right source in the first place.

The Batch gives the breakdown for the two largest categories:

Cause of error	Share
Retrieval failure — the story that was needed wasn’t found at all	38.8%
A source on the topic was found, but with different details — it doesn’t answer the specific question	32.7%
All other causes combined — comprehension, timeline and so on (by subtraction; not given separately in the source)	~28.5%

The second category isn’t “found nothing.” It’s “found something reasonable and relevant, just not the right thing.” The authors call the answers built on such sources “smart wrong” answers. From the outside the answer looks well grounded: the agent cites a real document, the document really is on topic, and it really does contain figures. The figures just refer to a different period, a different region or a different entity.

This is the most expensive category for a business, because it doesn’t look like an error. Any check catches an empty answer, but a confident answer with a live link to the wrong document slips past a human reader without resistance.

Why “let’s get a smarter model” doesn’t work

The standard reaction to an agent’s error: the model is dumb, let’s move up a tier. The paper’s data says most of that money will miss the problem.

The logic is almost arithmetic. An agent’s answer to a question that wasn’t in its training data rests on three steps: receive a well-formed question, find a relevant document, extract the fact from it. The model is responsible for the third step and a little for the first. Over 70% of errors sit in the second.

A model upgrade improves what already works at 90+ percent and doesn’t touch what breaks.

The authors draw a direct conclusion from this: improving retrieval will most likely yield a bigger quality gain than scaling model parameters — at least where queries involve timely information. They name three directions: index coverage (which documents make it into search at all), source ranking (the order in which the results are presented to the model) and handling of queries in languages other than English.

The Hindi story is telling. On the same multiple-choice questions every model showed its worst accuracy in Hindi: according to the same paper, 79% on average versus 89–91% in the other languages. And when answering Hindi queries, the models cited English Wikipedia more often than any Hindi-language news outlet. This isn’t “the model is worse at Hindi.” It’s an index built by the English-speaking world for the English-speaking world, and a search that slides onto an English-language source when the query isn’t in English. The exact same mechanism will kick in on your own document base in any language other than English if the embedding model and the search stack are left at their defaults.

Five ways to hand an agent the wrong document

What follows are the failure modes I regularly see in production pipelines. The first and the fifth map directly onto categories from the study; the other three come from hands-on work building retrieval systems.

1. Found something relevant, but not what was needed

That same 32.7% category. The agent is looking for the contract clause on payment terms, and the index returns the general page of the “Terms of cooperation” section. The topic matches, the specifics aren’t there. Or the question was about an article and what came back was its author’s profile: same words, different thing.

The mechanism is simple: relevance to the topic and relevance to the question are different things, and most search configurations optimize for the first by default.

2. A mixed-up entity

One name can be a company, a product and a project all at once. The agent pulls perfectly correct data — just about a different entity. Inside a single client this happens all the time: a pricing plan and a service of the same name, a legal entity and a brand, a project name and a folder name.

This error is the hardest to catch in review, because everything in the text is true — except what it’s attached to.

3. Semantic similarity instead of an exact match

The most typical engineering trap. Vector search by design looks for “about the same thing,” not “exactly this thing.” On a conversational query that’s an advantage; on an exact identifier it’s a source of errors.

A SKU, an order number, a document version, a regulation number, a plan name, a date — these are exactly the entities where a semantically close result is a wrong result. And they are exactly the entities where an error costs the most. Similarity of meaning beats the exact match, and the agent confidently takes the neighboring row.

4. The source is out of date

The document was correct once. The policy was rewritten, the plan changed, the procedure updated — and the old version stayed in the index with no penalty for age. Search dutifully returns the most relevant fragment, and the most relevant one turns out to be last year’s.

There’s a second half to the problem, too: the source was updated, but nobody reindexed it. On paper the data is fresh; in search it’s the old copy.

5. No “not found” state

The most architectural of the five. If the system has no built-in permission to say “there isn’t enough evidence,” it optimizes for a complete answer, not a reliable one.

The study measured this effect separately: a single false premise was slipped into the questions — a wrong surname, a shifted date, an invented detail — and the option “insufficient information to answer” was added. According to the study, accuracy fell from 88–96% to 19–70% depending on the model. The most vulnerable one took a fabricated fact on faith 64% of the time. In other words, two-thirds of the time the system didn’t notice that the question was built on a falsehood and dutifully went looking for confirmation of something that never happened.

The authors also noticed a separate oddity, which they called the detection-accuracy paradox: the best detector of false premises ranked only second in its ability to still give the right answer. Spotting the catch in a question and getting out of it appear to be two different abilities, and one doesn’t bring the other along.

An agent that can’t decline to answer is more dangerous than one that declines once too often. A similar mechanism is at work in other systems too — I walked through it with the example of how an agent breaks its own metric, when the system rewards the fact that the work got finished rather than its quality.

What to do about it

The order below runs from the cheapest to the most expensive to implement.

Separate retrieval from the answer. First the agent returns a structured retrieval result: which documents were found, with what date, from which source. Only then comes generation from those fragments. As long as the two steps are fused into one call, you can’t tell where exactly it broke, and any “prompt improvement” is a shot in the dark.

Check that the fragment matches the question, not the topic. Between “this text is about payment terms” and “this text contains the answer to the question about the payment term under contract No.” lies a chasm. A reranker that scores exactly the latter removes most of the “smart wrong” category.

Hybrid retrieval instead of pure embeddings. Split the load: keyword search (BM25, the classic algorithm that ranks by word match) takes exact identifiers, codes, names and versions; vector search takes conversational and paraphrased queries; a reranker orders the candidates; metadata filters screen out another client’s data, the wrong period and the wrong version. This isn’t “better embeddings”; it’s spreading risk across mechanisms, each of them strong where another is blind.

Then come two cheap things that for some reason get done last. Freshness should be a separate ranking signal: recency scoring plus a staleness policy, because a document without an indexing date is a document without an expiry date. And provenance — the fragment’s passport: source, version, indexing time, position in the file — is needed on every fragment: an answer without a valid reference simply shouldn’t reach the user. It’s the cheapest way to turn an invisible error into a visible one, and it almost always gets put off “for later.”

Test retrieval separately from generation. Build a set of real queries, record for each one which document is the right answer, and measure one thing: whether that document lands in the top-k, that is, in the handful of results that search actually passes to the agent. Not “is the agent’s answer good” — whether the right file was found. Thirty queries are enough to see the picture, so this is half a day’s work, not a quarter’s.

This metric doesn’t depend on the model at all, so it’s the only one that shows the pure effect of changes in chunking (how a long document is cut into fragments), indexing and ranking. It also answers the question of whether the next upgrade makes sense: if the right document doesn’t make it into the context a third of the time, the smartest model on the market won’t save that third.

The logic is the same as in agent self-improvement: as long as the result can’t be checked automatically, improvement has nothing to push off from.

Last comes an explicit “didn’t find enough evidence” state and an escalation route to a human. This is already part of the answer to the question of what blast radius your agent has: a system that knows how to stop limits the consequences of its own error.

The cheapest check, and one worth starting tomorrow: take the last five complaints about the agent and look at which documents it was given, not at what it answered. If a separate person is responsible for the agent at your company, this paragraph is for them, and the thing to ask for is the context logs, not examples of bad answers.

The chain where it breaks

If you boil it all down to one diagram, the path from data to answer looks like this:

Source → Extraction → Normalization → Storage → Indexing → Retrieval → Reranking → Answer

The model is responsible for the last link. The study says most errors sit in the two before it. And in my practice the first three are most often to blame: a raw dump of unstructured files gets poured into the vector database, the atomic unit of indexing is picked at random, an update to the source doesn’t trigger reindexing. The data seems to be there; the agent just can’t reliably get to it.

So diagnosis starts with the question “what exactly landed in the agent’s context when it got it wrong,” and only then gets to the choice of model. The answer is almost always in the logs — if you keep them. I evaluate new AI tools by roughly the same logic: first I look at what the system shows about its own work, and only then at how cleverly it talks.

RAG doesn’t remove the problem, it moves it

An agent that got it wrong most often didn’t “think badly.” It opened the wrong page and faithfully retold what was on it. A smarter model in this scenario changes only the quality of the wording — which is exactly why the upgrade so often disappoints: the same errors, put more professionally.

RAG on its own doesn’t save you either; it changes the nature of the error. Instead of “the model made it up” you get “the model confidently retold the wrong document.” The second can at least be traced to a specific file, but only if the system has provenance, dates and separate retrieval metrics. Without them you’re still looking at a smooth paragraph with no way to check where it came from.

In my experience, most “why did the agent get it wrong” post-mortems end in three log lines about which file made it into the context.

FAQ

Does RAG solve the hallucination problem?

No, RAG changes its nature. Instead of a made-up fact you get a real fact from the wrong document. That’s better, because the error can be traced to a source — but only if every fragment carries provenance: a reference, a version and a date.

How do I tell that the problem is retrieval and not the model?

Take a few dozen queries the agent got wrong and look at what context it was actually given. If the right document was in the context and the answer is wrong, it’s the model. If the document wasn’t there, or the wrong one was, it’s retrieval. In the Stanford and Together AI study the latter accounted for over 70% of cases.

What is hybrid retrieval, and why aren’t embeddings alone enough?

It’s a combination of keyword search (BM25) for exact matches, vector search for meaning, a reranker to order the candidates, and metadata filters. Pure vector search by design finds “about the same thing,” not “exactly this thing,” so it loses on SKUs, numbers, versions and dates — the places where an error costs the most.

Which is cheaper: improving retrieval or getting a more expensive model?

The study’s authors say outright that improving retrieval will most likely yield a bigger accuracy gain than scaling model parameters — at least for queries about timely information. An extra argument: you don’t control what a model change gives you, while changes to the index, ranking and metadata stay yours.

Can a large context window replace retrieval?

If the whole useful corpus fits in the context — yes, and that’s often simpler than a pipeline. The boundary is best set by measurement: run the same task at 10, 50, 200 and 500 thousand tokens and find the point where quality starts to drop. That point shows when retrieval is really needed.

What if the user’s query contains a false premise?

Provide an explicit “not enough data” option and teach the system to use it. In the study, models were given questions with one incorrect detail, with the answer “insufficient information” added as an option — accuracy on that set fell from 88–96% to 19–70%, and the most vulnerable model took a fabricated fact on faith 64% of the time. Without permission to decline, the system optimizes for a complete answer, not a truthful one.

How do I test retrieval separately from generation?

Assemble a set of real queries with the right documents known in advance and measure whether the needed document lands in the top-k results. This metric doesn’t depend on the model, so it shows the pure effect of changes in indexing, chunking and ranking.

---

Sources: Suzgun M., Shen E., Bianchi F., Spangher A., Icard T., Ho D. E., Jurafsky D., Zou J. “Evaluating Commercial AI Chatbots as News Intermediaries,” arXiv:2605.22785, 21 May 2026; The Batch’s write-up, “Web Retrieval Flusters LLMs”, 24 July 2026.

Published by Andrew Maryasov, AI consultant, founder of Auspex and Grow2.ai; implementations are run by Auspex (CRM) and Grow2.ai (AI agents).