Someone in ops asks the internal assistant when the Henderson contract comes up for renewal. It replies with a tidy paragraph and a date. The date is wrong by four months, and nobody catches it, because the answer reads exactly like the fifty correct ones before it.
Which is what makes this failure expensive. A tool that breaks loudly gets fixed. A tool that gives a smooth, well-formatted, entirely incorrect answer gets trusted, and the mistake travels. When an AI assistant gives wrong answers of this kind, the instinct is to blame the model and go shopping for a better one. That is almost always the wrong place to look.
The model is the last thing to check
Ask the same question again, but paste the actual contract into the chat first. If the answer is now right, the model was never the problem. It answered correctly the moment it could see the correct text.
Two minutes, and it settles the argument. What the test reveals is that the assistant isn't reasoning badly at all. It is reasoning perfectly well over the wrong source material, because something between your documents and the model handed it the wrong pages.
That something is retrieval, and it is where nearly all of these failures live.
How retrieval actually decides what the model sees
Most business assistants are built on retrieval-augmented generation. The mechanics are unglamorous: your documents get chopped into chunks of a few hundred words, each chunk gets converted into a numerical representation of its meaning, and all of those go into a searchable index. When a question arrives, the system finds the chunks that look most similar to the question, staples them into the prompt, and asks the model to answer from them.
Notice what that means. The model never sees your document library. It sees maybe twenty fragments that a search step decided were relevant, and it has no way of knowing whether the right fragment was among them. If retrieval hands over twenty near-misses, the model will write a confident answer from twenty near-misses. It has nothing else to work with.
So the useful question stops being "why did it say that" and becomes "what did it read".
The chunk that forgot which company it was about
Convincing wrong answers usually trace back to preprocessing, before anyone has asked anything.
Split a document into chunks and each chunk loses the context it was sitting in. Anthropic's engineering team uses a financial filing as the example: a chunk reading "The company's revenue grew by 3% over the previous quarter" no longer says which company, or which quarter. As a standalone fragment it is nearly meaningless, and it will match a question about almost any company's growth just as well as the right one.
Multiply that by a few thousand contracts, policies and support articles that all use similar phrasing, and you get an index full of fragments that are individually plausible and collectively useless for telling one client from another.
The fix is to put the context back before indexing. Anthropic's approach prepends a short generated line to every chunk situating it in its source document, so that fragment becomes "This chunk is from an SEC filing on ACME corp's performance in Q2 2023" followed by the original text. On their tests, doing that cut the rate at which the right chunk failed to appear in the top twenty results by 35%, from 5.7% to 3.7%. Combining it with keyword search took the reduction to 49%, and adding a reranking step to 67%.
Those are their numbers on their datasets, so don't treat them as a promise about yours. Treat them as evidence that where you spend effort matters. The gains came from preprocessing and search design. Not one of them came from a bigger model.
Semantic similarity can't find an exact string
A second failure runs quieter. Meaning-based search is good at concepts and bad at identifiers.
Ask about error code TS-999 and a semantic index will happily return chunks about error codes in general, having understood the gist of your question and missed the only part that mattered. Same for invoice numbers, SKUs, product version strings, surnames, and every internal code your team uses daily.
The answer is a decades-old keyword ranking method, BM25, run alongside the semantic search, with the results of both merged. Old technology, still the right tool, and adding it is usually an afternoon of work rather than a rebuild.
Confidence is a design decision, not a personality trait
None of the above explains why the assistant sounded so sure. That part is on whoever configured it.
A model given weak context and no instruction about what to do with weak context will still produce an answer, because producing an answer is what it was asked to do. If you want "I can't find this in the documents I have access to, here is who to ask", you have to build that: a relevance threshold below which it declines, a prompt that permits declining, and a route to a person. We cover the operational side of that in AI agents that earn their keep from week one, and it is a standing part of how we scope AI agent and integration work.
An assistant that says "I don't know" twice a day is doing its job. One that has never said it is hiding something.
Build the question set before you build anything else
This is the step almost everyone skips, and it costs less than any of the fixes above.
Write down thirty questions your team genuinely asks, and next to each one write the answer you already know to be correct, plus the document it comes from. That list takes an afternoon. It is worth more than any amount of tuning, because it turns "the assistant seems a bit unreliable" into a number you can watch move.
Then run it and score two things separately:
- Did the right document reach the model? Log which chunks were retrieved for every question. If the correct source isn't in there, no amount of prompting will save the answer.
- Given the right document, was the answer right? Only failures that survive step one are model or prompt problems.
Nearly everyone who does this for the first time finds the damage sits overwhelmingly in the first bucket. That is good news, because retrieval problems have known fixes: better chunk boundaries, context added at index time, keyword search alongside semantic, a reranker on top.
What to do tomorrow
Take the last three wrong answers anyone reported and, for each one, find out which chunks the system retrieved. If your setup doesn't log that, making it log that is the highest-value hour of engineering work available to you this week.
You will probably discover the same thing we find on most of these engagements. The model was never confused. It was answering a good question from the wrong page, exactly as instructed, and doing it beautifully.
