Chatbots that don't make things up: how retrieval works

A plain chatbot answers from memory: whatever it absorbed during training. That memory is general, it stops at a certain date, and it has never seen your organisation’s own documents. When it doesn’t know something, it may produce an answer that sounds right anyway. Retrieval changes the order of work. The assistant searches your own documents first, answers from what it found, and shows you the sources. That doesn’t make it perfect, but it makes it checkable, which is most of what you need.

The technique has a clunky name, retrieval-augmented generation (RAG), from a 2020 research paper. If you’d like the background on the models themselves, start with what a large language model is.

How it works, in three steps

1. Search. Your question is used to search a collection of your documents: policies, handbooks, product sheets, past answers. The search usually matches on meaning rather than exact words, so “Can I bring my dog?” can find a paragraph headed “Assistance animals and pets”.
2. Answer. The best-matching passages are handed to the language model with an instruction along the lines of: answer using only these passages, and if they don’t contain the answer, say so.
3. Show the sources. The answer comes back with links to the passages it used, so the person reading it can check.

Why it helps, and why it still slips

Retrieval helps for three reasons. The model has the right text in front of it. It has been told to stick to that text. And because the sources are shown, a person can check an answer in seconds instead of taking it on trust. Updating what the assistant knows means updating documents, not retraining anything.

It still goes wrong, in predictable ways:

• The search finds the wrong passage, such as last year’s policy or a product with a similar name.
• The right passage isn’t in the collection at all, and the model fills the gap.
• The model misreads a passage it was given, for example by dropping an “except” or a date.
• Two documents disagree, and it picks one without saying so.

The evidence matches. In a study published in 2025, researchers tested commercial legal research tools built this way. They made things up less often than a general-purpose chatbot, but still between 17% and 33% of the time. Legal questions are harder than most, and your FAQ is probably kinder, but the lesson holds: retrieval reduces wrong answers, it doesn’t end them. We go into the reasons in why AI gets things wrong.

Chunking: how you cut the documents matters

Before documents can be searched this way, they’re split into chunks, usually a few paragraphs each. The model can only read so much at once, and smaller pieces match questions more precisely.

How you cut them changes the answers. Cut too small and a rule gets separated from its exception. Cut too big and the search returns a whole chapter with the answer buried somewhere in the middle. Some good habits:

• Split along the document’s own headings, not every few hundred words.
• Keep tables, lists and their headings together.
• Label every chunk with the document title, section and date, so the source link makes sense to a reader.

Keep the documents current

An assistant is only as current as its documents. If last year’s price list and this year’s are both in the collection, it may quote either, with total confidence. Give each set of documents an owner. Archive old versions rather than leaving them next to the new ones. Re-index automatically when files change, and show the date of the source beside each answer. Retrieval turns a content problem you could ignore into one you can’t.

Permissions: answers only from what you’re allowed to see

An assistant should only answer from documents the person asking is allowed to open. If HR files sit in the same collection as the staff handbook and the assistant ignores permissions, anyone can ask it about anyone’s disciplinary record. Good systems check permissions at search time, using the same access rules as your file storage. Microsoft, for instance, says Microsoft 365 Copilot only shows people organisational data they already have at least view access to.

The flip side is that the assistant will find things people could technically open but never knew were there. If folders were shared with everyone years ago, tidy that up before you switch anything on. And if the documents contain personal data, the usual rules apply: our UK GDPR checklist covers them.

Test it with real questions

Before launch, collect a few dozen real questions from your inbox, your phone notes and your colleagues, including the awkward ones. Write down the right answer to each and where it lives. Then run them all and check three things:

1. Did it find the right source?
2. Does the answer say what the source says, no more and no less?
3. Did it say “I don’t know” when the answer wasn’t there?

Keep the list. Run it again whenever you change the documents, the chunking or the model, so you notice when something that used to work stops working.

Teach it to say “I don’t know”

An assistant that says “I can’t find that in our documents, here’s who to ask” is doing its job. Tell it to answer only from the passages it was given, and set it up so that weak matches produce “I don’t know” rather than a guess. Hand those questions to a person, and log them. A list of what people asked and couldn’t get answered is one of the most useful things the project produces, because it tells you which documents to write next.

If you’re thinking about an assistant for your own documents, we’d start with your questions, not the software.