Why AI gets things wrong, and how we design around it

AI tools get things wrong for a small number of predictable reasons, and most of them can be designed around. The root cause is simple: a language model predicts likely words. It doesn’t look facts up. So the useful question isn’t “is AI accurate?” but “what happens when it’s wrong, and who notices?”

It predicts, it doesn’t look things up

A large language model (the engine inside ChatGPT, Copilot, Gemini and Claude) is trained on huge amounts of text to do one thing: given some words, predict what comes next. Do that well enough and you get fluent, often correct answers. But there’s no database of facts inside it, and no step where it checks one. Our explainer on what a large language model is goes into more detail.

When a prediction sounds right but isn’t, people call it a hallucination: a plausible, confident statement that’s false. A case reference that doesn’t exist, a clause that isn’t in the contract, a quote nobody said. OpenAI researchers argued in September 2025 that one reason this persists is that standard training and testing “reward guessing over acknowledging uncertainty”. Think of a multiple-choice exam with no penalty for wrong answers: a guess might score, a blank never does.

Four more reasons it goes wrong

• Its knowledge is out of date. A model learns from text up to a cut-off date, then stops. Unless it’s connected to search or your own files, it doesn’t know about this year’s price list, last month’s policy change or a law that has come in since. It rarely mentions this.
• The question was vaguer than you thought. “What’s our holiday allowance?” depends on which policy, which year and whether someone works part-time. A colleague would ask. A model usually picks an interpretation and runs with it.
• Sums and counting are weak spots. Models read text in chunks called tokens, not single letters or digits, so they can stumble over counting words, adding up a column or comparing dates. Some tools quietly hand the arithmetic to a calculator or a bit of code, but don’t assume yours does.
• It sounds just as sure when it’s wrong. Tone tells you nothing about accuracy. In June 2025, dealing with two cases where lawyers had put fictitious case citations before the court, the High Court in England warned that answers from freely available AI tools “may make confident assertions that are simply untrue” and “may cite sources that do not exist”.

The jagged frontier: weak spots you can’t see

The awkward part is that AI’s weak spots aren’t where you’d expect. In a study of 758 consultants at Boston Consulting Group, published in Organization Science in 2026, those using AI on tasks it was good at completed 12.2% more tasks, 25.1% faster, with quality up by about 32% on average. But on a task deliberately chosen to sit outside what the AI could do well, consultants using it were 19 percentage points less likely to get it right than colleagues working without it.

The researchers call this a “jagged frontier”: AI is strong at some tasks and weak at others that look just as hard. They found the edge “is not immediately obvious to knowledge workers”, and that consultants tended to over-rely on the AI. The experiment used GPT-4 in 2023, and the authors describe the frontier as “expanding and changing”, so the line will have moved since. If skilled professionals can’t see where it sits, nobody should be judging it by feel. (For the productivity side of that research, see where AI actually saves time.)

How we design around it

None of this makes AI too unreliable to use. It means designing the work so that errors are rare, visible and cheap to fix. Six habits do most of the work.

Answer from your documents, and show the sources

Instead of asking the model what it remembers, give it the relevant passages from your own documents and tell it to answer only from those, citing each one. This is called retrieval, and how retrieval works explains it. It deals with most stale-knowledge problems, and the citations let a person check an answer in seconds. It’s how we build knowledge that cites itself.

Ask for a form, not an essay

Where the output feeds another system, ask for structured output: set fields in a fixed format, like a form. Then ordinary software checks it against validation rules. Is the date a real date? Do the line items add up to the total? Does that customer number exist? Anything that fails goes to a person instead of sailing through.

Make “I don’t know” an acceptable answer

Tell the model, in writing, that “I can’t find that in the documents” is a good answer. Then test that it actually says it. A tool that admits gaps is far more useful than one that fills them.

Have a person check anything that matters

Decide in advance what goes straight out and what gets reviewed. Internal first drafts can be loose. Anything sent to a customer, involving money or about a named person should be checked by someone who knows the subject.

Test it on real questions

Before launch, collect 30 to 50 real questions with answers your team agrees are right, including awkward ones and a few it should decline. This is called an evaluation set. Score the tool against it, then re-run the same set whenever the model, the instructions or the documents change. That turns “it seems fine” into a number you can track, and how to tell if an AI project worked covers what to measure.

Keep a log

Record each question, the sources used, the answer and who approved it. When something goes wrong, the log tells you whether the cause was a missing document, a vague question or a model update. Every failure you find becomes a new test question.

Wrong sometimes is not the same as useless

People get things wrong too, and most organisations already have checks for that: a second pair of eyes, a sign-off, an audit. The difference with AI is that its mistakes look exactly like its successes, so the checks have to be designed in from the start, not added after something has gone out the door. If you’d like help working out where those checks belong in your own work, tell us which job.