Introduction: the invisible risk

Hallucinations – statements generated by an LLM that are factually incorrect, and are among the most critical risks when working with generative AI. (For more on trust, see my other blogpost.)

RAG systems (Retrieval Augmented Generation), often promoted as the solution to hallucinations, introduce their own new sources of error. They promise to ground answers in your own, trusted data, but can still go dangerously off track.

The good news: hallucinations are not an inevitable part of technology. They are primarily a data-quality and architecture problem, and therefore manageable

What are hallucinations – and why are they dangerous?

A hallucination occurs when an LLM generates facts that are wrong but presents them with confidence and authority. Unlike a «normal» error, hallucinations are particularly tricky because they:

  • Sound plausible: the answer appears to fit perfectly and feels authoritative.
  • Look stylistically correct: wording, tone and structure are polished and professional.
  • Are hard to spot: even experts may miss them when reading quickly.
  • Exploit trust: users assume the system relies on verified data.

There are plenty of real-world examples, from invented sources to incorrect legal references and entirely fabricated process steps.

How do hallucinations arise from a technical perspective?

Large Language Models (LLMs) fundamentally work by predicting the most likely next word based on the preceding context. They are optimised to produce an answer – not to say: «I don’t know».

This leads to a critical issue:
Even if the underlying information is incomplete or entirely missing, the model will still construct an answer. That is exactly where hallucinations arise.

To counter this systematically, there are two core strategies:

  1. Provide context:
    By supplying the right, relevant context (for example via RAG), we increase the likelihood that the model draws on correct information, and therefore produces a correct answer.

  2. Detect out-of-distribution / «I don’t know» cases:
    If a query lies outside the system’s knowledge, it should recognise this and choose not to answer, rather than invent something uncertain or wrong.

RAG systems: promise versus reality

RAG systems aim to address exactly this problem. The idea:

  1. Instead of letting the LLM answer purely from its «learned world knowledge»,
  2. the system first searches a (customer-specific) knowledge base for relevant information,
  3. and the LLM formulates the answer based on the retrieved documents and the prompt.

A well-designed RAG will drastically reduce hallucinations.
A poorly designed RAG introduces new sources of error, and makes debugging significantly harder.

Typical sources of error in RAG systems

Problem 1: Retrieval fails

Before the LLM answers, it must receive the relevant documents. This is where many of the most fundamental errors occur:

Relevant information is not found:

  • The search algorithms do not recognise important documents as relevant.
  • Essential content is hidden inside tables, PDFs or poorly structured files and is not captured or only partially.
  • The LLM’s context window is filled with irrelevant material and critical information gets lost.

Irrelevant or outdated documents are used:

  • The system retrieves a press release from 2019 that is long outdated.
  • Contradictory information from different sources ends up in the same context.

Domain-specific nuances are not captured:

  • Embeddings fail to represent key distinctions in the domain.
    Example: on a public administration website, «register» almost always means «register your residence with the local authority», not «sign up for a salsa class at the community centre».

The result: the LLM builds on incorrect or incomplete context – and hallucinates.

Problem 2: The «Generation» -step misinterprets

Even if retrieval works well and delivers the right documents, the LLM may still process the information incorrectly:

Misinterpretation:

  • The LLM fails to grasp nuances in a legal text.
  • Conditional statements («If X, then Y») become absolute rules.
  • Exceptions are ignored or over-generalised.

Invalid combinations of information:

  • The LLM mixes statements from different documents in creative but incorrect ways.
  • It «infers» B from A even when that inference is not valid.
  • It tries to «update» outdated information intelligently, and gets it wrong.

Filling gaps with world knowledge:

  • The retrieved document only partially answers the question.
  • The LLM completes the missing parts using its general world knowledge.
  • The result is a mix of verified facts and invented content.

We have the least direct influence on this generation step. We can only choose an appropriate model and design strong prompts. This makes everything before and after even more important.

Three essential strategies to minimise risk

If hallucinations are primarily a data-quality and architecture problem, we can systematically fight them. From our experience, three strategies are essential:

1. Data quality as the foundation

A RAG system is only as good as the data it relies on.
The old IT principle applies perfectly: garbage in, garbage out.

Key principles:

  • Align sources with actual needs:
    Not everything stored in your intranet helps answer real user questions.
    The knowledge base should be shaped by real use cases – not by your organisation’s folder structure.

  • Enrich documents with additional context:
    Documents often need to be split (cut up) to fit into an LLM.
    Important context can be lost, and must be reintroduced explicitly (e.g. metadata, headings, validity ranges).

  • Ensure up-to-date information, once only:
    Information should ideally be maintained once, centrally.
    Multiple copies across systems make consistency nearly impossible.
    For RAG systems: clear «single sources of truth», not five slightly different versions of the same process.

2. Fallback mechanisms and uncertainty logic

A good system knows when it does not know, and behaves accordingly.
Our principle: a conservative «I cannot help you with this» is better than an invented answer.

In practice, this means:

  • Define uncertainty thresholds:

    • Thresholds across retrieval steps (e.g. vector distances, relevance scores).
    • Below these thresholds: no answer – instead, a transparent notice to users.

  • Routing for critical topics:

    • Determine which topics are «critical» (e.g. taxes, health, legal matters – depending on the client).
    • For these topics, use defined fallbacks:
      «Unfortunately, I cannot calculate your tax burden. Please use the official tax calculator.»

  • Label out-of-scope queries:

    • If something lies outside the system’s knowledge, it should say so, instead of answering with half knowledge.

3. Empower users – don’t mislead them

Even if hallucinations could be minimised technically, language remains ambiguous.
Example: «My dog brought me a ball. I kicked it» – What was kicked? Language is never 100% unambiguous.

This means: not every question will ever be answered 100% correctly. Therefore:

a) Provide sources for every answer

  • Show which documents the answer is based on.
  • Link directly and precisely (table, section, paragraph).
  • Be transparent when an answer draws from multiple sources.

b) Make «Confidence Scores» visible

  • Where possible, communicate - even in simple form how reliable the system is.
  • Long term, we aim to quantify and display «Confidence Scores» more precisely.
    Until then: better simple, honest indicators than false precision.

c) Honest disclaimers, not marketing language

The best technical solution is useless if users blindly trust the system.

  • State clearly in the UI: «This system can make mistakes.»
  • Explain what the system is suitable for – and what it is not.
  • Avoid marketing phrases that raise unrealistic expectations.
  • Avoid legalistic «cover your back» language that nobody reads.
    The goal: clear, honest communication.

Responsible AI is achievable

For me, three central points matter:

  1. Hallucinations are manageable:
    They are not inevitable – they result from data quality and system design. Both are in your control.

  2. Transparency beats perfection:
    You will never avoid hallucinations entirely.
    But you can design your system so that users can recognise, understand and handle errors.

  3. It is also a management topic, not just a technical one:
    The most important decisions – data quality, processes, responsibilities must be made by leadership, not by the IT team alone.

The use of RAG-based chatbots comes with opportunities and risks.
Those who understand the risks and address them systematically can leverage the opportunities – responsibly and safely.