RAG and knowledge bases: grounding an LLM in your own documents

How RAG works: retrieve, augment, generate

RAG runs in two phases. An indexing phase prepares your knowledge base: documents are split into passages (chunks), each chunk is turned into a numerical vector (embedding) that captures its meaning, and the vectors are stored in a vector database. At query time the user's question is embedded the same way, the system retrieves the most semantically similar chunks, and those passages are inserted into the prompt as context. The LLM then produces an answer grounded in the retrieved text rather than in its training memory.

• Chunking: split documents into coherent passages so retrieval returns focused, citable snippets.
• Embeddings: convert text and questions into vectors so semantic similarity, not just keywords, drives retrieval.
• Vector store and retrieval: find the best-matching passages, often combined with keyword search (hybrid search).
• Augmentation and generation: pass the retrieved passages plus the question to the LLM, which answers and can cite sources.

Why grounding reduces hallucinations

A standalone LLM answers from patterns learned during training. When it lacks a fact, it may still produce a fluent but wrong answer — a hallucination. RAG changes the task from 'recall' to 'read and summarise': the model is handed the relevant source text and asked to answer from it. This narrows its freedom to invent, keeps answers current as your documents change and — crucially — lets the system show which passage an answer came from, so humans can verify it. Grounding does not remove errors entirely: if retrieval returns the wrong passage, or the prompt does not instruct the model to stay within the provided context, mistakes still occur.

Building a knowledge base that actually works

The quality of a RAG system is decided less by the model than by the knowledge base behind it. Clean, well-structured, up-to-date documents beat a bigger model running on messy data.

• Curate sources: manuals, policies, FAQs, contracts, wiki pages — only content you trust and can maintain.
• Chunk sensibly: passages that are too large dilute relevance, too small lose context. Respect natural structure (headings, sections).
• Keep it fresh: re-index when documents change; stale chunks are a common cause of wrong answers.
• Add metadata: language, department, effective date, access level — so retrieval can filter (e.g. only current, only DE).
• Handle permissions: a RAG assistant must respect who is allowed to see which document.

RAG, fine-tuning or a bigger context window?

These approaches solve different problems and are often combined. Illustrative examples for a Swiss SME: an internal assistant that answers staff questions from the HR handbook and cites the clause; a support bot grounded in current product documentation in German and French; a sales assistant that pulls the latest price list instead of guessing.

• RAG adds knowledge: best when facts change often, must be citable, or are too numerous to fit in a prompt. You update by editing documents, not retraining.
• Fine-tuning shapes behaviour and style: it teaches format, tone or a narrow skill — not a reliable way to store many facts.
• Large context windows let you paste documents directly: simple for a few files, but costly and slower at scale — and you still have to choose what to include, which is exactly what retrieval does.

Common mistakes and how to avoid them

• Treating RAG as plug-and-play: retrieval quality must be tested and tuned (chunking, hybrid search, re-ranking).
• No source citations: without showing passages, users cannot verify and trust erodes.
• Ignoring evaluation: measure whether answers are grounded in the retrieved text, not just whether they sound fluent.
• Letting the model answer beyond the context: instruct it to say 'not found in the sources' rather than guess.
• Forgetting data governance: sensitive documents in an index still need access control and a lawful basis.

The Swiss and DACH perspective: data protection by design

For Swiss organisations, a RAG knowledge base often contains personal or confidential data, so data protection is part of the design, not an afterthought. Under the revised Federal Act on Data Protection (revDSG/nFADP), supervised by the EDÖB, you need a lawful basis, transparency and appropriate security for the personal data you index and process. If you serve EU users or deploy AI systems there, the EU AI Act can apply extraterritorially.

• Data residency and provider choice: know where indexing, embedding and generation happen; Swiss or EU hosting may be required for sensitive data.
• Access control: retrieval must respect document permissions so the assistant never surfaces data a user is not allowed to see.
• Transparency and human oversight: tell users they are interacting with KI, show sources, and keep a human in the loop for consequential answers.
• Data minimisation: index only what the use case needs; do not pull special-category data into the knowledge base without justification.