How does private RAG over company documents work?

What private RAG is – and why it is more than a chatbot

A language model on its own knows nothing about your company: nothing about your contracts, processes, offers or project files. Ask it anyway and it produces plausible-sounding guesses – often wrong. RAG solves this by supplying the model, for every question, with the relevant excerpts from your own documents. The model then formulates its answer not from memory but from the context it was given.

«Private» here means two things. First, your documents never leave the environment you control – they are neither used to train third-party models nor sent to a public service. Second, each user sees only what they are allowed to see. A well-built private RAG is therefore less a «chatbot» than a searchable, answering knowledge layer on top of your existing document estate.

The RAG pipeline step by step

Technically, RAG has two phases: a one-off (and then ongoing) preparation of your documents, and a lookup performed for every question. Preparation builds the searchable index; the lookup fetches context and produces the answer.

• Ingestion & cleaning: PDFs, Office files, wiki pages, emails or tickets are read in and converted into plain text plus metadata (source, date, access rights).
• Chunking: long documents are split into smaller, sensibly cut passages – large enough to carry meaning, small enough to be retrieved precisely.
• Embeddings: an embedding model turns each passage into a vector – a numeric representation of its meaning. Similar content ends up close together in vector space.
• Vector index: the vectors go into a vector database, often combined with classic keyword search (hybrid search) so that exact terms and reference numbers also hit reliably.
• Retrieval & re-ranking: the most fitting passages for the question are searched; a re-ranker sorts them by true relevance before the best ones are passed on.
• Generation with citations: the question plus retrieved passages go to the language model, which formulates an answer – ideally with source references, so every statement stays verifiable.

What «private» concretely means: deployment models

Privacy is not a property of RAG itself but a question of where the model, index and data run. There is a spectrum – from fully local operation to walled-off cloud endpoints. The right point depends on data sensitivity, the answer quality you need and your internal IT resources.

• Fully local / on-premises: open-weight models (e.g. Llama, Mistral or Switzerland's Apertus from ETH/EPFL) and the vector database run on your own hardware. Maximum control, no data outflow – but you carry operations, GPU cost and maintenance yourself.
• Private cloud in Switzerland / EU: a dedicated account or Swiss data centre with contractually guaranteed data residency and an assurance that your data is not used for model training. A good compromise between quality, effort and data sovereignty.
• Confidential cloud endpoint from a major provider: hosted models with zero-retention and data-processing agreements. Fast to start and powerful, but requires careful contract and location review.
• Important in all variants: the vector index contains fragments of your documents in plain text. It deserves the same protection as the original files and must be encrypted, access-restricted and kept in the same legal jurisdiction.

Swiss and DACH perspective: law, data protection, access

Company documents often contain personal data and trade secrets. In Switzerland the revised Data Protection Act (revDSG / nFADP) applies, supervised by the EDÖB; never call this «GDPR». If EU individuals are affected, the GDPR applies too, and – for certain AI uses – the extraterritorial EU AI Act. A private RAG makes compliance easier but does not replace legal assessment.

• Permission-aware retrieval: access rights from your source systems must flow through to the answer. Otherwise the RAG «leaks» content a user could never open in the original.
• Data residency: keep documents, index and model inference demonstrably in Switzerland or the EU where contracts or sector rules require it.
• Traceability: source citations, query logging and clear purpose limitation support accountability duties under revDSG and the transparency requirements of the EU AI Act.
• Data minimisation: index only documents needed for the purpose, and define deletion and refresh cycles so that outdated or to-be-deleted content is no longer answered from.

Retrieval quality: the difference between useful and misleading

A RAG system is only as good as what it retrieves. Give the model the wrong or incomplete passages and it will produce a convincing but wrong answer. Most of the quality work therefore sits not in the language model but in retrieval.

• Thoughtful chunking: cut at semantic boundaries (headings, paragraphs) rather than rigidly by character count; do not tear tables and lists apart mid-way.
• Hybrid search: combine semantic vector search with keyword search so that technical terms, product names and reference numbers hit exactly.
• Re-ranking and context limits: retrieve more candidates, then pick the best via a re-ranker – avoiding filling the context with irrelevant material.
• Allow «I don't know»: the system should openly say when the documents provide no basis, rather than guess. This is the most effective brake on hallucination.
• Measure, don't hope: evaluate regularly with a set of real questions and vetted expected answers (retrieval hit rate, faithfulness of the answer to its sources) before and after every change.

RAG, fine-tuning or long context – which when?

RAG is often confused with fine-tuning. They solve different problems and can be combined.

• RAG: delivers current, company-specific knowledge with source citations. Ideal when content changes often and traceability matters. New documents means re-indexing, not retraining.
• Fine-tuning: bakes style, format or domain tone into a model, but adds no reliable, verifiable factual knowledge. More costly and needed again with every knowledge change.
• Long context without retrieval: for a handful of documents you can feed them directly to the model. It does not scale to thousands of files, costs more per question and loses precision at very long context.
• Pragmatic rule: start with RAG. Add fine-tuning only if tone or format cannot be achieved otherwise – never to «store» factual knowledge.