How AI answers from your own documents

Ask ChatGPT about the notice period in your framework agreement and you get a polite answer that has nothing to do with your contract. The model knows the internet up to a cut-off date. It does not know your contracts, manuals or quotes.

That is what RAG is for. The acronym stands for Retrieval-Augmented Generation. Before the model answers, a system looks up the matching passages in your own documents and hands them to the model. The answer is then based on your documents, and it comes with a source.

What is a RAG system?

A RAG system has two parts. A search finds the relevant passages in your documents, and a language model writes an answer from them. The model itself stays unchanged. It gets the right context with every question, much like a new colleague who is handed the right pages of the manual before answering.

That is why new documents are available right away, without anyone retraining a model. And every answer can be checked, because the source is right there.

How is a RAG system built?

Behind a simple chat window there are five steps.

Ingesting documents. PDFs, Word files, wiki pages or emails are pulled from their sources and turned into text. Tables and scanned documents need extra care here.
Splitting into chunks. The text is cut into smaller pieces, known as chunking. A good chunk still makes sense on its own, for example a full clause of a contract with its heading.
Computing embeddings. Every chunk is translated into a vector of numbers that captures its meaning. These vectors go into a vector database.
Searching. When a question comes in, it is translated the same way and the system fetches the chunks with the closest meaning. Good systems combine this with classic keyword search, so product codes and names are found too.
Answering. The language model gets the question and the retrieved chunks and writes an answer that points to the source. If it finds nothing that fits, it should say so.

Then there are permissions. Someone who has no access to the payroll list must not get it through the chat either. The search has to respect that at retrieval time.

What is RAG good for?

RAG fits wherever people look up answers in documents today. In companies that is usually four areas.

Internal knowledge base. Questions about processes, policies and products that always land with the same few colleagues.
Customer service. Suggested answers for the support team, based on manuals, FAQs and past tickets.
Contracts and quotes. Finding clauses, looking up deadlines and using old quotes as a template.
Technical documentation. Maintenance guides, standards and data sheets that have grown over the years.

RAG, fine-tuning or an agent?

Fine-tuning changes the model itself by training it further on your own examples. That helps with style, format or a very specific task. For knowledge that keeps changing it is impractical, because every new version of a document would need new training and the answer shows no source.

An agent goes one step further. It answers a question and then carries out steps on its own, such as creating a ticket or entering data into a system. Many agents use RAG as a tool to find the right knowledge first. For most companies RAG is therefore the first step, and AI agents build on it later.

What to watch for under GDPR

As soon as personal data appears in the documents, GDPR applies, and that is the case in almost every company. Settle three questions before you start.

Where is the data processed? Model, vector database and logs should run in the EU, with a provider that signs a data processing agreement.
Who sees what? The system has to take over the access rights from the source systems.
What gets stored? Questions and answers end up in logs. Decide how long they stay and who may read them.

On top of that comes the EU AI Act. Since August 2026 transparency obligations apply, so a chat has to be recognisable as AI. Our GDPR-compliant AI chat is a RAG system like this, hosted in Europe and connected to your company knowledge.

Common mistakes when building one

Messy documents. Outdated versions, duplicates and scanned PDFs without text recognition lead to bad answers. Cleaning up is part of the project.
No tests. Without a list of real questions and their correct answers nobody knows whether the system is getting better or worse. This evaluation belongs before launch.
Semantic search only. Vector search is good at meaning but often misses product codes, clause numbers and names. Adding keyword search usually fixes that.
Answers without a source. If users cannot check an answer, they stop trusting it after the first mistake.
Nobody in charge. Documents change. Someone has to make sure new versions get ingested and old ones removed.

How to get started

Start with a clearly bounded area, for example the customer service manual or your internal policies. Collect 30 to 50 real questions people ask today, together with the correct answers. With that you can build a first prototype and measure it honestly.

If you are not sure which area pays off most, we work that out with your team in an AI workshop.

Want to use AI with your own documents? We look at which of your documents are a good fit. Book a free discovery call →

Frequently Asked Questions

What is a RAG system?

A RAG system combines a search across your own documents with a language model. Before every answer it fetches the matching passages and hands them to the model. That way the AI answers from your own data and names the source.

What does RAG stand for?

RAG stands for Retrieval-Augmented Generation, meaning text generation that is supplemented by retrieving information. Before answering, the model receives relevant passages from a knowledge source.

What is the difference between RAG and fine-tuning?

Fine-tuning trains the model further on your own examples and suits style and specialised tasks. RAG leaves the model unchanged and supplies current knowledge from your own documents with every question. For company knowledge that keeps changing, RAG is usually the better choice.

Is a RAG system GDPR compliant?

It can be run in a GDPR-compliant way if model, vector database and logs run in the EU, a data processing agreement is in place and the system takes over the access rights of the source systems. A clear rule on how long questions and answers are stored is part of it.

Which documents work for RAG?

Anything that exists as text or can be turned into text, such as PDFs, Word files, wiki pages, tickets and emails. Scanned documents need text recognition first, and tables need their own preparation.

How do I get started with a RAG system?

With a clearly bounded area such as the customer service manual. Collect 30 to 50 real questions with their correct answers so the first prototype can be measured honestly. Only then roll the system out to further areas.