A general-purpose AI model can write a polished paragraph about refund policies. It cannot tell a customer what your refund policy says, because it has never read it. Ask anyway and it may produce a confident answer that is simply wrong. Retrieval-augmented generation, usually shortened to RAG, is the most common way to close that gap.

This guide explains RAG for business readers: what it is, how it works step by step, how it compares with fine-tuning and with simply pasting documents into a large prompt, where it goes wrong, and what a sensible first project looks like.

Quick answer

Retrieval-augmented generation (RAG) is a pattern in which software first searches your own documents for passages relevant to a question, then hands those passages to a language model and asks it to answer using only that material, ideally with citations. The model's training is not changed. Your content stays in a search index that you control and can update at any time.

• Use RAG when answers must come from a large or frequently changing body of your own content, and people need to see where an answer came from.
• Skip the retrieval layer when the material is small enough to place in the prompt directly.
• Do not expect fine-tuning to be a substitute. It changes how a model behaves, and is a poor way to keep facts current.
• Most RAG failures are content and permission problems, not model problems: outdated documents, contradictory versions, and an index that ignores who is allowed to see what.

Why RAG exists

A language model has two limits that matter to a business. Its knowledge stops at the point its training data was collected, and it has never seen your private material: contracts, policies, product manuals, support tickets, pricing sheets.

The term comes from a 2020 research paper, Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, by Patrick Lewis and colleagues. The authors pointed to two open problems with language models: providing provenance for what they say, and updating what they know. Their approach was to pair the model's built-in knowledge with an external, searchable store of documents. Both problems are exactly what a business cares about. You want to know which document an answer came from, and you want to correct an answer by editing a document rather than retraining a model.

A useful way to picture it: the model is a capable new employee who has read widely but knows nothing about your company. RAG is the practice of handing that employee the right three pages from the filing cabinet before they reply.

How RAG works, step by step

A RAG system has two halves. The first four steps prepare your content and happen ahead of time. The last three happen each time someone asks a question.

1. Ingestion

The system collects content from where it lives: shared drives, a help center, a wiki, a document management system, a database. Text is extracted from each file. This is less trivial than it sounds. Scanned PDFs need optical character recognition, tables lose their structure easily, and slide decks often make little sense without the speaker.

2. Chunking

Long documents are split into smaller passages, called chunks, so that each can be matched to a question independently. Chunk size is a real trade-off. Too large and the passage contains a lot of irrelevant text. Too small and it loses meaning. Anthropic gives a clear example in its write-up on contextual retrieval: a chunk stating that revenue grew 3 percent is of little use if the chunk no longer says which company or which quarter it refers to.

3. Embeddings

Each chunk is passed through an embedding model, which converts the text into a long list of numbers (a vector) representing its meaning. Passages about similar topics end up with similar numbers, even when they share no words. That is how a question about "time off" can find a policy titled "annual leave".

4. Indexing

The vectors are stored in a vector database or a search service, along with the original text and metadata such as source, date, owner and access permissions. That metadata matters more than most teams expect, as the failure section below shows.

5. Retrieval

When a user asks a question, the question is also turned into a vector and the system finds the chunks closest in meaning. Good systems rarely rely on vector search alone. Microsoft's Azure AI Search guidance on RAG recommends hybrid queries that combine keyword and vector search, followed by a ranking step that re-scores the results. Keyword search catches exact terms such as part numbers, error codes and names, which meaning-based search tends to blur.

6. Prompt assembly

The application builds a prompt containing instructions ("answer only from the passages below, and say so if the answer is not there"), the retrieved passages with their sources, and the user's question.

7. Answer with citations

The model writes an answer from the supplied passages and points to the ones it used. Citations are what turn an AI answer from something you have to trust into something you can check. Some model providers now support this directly: Anthropic's citations feature, for example, returns the exact passages that support each claim. Note that a citation proves the answer was drawn from a document. It does not prove the document was correct.

RAG vs fine-tuning vs long context windows

These three approaches are often presented as rivals. They solve different problems.

Fine-tuning means further training a model on examples of the inputs and outputs you want. OpenAI's model optimization guide describes it as a way to make a model excel at a particular task, including training a smaller, cheaper model for a job where a larger one is not cost-effective. It is suited to behavior: tone, format, classification, a specialized task. It is not suited to facts that change, because every change means another training run, and a fine-tuned model cannot show you its source.

Long context windows mean putting the documents straight into the prompt. Context windows have grown considerably: Anthropic's documentation lists a context window of up to 1 million tokens on its current models. For a small body of content this is often the right answer, and it avoids building a retrieval layer at all. In its 2024 contextual retrieval article, linked above, Anthropic suggested that a knowledge base under roughly 200,000 tokens, about 500 pages, could simply be included in the prompt. The same documentation is candid about the limit, though: more context is not automatically better, and accuracy and recall degrade as the amount of text grows. You also pay to process all of that text on every question.

The approaches combine. A common mature setup uses RAG for facts, a carefully written prompt for behavior, and fine-tuning only if a specific, measured quality or cost problem remains.

Where RAG fails

RAG reduces invented answers. It does not eliminate wrong ones. The usual causes are predictable.

Stale or contradictory content

If the 2023 and 2026 versions of a price list are both in the index, the system may retrieve either one, and the model will answer fluently from whichever it received. RAG does not know which document is authoritative unless you tell it, through metadata, archiving rules or by removing superseded files. Many organizations discover during a RAG project that nobody owns their documentation.

Access control

This is the failure with the most serious consequences. If every document is indexed into one pool, any employee can ask about salaries, board papers or another client's contract and receive a tidy summary. The OWASP Top 10 for LLM Applications lists this under vector and embedding weaknesses, including leaks between user groups that share one vector store, and recommends permission-aware vector stores with fine-grained access controls. In practice, permissions must be carried into the index and applied at retrieval time, so the model is never shown a passage the person asking could not open themselves. Asking the model to keep secrets is not a control.

Poor retrieval

If the right passage is not retrieved, the model cannot use it. Typical causes: the question uses different vocabulary from the documents, the answer is spread across several documents, the chunk lost its context, or the question needs counting or comparison across the whole collection ("how many contracts renew next quarter?"), which is a database query, not a search problem. A well-instructed system says "I could not find that". A poorly instructed one fills the gap from general knowledge.

Instructions hidden in documents

Retrieved text is fed to the model, so text written by an outsider, such as an inbound email or a web page, can contain instructions aimed at the model. OWASP calls this indirect prompt injection and notes that RAG does not fully mitigate it. The risk is modest for a read-only question-answering tool over internal documents and grows sharply once the system can take actions. Our comparison of AI agents, chatbots and workflow automation covers guardrails and human approval for that case.

Data protection considerations

This section is general information, not legal advice. A RAG system creates a new copy of your content in an index and sends extracts of it to a model provider, so three questions come up in almost every project.

Where does the content go, and what does the provider do with it? Read the provider's current terms for the specific product you are buying. As examples, OpenAI states that data sent to its API is not used to train its models unless you opt in, and that abuse monitoring logs are retained for up to 30 days by default. Anthropic states that it does not use inputs or outputs from its commercial products to train models by default. Consumer chat products often have different terms from business APIs, so check which one your team is actually using.

Does the index contain personal data? Support tickets, HR files and customer emails almost always do. In the UK, the Information Commissioner's Office publishes guidance on AI and data protection covering accountability, lawfulness, data minimization, security and individual rights, and explains when a data protection impact assessment is required. Both pages currently carry a notice that they are under review following the Data (Use and Access) Act, so check for updates. In the US there is no single federal privacy law covering this. State laws apply instead, such as California's CCPA, which gives residents rights to know, delete and correct personal information held by businesses it covers, alongside sector rules for areas such as health and finance. NIST's AI Risk Management Framework and its Generative AI Profile are voluntary but give a useful structure for risk reviews.

Can you delete it? When a document or a person's data must be removed, it has to go from the source, the index, any caches and any logs of questions and answers. Design for that from the start. Index only what the use case needs.

What a sensible first RAG project looks like

The projects that succeed tend to be narrow and unglamorous.

1. Pick one audience and one body of content. For example: support staff answering from product manuals and published policies. Avoid "everything on the shared drive".
2. Choose content that has an owner. Someone must be responsible for keeping it current and deciding which version is authoritative.
3. Start internal and read-only. Staff can spot a wrong answer and check the citation. A customer-facing assistant raises the stakes and should come later.
4. Write the test questions first. Collect a set of real questions with known correct answers and the document each answer lives in. This is the only objective way to tell whether a change made the system better or worse.
5. Settle permissions before indexing. Either limit the pilot to content everyone in the audience may see, or build permission filtering in from day one.
6. Require citations and allow "I don't know". Both build trust faster than a system that always has an answer.
7. Plan for upkeep. Content must be re-indexed when it changes, and someone needs to review unanswered questions, which usually reveal gaps in your documentation.

Illustrative scenario, not a client case study: a property management firm indexes its tenant handbook, maintenance procedures and standard lease clauses so that front-office staff can answer tenant queries with a cited source. Individual tenancy files, which contain personal data, stay out of the first phase.

The effort and cost of a project like this are driven by the number and messiness of content sources, document formats (scans and tables are harder), the permission model, how often content changes, the accuracy the use case demands, and whether the assistant must plug into existing software. Model usage fees are usually the smaller part.

For technical readers

• Retrieval: start with hybrid search (BM25 plus dense vectors) and a reranker. Anthropic reported in 2024 that adding chunk-level context to embeddings and BM25, then reranking, reduced retrieval failures by 67 percent in its own tests; treat that as a result on its datasets and measure on yours.
• Chunking: split on document structure (headings, sections, table rows) before falling back to fixed sizes, and keep title and section path with each chunk.
• Authorization: store access control metadata with each chunk and filter at query time using the caller's identity. Never filter after generation.
• Evaluation: measure retrieval (was the right chunk in the top results?) separately from generation (was the answer faithful to the chunks?). Run the test set on every change.
• Freshness: use incremental indexing with deletion propagation, and record source version and last-modified date so answers can show them.
• Observability: log the query, retrieved chunk IDs and the answer, with a retention policy, since those logs may hold personal data.
• Agentic retrieval: having a model plan several sub-queries can help with complex questions. Microsoft currently labels the query planning part of its implementation as a preview feature, so check maturity before depending on it.

Frequently asked questions

What does RAG stand for?

RAG stands for retrieval-augmented generation. The system retrieves relevant passages from your documents and adds them to the prompt, so the language model generates its answer from that material instead of from memory alone.

Does RAG stop AI hallucinations?

No. It reduces them by giving the model the relevant source text and makes answers checkable through citations. Wrong answers still occur when the wrong passage is retrieved, when documents are outdated or contradict each other, or when the model misreads a passage.

Is RAG the same as training the AI on our data?

No. With RAG the model itself is unchanged. Your documents sit in a separate index and relevant extracts are shown to the model only when a question is asked. Removing a document from the index removes it from future answers, which is not true of a model that was trained on it.

Do we still need RAG now that models accept very large prompts?

Sometimes not. If your material is small and stable, placing it in the prompt is simpler. RAG earns its place when the content is large, changes often, needs per-user access control, or when processing everything on every question would be too slow or costly.

Is RAG safe for confidential documents?

It can be, if it is designed that way. The key requirements are permission filtering at retrieval time, a model provider whose terms on training, retention and data location suit your obligations, and a way to delete content from the index and logs.

What kinds of documents work best?

Well-structured, current text with clear headings: policies, manuals, help articles, procedures. Scanned documents, complex tables, spreadsheets and slide decks need extra processing and usually produce weaker results at first.

Choosing your next step

RAG is a way of making a language model answer from your documents, with sources, without retraining it. Whether it works depends less on the model than on the content you feed it, the permissions you enforce and the test questions you measure against.

Before talking to any vendor, write down three things: the one group of people the assistant is for, the specific documents it should answer from, and twenty real questions it must get right. That short list will tell you whether you need RAG at all or whether a large prompt will do. If you would like a second opinion on that list or a scoped estimate for a pilot, Entrant Technologies builds custom software and web applications, and you can request a quote or review our development services.