RAG (retrieval-augmented generation)
Letting an AI answer from your own documents by finding relevant passages first.
What is it?
Retrieval-augmented generation (RAG) combines search with text generation. When a question arrives, the system first searches a collection of documents, such as manuals, policies or notes, for the most relevant passages. Those passages are added to the prompt, and the model writes its answer based on them.
RAG lets a model use information it was never trained on, including private or very recent material, without retraining. It also makes it possible to show sources, so people can check where an answer came from.
Which models use it?
- Many company chatbots and help centers answer questions from their own documentation using RAG.
- ChatGPT, Claude and Gemini can answer from files you upload, which uses similar retrieval ideas.
- AI search tools such as Perplexity retrieve web pages and cite them in answers.
- Developer frameworks such as LlamaIndex and LangChain provide RAG building blocks.
How to use it
- Collect the documents the assistant should answer from and remove outdated versions.
- Split them into small passages (chunks) and index them for search.
- For each question, retrieve the most relevant passages.
- Put those passages in the prompt and instruct the model to answer only from them and cite them.
- Tell the model to say "I don't know" when the passages do not contain the answer.
- Test with real questions and check both the retrieved passages and the answers.
Benefits
- Answers can use private and up-to-date information.
- Citations make answers easier to verify.
- Fewer made-up facts compared with relying on the model's memory.
- Updating documents updates answers without retraining a model.
Limits and risks
- If retrieval finds the wrong passages, the answer will be wrong too.
- The model can still misread or over-generalize the sources.
- Documents must be kept current and access-controlled.
- Retrieved content may contain prompt injection if the sources are not trusted.
Prompt template
Copy it, replace the parts in [brackets] and paste it into any AI chat.
Answer the question below using ONLY the sources provided. Quote or cite the source number for each claim, like [1]. If the sources do not contain the answer, say "The sources do not say" and suggest what document might help. Sources: [1] [paste passage] [2] [paste passage] Question: [your question]
More in Building with AI
- Using AI through APIsHow apps and scripts send requests to AI models in code instead of a chat window.
- AI agents and tool useAI systems that can take actions, such as searching, running code or using apps, to finish a task.
- MCP (Model Context Protocol) and connectorsAn open standard for connecting AI assistants to tools and data sources in a consistent way.
- Embeddings and semantic searchTurning text into numbers that capture meaning, so you can search by idea rather than exact words.
- Fine-tuning vs promptingWhen to customize a model with extra training, and when better prompts are enough.
- Evaluating AI outputPractical ways to check whether AI answers are accurate, useful and safe before you rely on them.