What is RAG? When AI searches before answering
Simple and practical explanation of "Retrieval-Augmented Generation" (RAG); a method that connects chatbots to your organization's documents, site, or database to provide accurate and documented answers.
Problem: The Model Doesn't Know Everything
An AI language model only "knows" what it was trained on during its training phase. Therefore:
- It does not know about events that occurred after its training cutoff date.
- It does not know your company's internal bylaws, product prices, or customer records.
- When it doesn't know an answer, it may invent a made-up yet convincing response (Hallucination).
Solution: Search First, Then Answer
RAG (Retrieval-Augmented Generation) has a simple idea: before the question reaches the model, the system finds relevant documents and passes them along with the question to the model. The model then writes the response based on these exact documents.
An analogy: An "open-book" exam. The student doesn't need to memorize the entire book; it's enough to know which page to open and write the answer from it.
How Does RAG Work Step-by-Step?
Preparation (One-time)
- Document Collection: PDFs, website pages, support articles, tables.
- Chunking: Each document is divided into small pieces (e.g., a few paragraphs).
- Creating Semantic Vectors: Each piece is converted into an Embedding; a number representing its meaning.
- Storage: The vectors are stored in a Vector Database.
During Answering (Every Question)
- The user's question is also converted into a vector.
- The semantically closest chunks are found (not just based on matching words).
- These chunks are given to the model along with the question and an instruction like "Answer only based on the following texts and cite the source."
- The model generates a grounded response.
Real-World Example
Suppose your online store has 200 pages of FAQs and return policies. A customer asks: "Can I return shoes that are the wrong size after ten days?"
- Without RAG: The model gives a general and likely incorrect answer about how "stores usually..."
- With RAG: The system finds the clause regarding the "apparel return deadline" and the model answers strictly based on your store's policy.
Pros and Limitations
| Pros | Limitations |
|---|---|
| Up-to-date answers without retraining the model | Response quality depends on search quality |
| Ability to cite a source for every answer | Bad or outdated documents = bad answers |
| Cheaper and faster than Fine-Tuning | Incorrect chunking might break the context |
| Precise control over the information the model sees | Requires maintenance and updating of the document index |
RAG or Fine-Tuning?
- If you want the model to know new information (prices, rules, documents) → RAG.
- If you want the model to have a specific behavior or style (brand tone, fixed output format) → Fine-Tuning.
- In many real-world projects, both are used together.
Where Do We See RAG?
- Perplexity and ChatGPT and Gemini web search, which provide answers along with sources.
- Google's NotebookLM, which answers solely based on your sources.
- The "Projects" feature in Claude and ChatGPT, which takes your files into account during the conversation.
- Organizational support chatbots.
Get Started
To test RAG without coding, upload a few PDF documents into NotebookLM and ask questions about them. If privacy is important, build a fully local RAG system with Ollama and Open WebUI.