Retrieval-augmented generation (RAG) is a technique where an AI system first searches a collection of trusted documents for passages relevant to a question, then gives those passages to a language model to write the answer. It lets AI answer from your own, up-to-date information and reduces hallucinations, as long as the right documents are retrieved.
The problem RAG solves
A language model only knows what was in its training data. It does not know your school's policies, your company's product manual, last week's price list, or the notes in your folder. If you ask about them, it will either say it does not know or, worse, make something up.
You could paste documents into every chat, but that does not scale to thousands of pages. RAG solves this by retrieving only the relevant pieces automatically and giving them to the model at the moment it answers.
How RAG works, step by step
Preparation (done once, then updated)
- Collect documents: PDFs, web pages, policies, notes, FAQs.
- Split into chunks: break them into passages of a few paragraphs.
- Create embeddings: convert each chunk into an embedding, a list of numbers that represents its meaning. Passages about similar topics end up with similar numbers.
- Store in a vector database: a database designed to find embeddings that are close to each other.
Answering a question
- The user asks: "Can I reschedule a session less than 24 hours before?"
- The question is converted into an embedding.
- The system searches the vector database for the most similar chunks, such as the rescheduling section of the policy.
- Those chunks are inserted into the prompt with instructions: "Answer using only the information below and cite the section."
- The language model writes an answer grounded in the retrieved text.
Where RAG is used
- Customer support assistants that answer from help-centre articles.
- Internal company assistants that search policies and documentation.
- Study assistants that answer from a student's notes or a course syllabus.
- Legal, medical and research tools that must point to sources.
What can go wrong
RAG is powerful but not magic:
- Bad retrieval: if the search finds the wrong passages, the answer will be wrong, confidently.
- Poor chunking: splitting documents in awkward places loses context.
- Outdated documents: the answer is only as current as the stored data.
- Ignoring the context: models sometimes still add information not in the sources.
Good RAG systems therefore test retrieval quality, re-rank results, show citations, and are evaluated on real questions before launch.
RAG versus fine-tuning
| RAG | Fine-tuning | |
|---|---|---|
| What it does | Gives the model relevant documents at answer time | Changes the model's behaviour by further training |
| Best for | Facts that change, private knowledge, citations | Style, format, specialised behaviour |
| Updating | Add or edit documents | Retrain the model |
| Cost and effort | Usually lower | Usually higher |
Many real systems use RAG for knowledge and prompting or light fine-tuning for style.
Learn to build RAG
RAG is one of the most in-demand skills for AI developers because almost every organisation wants an assistant that answers from its own information. In our generative AI development course, students build a document Q&A assistant with citations as one of their main projects. Beginners can start by building a simple chatbot.
Frequently asked questions
Do I need to code to build a RAG system?
Some no-code tools offer document-based assistants, which use RAG behind the scenes. Building a reliable custom system usually requires Python or JavaScript.
Does RAG eliminate hallucinations?
It reduces them significantly when retrieval is good, but does not eliminate them. Citations and testing remain important.
What is a vector database?
A database that stores embeddings and quickly finds the ones most similar in meaning to a query.
