Retrieval-Augmented Generation (RAG): A Complete Guide for Businesses
How retrieval-augmented generation works and how to build it: ingestion, chunking, embeddings, retrieval, reranking, cited answers, evaluation, data location.
Quick answer
Retrieval-augmented generation (RAG) answers questions using your own information. At indexing time, documents are collected, parsed, split into chunks, embedded and indexed with metadata and permissions. At question time, the system retrieves the most relevant chunks (ideally with hybrid keyword and vector search plus reranking), gives them to a language model with instructions to answer only from them and cite sources, and refuses when evidence is missing. Evaluate retrieval and answers separately, because most failures start with retrieving the wrong content. For where RAG pays off, with the data and permissions each use needs, see 10 RAG use cases for business applications.
Where This Fits
This is the hub for ZSpace Labs' RAG guides. Deeper topics: chunking, embeddings, vector databases, hybrid search, reranking, enterprise RAG architecture, GraphRAG and RAG vs fine-tuning. For a product view, see AI knowledge base.
For building complete generative AI products around RAG, see generative AI application development.
How RAG Works
RAG has two phases. Indexing prepares your content: connectors pull documents, parsers extract text and structure, chunking splits it into retrievable passages, an embedding model converts passages into vectors, and an index stores vectors, text, metadata and permissions. Query time finds and uses that content: the question may be rewritten, retrieval finds candidate passages, a reranker orders them, and the model generates an answer from the top passages with citations.
Ingestion and Parsing
Garbage in, garbage retrieved. Parse documents in a way that keeps structure: headings, lists, tables and page numbers. PDFs with columns, scanned pages and complex tables need specialised parsing or OCR. Store metadata (source, title, section, date, owner, access groups) with every chunk; it powers filtering, citations and freshness. Re-index on change rather than on a slow schedule where content changes often.
Chunking and Embeddings
Chunks should be small enough to be specific and large enough to make sense on their own. Structure-aware chunking (by section) usually beats fixed-size splitting for business documents; see RAG chunking strategies. Embeddings turn chunks into vectors for semantic search; the model you choose affects quality, cost and language support; see vector embeddings explained.
Retrieval: Hybrid Search, Filters and Reranking
Vector search finds passages with similar meaning; keyword search (BM25) finds exact terms such as product codes, names and error messages. Combining them in hybrid search usually beats either alone. Apply metadata filters (permissions, product, date) during retrieval. Then rerank a larger candidate set with a cross-encoder or similar model so the best passages reach the prompt.
| Technique | What it fixes |
|---|---|
| Query rewriting | Vague or conversational questions |
| Hybrid search | Missed exact terms and codes |
| Metadata filters | Wrong product, region, date or permission |
| Reranking | Relevant passages ranked too low |
| Parent-document retrieval | Chunks too small to answer alone |
Generation: Grounded Answers With Citations
Instruct the model to answer only from the provided sources, cite them, say when the sources do not contain the answer and avoid speculation. Keep the context focused: more passages are not always better, and irrelevant text can confuse the model and raises cost. Validate citations where accuracy matters (does the cited passage support the claim?). For structured outputs from documents, see AI document extraction.
Building an AI assistant on your company's documents?
ZSpace Labs builds RAG systems with permission-aware retrieval, citations and evaluation, connected to the sources your teams already use.
Evaluating RAG
Build a question set from real queries with expected answers and the sources that contain them. Measure retrieval (are the right sources in the top results?) separately from generation (is the answer faithful to the sources, correct and complete?). Use deterministic checks where possible and calibrated LLM judges for faithfulness. Re-run on every change to parsing, chunking, embeddings, retrieval settings or model. See AI evaluation.
- Retrieval recall at k: is a correct source in the top k?
- Ranking quality: how high does the correct source appear?
- Faithfulness: are all claims supported by retrieved text?
- Answer correctness and completeness
- Correct refusals when the answer is not in the sources
- Latency and cost per question
Security and Permissions
A RAG system must not show people documents they cannot open in the source system. Copy access controls with content and filter at retrieval time by the user's identity and groups. Treat retrieved text as untrusted: a document could contain instructions aimed at the model, so retrieval-only assistants should not have powerful tools. The OWASP Top 10 for LLM Applications lists vector and embedding weaknesses among its risks. See enterprise RAG architecture.
Costs
Indexing costs come from parsing and embedding (mostly up front, plus updates). Query costs come from retrieval infrastructure, reranking and model tokens, which scale with the number of passages included. Keep context lean, cache frequent answers where safe and choose the smallest model that meets quality targets; see LLM cost optimization.
Advantages and Limitations
| Advantages | Limitations |
|---|---|
| Answers from current, private information | Quality depends on content quality and coverage |
| Citations make answers checkable | Retrieval can miss or misrank the right passage |
| Update knowledge by re-indexing, not retraining | Complex questions across many documents are hard |
| Permissions can mirror source systems | Needs ongoing evaluation and content ownership |
How to Build a RAG System Step by Step
- 1. Define the questions users need answered and collect real examples
- 2. Inventory sources and their owners, formats and permissions
- 3. Build ingestion with structure-preserving parsing and metadata
- 4. Choose chunking and embeddings and test on your questions
- 5. Implement hybrid retrieval with filters and reranking
- 6. Write generation instructions for grounded, cited answers and refusals
- 7. Evaluate retrieval and answers separately and fix the weakest stage
- 8. Launch with feedback and a process for content owners to fix gaps
RAG Architecture Patterns
| Pattern | How it works | When to use |
|---|---|---|
| Basic RAG | Retrieve top chunks, generate once | Prototypes, simple FAQs |
| Advanced RAG | Query rewriting, hybrid search, reranking, citations | Most production systems |
| Parent-document RAG | Match small chunks, pass larger sections | Long structured documents |
| Agentic RAG | An agent decides when and what to retrieve, possibly several times | Complex, multi-part questions |
| GraphRAG | Knowledge graph and community summaries | Relationship and corpus-wide questions |
Tools and Technology Choices
A RAG stack typically includes connectors and parsers, an embedding model, an index (Postgres with pgvector, a dedicated vector database or a search engine with vector support), a reranker, a language model and an evaluation harness. Frameworks such as LlamaIndex and LangChain speed up assembly; cloud platforms and enterprise search products offer managed options. Choose components based on your sources, scale, permission model and data residency, and keep them swappable behind your own interfaces. Storage choices are compared in vector databases for AI.
RAG Use Cases by Function
| Function | Questions RAG answers | Typical sources |
|---|---|---|
| HR and people | Leave, benefits, policies, onboarding | Handbooks, policy sites |
| Customer support | Product how-to, troubleshooting, policies | Help centre, runbooks, resolved tickets |
| Sales | Product capabilities, pricing rules, security answers | Product docs, approved security questionnaires |
| Legal and compliance | Clause positions, policy interpretation | Playbooks, contract libraries |
| Engineering and IT | Architecture decisions, runbooks, incidents | Wikis, repositories, postmortems |
| Operations | Procedures, specifications, standards | SOPs, manuals, specifications |
Operating RAG in Production
Launching is the start. Production RAG needs ingestion monitoring (failed syncs, parsing errors, document counts), freshness tracking, evaluation runs after every pipeline change, sampled answer reviews, user feedback triage and cost and latency dashboards. Assign owners: an engineering owner for the pipeline and content owners for each source area. Schedule re-evaluation when you change embedding models, chunking, retrieval settings or the generation model, and keep the previous index available until the new one is proven.
- Ingestion health and freshness alerts
- Evaluation in CI for pipeline and prompt changes
- Weekly review of low-rated answers and unanswered questions
- Content owner reports per source area
- Cost per question and latency percentiles
- Versioned indexes with rollback
Worked Example
An illustrative scenario, not a client case: an engineering firm's RAG assistant gives vague answers about project standards. Evaluation shows the right document is retrieved but split mid-table, and part numbers are missed by vector search. Switching to section-aware chunking that keeps tables intact and adding keyword search with rank fusion fixes most failures; a reranker improves the rest.
Common Mistakes
- Tuning prompts when retrieval is the problem
- Vector-only search for content full of codes and names
- Losing tables and headings during parsing
- No permission filtering
- No evaluation set
- Stale indexes nobody owns
Want a RAG system your team can trust?
Talk to ZSpace Labs about RAG development and data integration and deployment.
RAG for Australian Organizations
The architecture is the same everywhere; Australian organizations should add a few checks.
- Data location: embeddings, vector stores, model calls and logs may each run in different regions; check all of them. See private AI development in Australia for model availability in Australian regions.
- Privacy Act: if covered, privacy obligations apply to personal information in source documents, prompts and outputs; APP 8 applies to overseas processing.
- Permissions: retrieval must respect source-document permissions so users cannot see through the AI what they cannot open directly.
- Customer-facing answers: answers are your representations under the Australian Consumer Law; ground them in reviewed content and hand off when unsure; see AI customer service in Australia.
Conclusion
RAG is a retrieval problem first and a generation problem second. Invest in parsing, chunking, hybrid retrieval, reranking, permissions and evaluation, and keep content owners involved. Next: enterprise RAG, hybrid search and AI knowledge base.
Retrieval is one of several layers that reduce wrong answers; for the others (tool verification, structured outputs, business rules and post-action checks), see how to reduce AI agent hallucinations.
Common questions.
A technique where an AI system first retrieves relevant information from your own sources, such as documents or databases, and then gives it to a language model to generate an answer grounded in that information, usually with citations.
Models do not know your private or recent information and can produce confident but wrong answers. RAG supplies the right source material at answer time, which improves accuracy, freshness and traceability.
Data connectors and ingestion, parsing and chunking, embeddings and an index (vector, keyword or both), a retriever with filters, optional reranking, a generation step with instructions and citations, and evaluation and monitoring.
Not necessarily. You need a searchable index; that can be a vector database, Postgres with pgvector, or a search engine with vector and keyword support. Many systems use hybrid search.
Usually retrieval: documents missing or stale, poor parsing, chunks that split key information, missing keyword matching for codes and names, no reranking, or permissions filtering out the right source. Generation issues are less common than teams expect.
Separately for retrieval (did the right sources appear in the top results?) and generation (is the answer faithful to sources, complete and correct?), using a question set with known answers and sources.
Costs include embedding documents (mostly one-time plus updates), index storage, retrieval infrastructure and model tokens per question, which grow with how much context you include. Reranking and larger models add cost per query.
It reduces them but does not eliminate them. Instructions to answer only from sources, citation checks, refusal when evidence is missing and evaluation of faithfulness are still needed.
It can be, if retrieval enforces the same permissions as the source systems, data is processed by approved providers and retrieved content is treated as untrusted input that cannot trigger actions.
When the problem is the model's behaviour, format or specialised skill rather than missing knowledge. For knowledge that changes, RAG is usually the better fit.
Often, if each component is placed carefully: the vector store and application in an Australian cloud region, and a model available for processing in that region. Some model options route requests to other regions, so check provider documentation. Logs, evaluation tools and embeddings services are frequently overlooked.