Skip to content
AI & Automation10 min read

Retrieval-Augmented Generation (RAG): A Complete Guide for Businesses

How retrieval-augmented generation works and how to build it: ingestion, chunking, embeddings, retrieval, reranking, cited answers, evaluation, data location.

01

Quick answer

Retrieval-augmented generation (RAG) answers questions using your own information. At indexing time, documents are collected, parsed, split into chunks, embedded and indexed with metadata and permissions. At question time, the system retrieves the most relevant chunks (ideally with hybrid keyword and vector search plus reranking), gives them to a language model with instructions to answer only from them and cite sources, and refuses when evidence is missing. Evaluate retrieval and answers separately, because most failures start with retrieving the wrong content. For where RAG pays off, with the data and permissions each use needs, see 10 RAG use cases for business applications.

02

Where This Fits

This is the hub for ZSpace Labs' RAG guides. Deeper topics: chunking, embeddings, vector databases, hybrid search, reranking, enterprise RAG architecture, GraphRAG and RAG vs fine-tuning. For a product view, see AI knowledge base.

For building complete generative AI products around RAG, see generative AI application development.

03

How RAG Works

RAG has two phases. Indexing prepares your content: connectors pull documents, parsers extract text and structure, chunking splits it into retrievable passages, an embedding model converts passages into vectors, and an index stores vectors, text, metadata and permissions. Query time finds and uses that content: the question may be rewritten, retrieval finds candidate passages, a reranker orders them, and the model generates an answer from the top passages with citations.

04

Ingestion and Parsing

Garbage in, garbage retrieved. Parse documents in a way that keeps structure: headings, lists, tables and page numbers. PDFs with columns, scanned pages and complex tables need specialised parsing or OCR. Store metadata (source, title, section, date, owner, access groups) with every chunk; it powers filtering, citations and freshness. Re-index on change rather than on a slow schedule where content changes often.

05

Chunking and Embeddings

Chunks should be small enough to be specific and large enough to make sense on their own. Structure-aware chunking (by section) usually beats fixed-size splitting for business documents; see RAG chunking strategies. Embeddings turn chunks into vectors for semantic search; the model you choose affects quality, cost and language support; see vector embeddings explained.

06

Retrieval: Hybrid Search, Filters and Reranking

Vector search finds passages with similar meaning; keyword search (BM25) finds exact terms such as product codes, names and error messages. Combining them in hybrid search usually beats either alone. Apply metadata filters (permissions, product, date) during retrieval. Then rerank a larger candidate set with a cross-encoder or similar model so the best passages reach the prompt.

TechniqueWhat it fixes
Query rewritingVague or conversational questions
Hybrid searchMissed exact terms and codes
Metadata filtersWrong product, region, date or permission
RerankingRelevant passages ranked too low
Parent-document retrievalChunks too small to answer alone
07

Generation: Grounded Answers With Citations

Instruct the model to answer only from the provided sources, cite them, say when the sources do not contain the answer and avoid speculation. Keep the context focused: more passages are not always better, and irrelevant text can confuse the model and raises cost. Validate citations where accuracy matters (does the cited passage support the claim?). For structured outputs from documents, see AI document extraction.

Building an AI assistant on your company's documents?

ZSpace Labs builds RAG systems with permission-aware retrieval, citations and evaluation, connected to the sources your teams already use.

Start a Project
08

Evaluating RAG

Build a question set from real queries with expected answers and the sources that contain them. Measure retrieval (are the right sources in the top results?) separately from generation (is the answer faithful to the sources, correct and complete?). Use deterministic checks where possible and calibrated LLM judges for faithfulness. Re-run on every change to parsing, chunking, embeddings, retrieval settings or model. See AI evaluation.

  • Retrieval recall at k: is a correct source in the top k?
  • Ranking quality: how high does the correct source appear?
  • Faithfulness: are all claims supported by retrieved text?
  • Answer correctness and completeness
  • Correct refusals when the answer is not in the sources
  • Latency and cost per question
09

Security and Permissions

A RAG system must not show people documents they cannot open in the source system. Copy access controls with content and filter at retrieval time by the user's identity and groups. Treat retrieved text as untrusted: a document could contain instructions aimed at the model, so retrieval-only assistants should not have powerful tools. The OWASP Top 10 for LLM Applications lists vector and embedding weaknesses among its risks. See enterprise RAG architecture.

10

Costs

Indexing costs come from parsing and embedding (mostly up front, plus updates). Query costs come from retrieval infrastructure, reranking and model tokens, which scale with the number of passages included. Keep context lean, cache frequent answers where safe and choose the smallest model that meets quality targets; see LLM cost optimization.

11

Advantages and Limitations

AdvantagesLimitations
Answers from current, private informationQuality depends on content quality and coverage
Citations make answers checkableRetrieval can miss or misrank the right passage
Update knowledge by re-indexing, not retrainingComplex questions across many documents are hard
Permissions can mirror source systemsNeeds ongoing evaluation and content ownership
12

How to Build a RAG System Step by Step

  • 1. Define the questions users need answered and collect real examples
  • 2. Inventory sources and their owners, formats and permissions
  • 3. Build ingestion with structure-preserving parsing and metadata
  • 4. Choose chunking and embeddings and test on your questions
  • 5. Implement hybrid retrieval with filters and reranking
  • 6. Write generation instructions for grounded, cited answers and refusals
  • 7. Evaluate retrieval and answers separately and fix the weakest stage
  • 8. Launch with feedback and a process for content owners to fix gaps
13

RAG Architecture Patterns

PatternHow it worksWhen to use
Basic RAGRetrieve top chunks, generate oncePrototypes, simple FAQs
Advanced RAGQuery rewriting, hybrid search, reranking, citationsMost production systems
Parent-document RAGMatch small chunks, pass larger sectionsLong structured documents
Agentic RAGAn agent decides when and what to retrieve, possibly several timesComplex, multi-part questions
GraphRAGKnowledge graph and community summariesRelationship and corpus-wide questions
14

Tools and Technology Choices

A RAG stack typically includes connectors and parsers, an embedding model, an index (Postgres with pgvector, a dedicated vector database or a search engine with vector support), a reranker, a language model and an evaluation harness. Frameworks such as LlamaIndex and LangChain speed up assembly; cloud platforms and enterprise search products offer managed options. Choose components based on your sources, scale, permission model and data residency, and keep them swappable behind your own interfaces. Storage choices are compared in vector databases for AI.

15

RAG Use Cases by Function

FunctionQuestions RAG answersTypical sources
HR and peopleLeave, benefits, policies, onboardingHandbooks, policy sites
Customer supportProduct how-to, troubleshooting, policiesHelp centre, runbooks, resolved tickets
SalesProduct capabilities, pricing rules, security answersProduct docs, approved security questionnaires
Legal and complianceClause positions, policy interpretationPlaybooks, contract libraries
Engineering and ITArchitecture decisions, runbooks, incidentsWikis, repositories, postmortems
OperationsProcedures, specifications, standardsSOPs, manuals, specifications
16

Operating RAG in Production

Launching is the start. Production RAG needs ingestion monitoring (failed syncs, parsing errors, document counts), freshness tracking, evaluation runs after every pipeline change, sampled answer reviews, user feedback triage and cost and latency dashboards. Assign owners: an engineering owner for the pipeline and content owners for each source area. Schedule re-evaluation when you change embedding models, chunking, retrieval settings or the generation model, and keep the previous index available until the new one is proven.

  • Ingestion health and freshness alerts
  • Evaluation in CI for pipeline and prompt changes
  • Weekly review of low-rated answers and unanswered questions
  • Content owner reports per source area
  • Cost per question and latency percentiles
  • Versioned indexes with rollback
17

Worked Example

An illustrative scenario, not a client case: an engineering firm's RAG assistant gives vague answers about project standards. Evaluation shows the right document is retrieved but split mid-table, and part numbers are missed by vector search. Switching to section-aware chunking that keeps tables intact and adding keyword search with rank fusion fixes most failures; a reranker improves the rest.

18

Common Mistakes

  • Tuning prompts when retrieval is the problem
  • Vector-only search for content full of codes and names
  • Losing tables and headings during parsing
  • No permission filtering
  • No evaluation set
  • Stale indexes nobody owns

Want a RAG system your team can trust?

Talk to ZSpace Labs about RAG development and data integration and deployment.

Start a Project
19

RAG for Australian Organizations

The architecture is the same everywhere; Australian organizations should add a few checks.

  • Data location: embeddings, vector stores, model calls and logs may each run in different regions; check all of them. See private AI development in Australia for model availability in Australian regions.
  • Privacy Act: if covered, privacy obligations apply to personal information in source documents, prompts and outputs; APP 8 applies to overseas processing.
  • Permissions: retrieval must respect source-document permissions so users cannot see through the AI what they cannot open directly.
  • Customer-facing answers: answers are your representations under the Australian Consumer Law; ground them in reviewed content and hand off when unsure; see AI customer service in Australia.
20

Conclusion

RAG is a retrieval problem first and a generation problem second. Invest in parsing, chunking, hybrid retrieval, reranking, permissions and evaluation, and keep content owners involved. Next: enterprise RAG, hybrid search and AI knowledge base.

Retrieval is one of several layers that reduce wrong answers; for the others (tool verification, structured outputs, business rules and post-action checks), see how to reduce AI agent hallucinations.

FAQ

Common questions.

A technique where an AI system first retrieves relevant information from your own sources, such as documents or databases, and then gives it to a language model to generate an answer grounded in that information, usually with citations.

Models do not know your private or recent information and can produce confident but wrong answers. RAG supplies the right source material at answer time, which improves accuracy, freshness and traceability.

Data connectors and ingestion, parsing and chunking, embeddings and an index (vector, keyword or both), a retriever with filters, optional reranking, a generation step with instructions and citations, and evaluation and monitoring.

Not necessarily. You need a searchable index; that can be a vector database, Postgres with pgvector, or a search engine with vector and keyword support. Many systems use hybrid search.

Usually retrieval: documents missing or stale, poor parsing, chunks that split key information, missing keyword matching for codes and names, no reranking, or permissions filtering out the right source. Generation issues are less common than teams expect.

Separately for retrieval (did the right sources appear in the top results?) and generation (is the answer faithful to sources, complete and correct?), using a question set with known answers and sources.

Costs include embedding documents (mostly one-time plus updates), index storage, retrieval infrastructure and model tokens per question, which grow with how much context you include. Reranking and larger models add cost per query.

It reduces them but does not eliminate them. Instructions to answer only from sources, citation checks, refusal when evidence is missing and evaluation of faithfulness are still needed.

It can be, if retrieval enforces the same permissions as the source systems, data is processed by approved providers and retrieved content is treated as untrusted input that cannot trigger actions.

When the problem is the model's behaviour, format or specialised skill rather than missing knowledge. For knowledge that changes, RAG is usually the better fit.

Often, if each component is placed carefully: the vector store and application in an Australian cloud region, and a model available for processing in that region. Some model options route requests to other regions, so check provider documentation. Logs, evaluation tools and embeddings services are frequently overlooked.

Get in touch

Have a project in mind?

Whether you're building a new digital product, improving an existing website, or looking to automate part of your business — let's talk.