How to Build a RAG System: Complete Technical Guide

What Is RAG and Why Should You Build It?

Retrieval Augmented Generation (RAG) solves the fundamental limitation of large language models: they know only what they were trained on. RAG connects an LLM to your proprietary documents, databases, and knowledge at inference time — grounding every response in your actual data. A well-built RAG system can answer "What does our refund policy say about digital goods?" or "What were the Q3 revenue figures for the DACH region?" accurately, citing specific sources, without any model fine-tuning.

RAG was formalised in the 2020 paper by Lewis et al. at Facebook AI Research and has become the dominant architecture for enterprise AI applications because it offers the best trade-off between accuracy, maintainability, and cost. Unlike fine-tuning, RAG knowledge can be updated instantly — add a new document and it's immediately available, with no retraining required.

RAG Architecture Overview

A complete RAG system has two distinct pipelines:

1. Indexing pipeline (offline) — processes your documents into searchable vector representations
2. Query pipeline (online) — handles user queries by retrieving relevant content and generating responses

Building the Indexing Pipeline

Step 1: Document Ingestion

Your RAG system is only as good as the documents it can access. Build ingestion connectors for all relevant data sources:

• PDF documents — Use PyMuPDF (fitz) for high-quality text extraction preserving layout. For scanned PDFs, add OCR via Tesseract or AWS Textract.
• Microsoft Office — python-docx for Word, openpyxl for Excel. LlamaIndex provides ready-made loaders for most common formats.
• Web content — Playwright for JavaScript-rendered pages, BeautifulSoup for static HTML. Strip navigation, footers, and boilerplate before indexing.
• Databases — Query and format structured data as natural-language descriptions or JSON for indexing. For SQL databases, consider indexing schema documentation and generating natural-language summaries of key datasets.
• Confluence / Notion / SharePoint — API connectors available through LlamaIndex and LangChain. For SharePoint, Microsoft Graph API provides programmatic access.

Key principle: Garbage in, garbage out. Invest time in cleaning your documents — removing boilerplate, fixing encoding issues, handling tables and charts — before indexing. Poor-quality source documents are the most common cause of poor RAG system quality.

Step 2: Chunking Strategy

Chunking divides documents into segments small enough to be meaningfully compared by vector similarity. Chunk size is one of the most impactful and most misunderstood parameters in RAG design.

Fixed-Size Chunking

Split text every N tokens with a configurable overlap. Simple, predictable, but breaks semantic units (sentences, paragraphs) arbitrarily.

• Chunk size: 512–1,000 tokens
• Overlap: 10–20% of chunk size
• Best for: unstructured prose where paragraph boundaries are unclear

Semantic Chunking

Split at natural semantic boundaries — sentences, paragraphs, sections. Preserve meaning at the cost of variable chunk sizes.

• Use spaCy or NLTK for sentence boundary detection
• Split on paragraph markers ( ) and section headers
• Best for: well-structured documents (manuals, policies, technical documentation)

Hierarchical Chunking

Create both large "parent" chunks (full paragraphs or sections) and small "child" chunks (individual sentences). Retrieve using child chunks (high precision), but return parent chunks to the model (more context). This parent-child approach significantly improves answer quality for complex questions that require paragraph-level context.

Implementation: LlamaIndex's ParentDocumentRetriever or LangChain's equivalent provide ready-made implementations.

Step 3: Embedding Generation

Embeddings are dense vector representations that capture semantic meaning. Similar texts have similar vectors — this is what enables semantic search.

Embedding Model Selection

Model	Dimensions	Cost	Quality	Best For
OpenAI text-embedding-3-large	3,072	€0.00013/1K tokens	Excellent	Production, general purpose
OpenAI text-embedding-3-small	1,536	€0.000020/1K tokens	Very Good	Cost-sensitive production
Cohere embed-v3-multilingual	1,024	€0.00010/1K tokens	Excellent (multilingual)	EU multilingual content
BGE-m3 (BAAI, open source)	1,024	Self-hosted GPU	Excellent	GDPR-sensitive (self-hosted)
sentence-transformers/all-MiniLM-L6	384	Free (self-hosted CPU)	Good	Development, low-budget

Critical: Use the same embedding model for indexing and querying. Never mix models — a vector from model A is not semantically comparable to a vector from model B.

Step 4: Vector Database Selection

Pinecone

Managed cloud vector database. Simplest to get started, strong performance at scale, automatic scaling, multi-tenant support.

• Pros: No infrastructure management, fast, excellent developer experience
• Cons: Data leaves your infrastructure (GDPR consideration), cost at scale (€70–€700/month for production)
• Best for: Fast production deployments where data sensitivity is manageable

Weaviate

Open source, deployable on your own infrastructure or managed cloud. Supports hybrid search (vector + keyword) natively. Strong EU compliance story with EU-based cloud option.

• Pros: Self-hostable, hybrid search built-in, GraphQL API, active community
• Cons: More infrastructure to manage, managed offering less mature than Pinecone
• Best for: Enterprises requiring self-hosted or EU-region managed deployment

pgvector (PostgreSQL Extension)

Add vector storage and similarity search to your existing PostgreSQL database. If you already run PostgreSQL, pgvector is often the simplest option for RAG at moderate scale.

• Pros: No new infrastructure, leverages existing PostgreSQL tooling, strong ACID guarantees, GDPR-simple (your existing DB, your existing controls)
• Cons: Performance ceiling (millions not billions of vectors), requires PostgreSQL tuning for vector workloads
• Best for: Up to ~1 million vectors, existing PostgreSQL users, GDPR-sensitive deployments

Qdrant

Rust-based open source vector database with excellent performance and filtering capabilities. Strong EU story — Qdrant is a German company with EU-based cloud.

• Pros: Excellent performance, rich filtering, self-hostable, EU cloud available
• Cons: Smaller ecosystem than Pinecone/Weaviate
• Best for: Performance-critical applications, EU data residency requirements

Building the Query Pipeline

Step 5: Query Processing

Raw user queries often perform poorly in vector search. Improve retrieval with:

Query Rewriting

Use an LLM to rewrite the user's query before searching. Expand abbreviations, add context, generate multiple phrasings.

System: Rewrite the following user question to be more specific and expand any abbreviations. Generate 3 alternative phrasings.
User: What's our SLA for P1 incidents?

Search with all 4 versions (original + 3 rewrites), merge results.

Hypothetical Document Embeddings (HyDE)

Generate a hypothetical answer to the query, embed the hypothetical answer, and search for documents similar to the hypothetical answer. This often significantly improves recall for questions where the documents use different terminology than the query.

Step 6: Retrieval Optimisation

Hybrid Search

Combine vector similarity search with BM25 keyword search. Keyword search catches exact-match terms (product codes, names, technical identifiers) that vector search misses; vector search catches semantic similarity that keyword search misses.

Implementation with Weaviate or Qdrant: both support hybrid search natively with configurable alpha (weighting between vector and keyword). Starting point: alpha = 0.5 (equal weight), tune based on evaluation.

Reranking

Retrieved chunks are ordered by approximate similarity. A cross-encoder reranker re-scores the top-K results with a more accurate (but slower) model. Use Cohere Rerank or a sentence-transformers cross-encoder to rerank the top 20 chunks and return the top 5 to the LLM.

Cost: Cohere Rerank costs ~€0.002 per 1,000 chunks ranked. For a system retrieving 20 chunks per query at 10,000 queries/day: €0.40/day. Well worth the quality improvement.

Step 7: Context Assembly and Prompt Construction

Assemble the final prompt with retrieved context, conversation history, and system instructions:

System: You are a helpful assistant for Acme Corp. Answer questions based ONLY on the provided context. If the answer is not in the context, say so. Always cite the source document.

Context:
[CHUNK 1 - Source: refund-policy-v3.pdf, Section 3.2]
Digital goods are eligible for refund within 14 days of purchase if...

[CHUNK 2 - Source: faq-digital-products.pdf]
Our digital products policy aligns with EU Consumer Rights Directive...

User: Can I get a refund on a digital download I bought last week?

Key construction principles:

• Place retrieved context after the system prompt but before the user query
• Include source metadata (filename, section, date) so the model can cite sources
• Explicitly instruct the model to say "I don't know" rather than hallucinating
• Include conversation history for multi-turn interactions (last 3–5 turns typically sufficient)

Evaluation: Measuring RAG Quality

A RAG system without evaluation metrics cannot be improved systematically. Implement these metrics:

• Context Precision — Of the chunks retrieved, what percentage were actually relevant? Measures retrieval precision.
• Context Recall — Of all relevant documents in the knowledge base, what percentage were retrieved? Measures retrieval completeness.
• Answer Faithfulness — Is every claim in the generated answer supported by the retrieved context? Measures hallucination rate.
• Answer Relevance — Does the generated answer actually address the user's question? Measures response quality.

Tools: RAGAS (open source) provides automated evaluation of all four metrics against a test question set. Build an evaluation set of 50–100 representative questions with ground-truth answers before deployment, and run evaluation on every system change.

Production Deployment Checklist

• Vector database running with replication and backup enabled
• Embedding model API calls have retry logic and circuit breakers
• LLM API calls have timeout, retry, and fallback model configured
• Semantic caching enabled for frequent/repeated queries
• All LLM calls logged with input/output/latency/cost attribution
• RAGAS evaluation running on weekly sample of production queries
• Cost alerting configured with budget caps
• Access controls implemented at vector DB retrieval layer
• PII redaction applied before logging user queries

For enterprise architecture guidance on the broader LLM stack, see our enterprise LLM architecture guide. For EU compliance considerations, our EU AI strategy guide covers GDPR and the AI Act. Our LLM integration service handles end-to-end RAG design and deployment.