Why Most RAG Systems Disappoint
Retrieval-Augmented Generation promises to ground LLM responses in your proprietary data. In practice, most RAG implementations produce answers that are confidently wrong, missing critical context, or retrieving irrelevant documents. A 2024 survey from Stanford found that naive RAG implementations answer correctly only 45-55% of the time on domain-specific benchmarks, compared to 85-95% for well-engineered systems.
The gap is not in the LLM — it is in the retrieval. This guide covers the engineering decisions that close that gap, based on RAG systems we have built for enterprise clients handling hundreds of thousands of documents.
The RAG Architecture Stack
A production RAG system has five layers, each with critical design decisions:
- Data Ingestion — Parsing, cleaning, and structuring source documents
- Chunking — Breaking documents into retrievable units
- Embedding — Converting chunks into vector representations
- Retrieval — Finding the most relevant chunks for a query
- Generation — Feeding retrieved context to the LLM for answer synthesis
Most teams spend 90% of their time on layer 5 (prompt engineering) and 10% on layers 1-4. Invert this. The retrieval quality determines the ceiling of your system's performance.
Layer 1: Data Ingestion Done Right
Garbage in, garbage out. If your parser mangles tables, strips headers, or loses document structure, no amount of prompt engineering will fix it.
- PDFs: Use Unstructured.io or LlamaParse for layout-aware PDF parsing. Standard text extraction (PyPDF2) loses table structure and multi-column layouts.
- HTML: Strip navigation, footers, and boilerplate. Extract the semantic content and preserve heading hierarchy.
- Structured data: Convert database records into natural language descriptions. A row with
status: "overdue", amount: 5000should become "Invoice #1234 is overdue with an outstanding amount of EUR 5,000."
// Example: structured data ingestion for RAG
function recordToDocument(invoice: Invoice): string {
return [
`Invoice ${invoice.id} for ${invoice.customerName}.`,
`Amount: EUR ${invoice.amount.toLocaleString()}.`,
`Status: ${invoice.status}.`,
`Due date: ${invoice.dueDate.toISOString().split("T")[0]}.`,
invoice.isOverdue ? "This invoice is overdue." : "",
invoice.notes ? `Notes: ${invoice.notes}` : "",
].filter(Boolean).join(" ");
}Layer 2: Chunking Strategy — The Most Underrated Decision
Chunking determines what the retriever can find. Get it wrong and relevant information becomes invisible.
Chunk Size
Smaller chunks (200-400 tokens) improve retrieval precision — the chunk is more likely to be entirely relevant to the query. Larger chunks (800-1500 tokens) provide more context to the LLM but may dilute relevance. Our recommendation:
- Factual Q&A: 200-400 tokens with 50-token overlap
- Summarisation/analysis: 800-1200 tokens with 200-token overlap
- Code documentation: Chunk by function/class, not by token count
Semantic Chunking
Rather than splitting on arbitrary token boundaries, split on semantic boundaries — paragraph breaks, section headers, topic changes. This preserves the coherence of each chunk:
// Semantic chunking using heading-based splitting
function semanticChunk(markdown: string, maxTokens: number = 400): Chunk[] {
const sections = markdown.split(/(?=^#{1,3} )/gm);
const chunks: Chunk[] = [];
for (const section of sections) {
const heading = section.match(/^(#{1,3}) (.+)/)?.[2] || "Untitled";
if (estimateTokens(section) <= maxTokens) {
chunks.push({ text: section.trim(), metadata: { heading } });
} else {
// Split large sections by paragraph with overlap
const paragraphs = section.split(/\n\n+/);
let buffer = "";
for (const para of paragraphs) {
if (estimateTokens(buffer + para) > maxTokens && buffer) {
chunks.push({ text: buffer.trim(), metadata: { heading } });
// Keep last paragraph as overlap
buffer = para;
} else {
buffer += (buffer ? "\n\n" : "") + para;
}
}
if (buffer) chunks.push({ text: buffer.trim(), metadata: { heading } });
}
}
return chunks;
}Parent-Child Chunking (The Secret Weapon)
Embed small chunks for precise retrieval, but return the parent chunk (or full section) as context to the LLM. This gives you the best of both worlds — precise retrieval with rich context. LlamaIndex implements this as the "Small-to-Big" retrieval pattern.
Layer 3: Embedding Model Selection
The embedding model determines the quality of semantic similarity matching. In 2026, the top performers on the MTEB benchmark are:
| Model | Dimensions | MTEB Score | Best For |
|---|---|---|---|
| Cohere embed-v4 | 1024 | 70.4 | General purpose, multilingual |
| OpenAI text-embedding-3-large | 3072 | 69.6 | Highest quality, higher cost |
| Voyage-3-large | 1024 | 68.5 | Code and technical content |
| BGE-M3 | 1024 | 66.1 | Open source, self-hosted |
Key decision: If your documents are multilingual (common for European companies), test multilingual models explicitly on your language mix. English-optimised models lose 10-20% retrieval quality on Dutch, German, or French content.
Layer 4: Retrieval Architecture
Pure vector similarity search is not enough for production systems. Combine multiple retrieval strategies:
Hybrid Search (Vector + Keyword)
Vector search captures semantic similarity; keyword search captures exact matches for product names, error codes, and technical terms. Combine them with reciprocal rank fusion:
// Hybrid retrieval with reciprocal rank fusion
async function hybridRetrieve(
query: string,
vectorStore: VectorStore,
k: number = 10
): Promise<Document[]> {
// Run vector and keyword search in parallel
const [vectorResults, keywordResults] = await Promise.all([
vectorStore.vectorSearch(query, k * 2),
vectorStore.keywordSearch(query, k * 2),
]);
// Reciprocal rank fusion
const scores = new Map<string, number>();
const RRF_K = 60;
vectorResults.forEach((doc, rank) => {
const score = 1 / (RRF_K + rank + 1);
scores.set(doc.id, (scores.get(doc.id) || 0) + score);
});
keywordResults.forEach((doc, rank) => {
const score = 1 / (RRF_K + rank + 1);
scores.set(doc.id, (scores.get(doc.id) || 0) + score);
});
// Sort by combined score and return top k
return [...scores.entries()]
.sort((a, b) => b[1] - a[1])
.slice(0, k)
.map(([id]) => vectorStore.getById(id));
}Re-Ranking
After initial retrieval, use a cross-encoder re-ranker (Cohere Rerank, Jina Reranker) to re-score the top 20-50 results. Cross-encoders are slower but significantly more accurate than bi-encoder embeddings. This step typically improves top-5 retrieval accuracy by 15-30%.
Query Transformation
User queries are often vague or poorly structured. Transform them before retrieval:
- HyDE (Hypothetical Document Embedding): Use the LLM to generate a hypothetical answer, then embed and search for that answer. This bridges the vocabulary gap between user queries and documents.
- Multi-query: Generate 3-5 rephrased versions of the query and retrieve for all of them, then deduplicate results.
- Step-back prompting: For specific questions, generate a more general version of the query to retrieve broader context first.
Layer 5: Generation with Retrieved Context
With high-quality retrieval in place, the generation layer is straightforward. Key practices:
- Include source metadata (document title, section, page number) in the context so the LLM can cite sources
- Instruct the model to say "I don't have enough information" when the retrieved context does not contain the answer
- Limit the number of retrieved chunks to 5-10 — more context is not always better and increases latency and cost
Evaluation Framework
Measure your RAG system on three axes:
- Retrieval quality: Are the right documents being retrieved? Measure with Recall@k and MRR (Mean Reciprocal Rank).
- Answer quality: Is the generated answer correct and complete? Use human evaluation or LLM-as-judge on a golden dataset.
- Faithfulness: Is the answer grounded in the retrieved documents? Detect hallucinations by checking if claims are supported by the context.
Use frameworks like RAGAS to automate this evaluation across your test set.
Build Your RAG System Right
If you are building a RAG system for your organisation, invest 70% of your effort in layers 1-4 and 30% in prompt engineering. The retrieval quality determines the ceiling; prompt engineering determines how close you get to it.
Our AI consulting team has built RAG systems handling 500K+ documents for enterprise clients. Get in touch for a free architecture review of your RAG implementation.