RAG System Design: Architecture, Trade-offs & Production Patterns

RAG (Retrieval-Augmented Generation) is the dominant pattern for giving LLMs access to private, domain-specific, or up-to-date knowledge. A naive implementation is easy to build. A production-ready one requires careful decisions at every stage of the pipeline.

This breakdown covers every layer of the RAG architecture, the trade-offs at each decision point, and the questions you need to answer before you ship.

The RAG Pipeline Architecture

A production RAG system has two phases:

INGESTION PHASE
Documents → Chunking → Embedding → Vector Store

QUERY PHASE User Query → Query Transform → Retrieval → Reranking → LLM → Response ```

Each arrow is a decision. Each decision has trade-offs.

1. Document Ingestion

Chunking Strategy

Chunking is how you split documents before embedding them. The wrong chunk size is the most common reason RAG systems return irrelevant context.

• Fixed-size chunks (e.g., 512 tokens, 50 token overlap): Simple and fast. Splits mid-sentence. Best for uniform, dense text like legal documents or manuals.
• Semantic chunking: Split at natural boundaries (paragraphs, headings). Preserves meaning. Requires a parser. Best for structured documents like docs, articles, and wikis.
• Hierarchical chunking: Store both small (sentence-level) and large (paragraph-level) chunks. Retrieve small, pass large. Best recall and context quality but doubles storage.

Rule of thumb: chunk size should match the granularity of the questions users will ask. If users ask about specific facts, use small chunks. If they ask for explanations, use larger chunks.

Embedding Models

The embedding model converts text to vectors. The choice affects retrieval quality significantly.

• text-embedding-3-small (OpenAI): Fast, cheap, good for English. 1536 dimensions.
• text-embedding-3-large (OpenAI): Higher quality, 3072 dimensions, higher cost.
• bge-m3 (BAAI): Open-source, multilingual, strong benchmark performance.
• Domain-specific embeddings: Fine-tuned on your corpus. Best retrieval quality, highest cost to produce.

Never mix embedding models in the same index. If you upgrade the model, re-embed the entire corpus.

2. Vector Storage

Choosing a Vector Database

Index Types

• Flat index (exact search): 100% recall, $O(N)$ query time. Fine up to ~100K vectors.
• HNSW: Sub-linear query time, ~95–98% recall. Standard for production.
• IVF (Inverted File Index): Good for very large datasets (>10M vectors). Requires tuning nlist and nprobe.

-- pgvector: create HNSW index
CREATE INDEX ON documents
USING hnsw (embedding vector_cosine_ops)
WITH (m = 16, ef_construction = 64);

3. Query Phase

Query Transformation

Never send the raw user query to the vector store. Transform it first.

• HyDE (Hypothetical Document Embeddings): Use the LLM to generate a hypothetical answer to the query, then embed that answer and retrieve with it. Improves recall by 10–25% on knowledge-dense tasks.
• Query expansion: Rephrase the query in multiple ways, retrieve with each, merge results.
• Step-back prompting: Abstract the query to a broader question before retrieving, then answer the specific question.

# HyDE example
hypothetical_answer = llm.invoke(
    f"Write a paragraph that would answer: {user_query}"
)
results = vector_store.similarity_search(hypothetical_answer, k=10)

Hybrid Search

Combine dense (vector) and sparse (BM25/keyword) retrieval for best results:

• Dense retrieval catches semantic intent: "how do I cancel?" matches "subscription termination"
• Sparse retrieval catches exact terms: product codes, names, acronyms

Fuse results with Reciprocal Rank Fusion (RRF):

def rrf_score(ranks, k=60):
    return sum(1 / (k + r) for r in ranks)

Hybrid search consistently outperforms either approach alone by 5–15% on recall@10.

Reranking

Reranking re-scores the top-K retrieved documents using a more expensive cross-encoder model. Retrieve 20–50 candidates, rerank, pass top 3–5 to the LLM.

• Cohere Rerank: Strong performance, API-based, adds ~100ms latency.
• BGE-Reranker: Open-source, self-hosted, comparable quality.
• Cross-encoder (sentence-transformers): Full control, highest latency.

Reranking typically improves answer quality by 15–20% at the cost of 100–300ms additional latency. Worth it for high-value workflows.

4. Context Assembly

How many chunks to pass to the LLM?

More context = more tokens = higher cost and latency. Too little = missing information.

Common approach: retrieve top-10, rerank, pass top-3. Increase to top-5 for complex queries.

Lost-in-the-Middle Problem

LLMs pay more attention to content at the start and end of the context window. Critical information buried in the middle of a long context gets ignored.

Solution: put the most relevant chunks first, less relevant ones last.

Citation and Grounding

Always pass document metadata (source, page, section) alongside content. Generate responses that cite sources. This enables: - User trust verification - Hallucination detection - Audit trails

5. Evaluation

You cannot improve RAG without measuring it. The core metrics:

Retrieval Metrics - **Recall@K**: What fraction of relevant documents appear in the top-K results? - **MRR (Mean Reciprocal Rank)**: How high is the first relevant document ranked? - **Context Relevance**: Are the retrieved chunks actually relevant to the query?

Generation Metrics - **Faithfulness**: Does the answer stay grounded in the retrieved context? (No hallucinations) - **Answer Relevance**: Does the answer actually address the question? - **Answer Correctness**: Is the answer factually correct?

Evaluation Frameworks - **RAGAs**: Open-source, computes faithfulness and relevance automatically. - **TruLens**: Tracks feedback functions over time, integrates with LangChain. - **LangSmith**: Full trace capture with annotation and comparison views.

from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy, context_recall

scores = evaluate( dataset, metrics=[faithfulness, answer_relevancy, context_recall] ) ```

6. Production Architecture Decisions

When to add a semantic cache?

Cache embeddings of past queries and their responses. If a new query is semantically similar (cosine similarity > 0.95), return the cached response. Reduces LLM calls by 30–60% for repetitive query patterns.

# GPTCache / custom semantic cache
cached = cache.lookup(query_embedding, threshold=0.95)
if cached:
    return cached.response

Handling stale documents

Vector stores don't auto-update. Design a document lifecycle: - Track document versions and last-ingested timestamps - Delete and re-embed when source documents change - Use soft deletes + filtering by updated_at metadata

Multi-tenancy

Store tenant_id in vector metadata. Filter by tenant at query time — never retrieve across tenant boundaries:

results = vector_store.similarity_search(
    query_embedding,
    filter={"tenant_id": current_user.org_id}
)

Common RAG Failure Modes