RAG Architecture

Three Architectural Tiers

Tier	Components	Best For	Complexity
Naive RAG	Chunk + embed + top-K + stuff	Prototype, < 10k docs, homogeneous content	Low
Advanced RAG	+ query rewriting, hybrid search, reranking, metadata filters	Production, heterogeneous content, > 10k docs	Medium
Agentic RAG	+ self-reflection, retrieval-as-tool, multi-hop, corrective fallback	Complex research, multi-source synthesis, high-stakes answers	High

Naive RAG Pipeline

[Docs] -> [Splitter] -> [Embedder] -> [Vector DB]
                                          |
[Query] -> [Embedder] -> [Top-K Search] --+-> [Prompt Stuffer] -> [LLM] -> [Answer]

Single failure mode: bad retrieval = bad answer. No recovery path. Works for well-scoped FAQ bots on small corpora.

Advanced RAG Pipeline

                          pre-retrieval         retrieval              post-retrieval
[Query] -> [Router] -> [Rewrite/HyDE/Multi-Q] -> [Hybrid Search] -> [Rerank] -> [Compress] -> [LLM]
                                                      |
                           [BM25] + [Dense Vector] + [Metadata Filter]

Each stage is independently replaceable and measurable. See query-transformations, hybrid-search, reranking.

Agentic RAG Pipeline

[Query] -> [Planner Agent]
             |
             v
       +-----+-----+-------------------+
       |           |                   |
   [Retrieve]  [Web Search]      [Code Tool]
       |           |                   |
       +-----+-----+-------------------+
             |
        [Critic / Self-reflection]
             |
       +-----+-----+
       |           |
    [Answer]  [Replan / More retrieval]

Dynamic step count, dynamic tool selection, self-correction. See agentic-rag.

Decision Tree: RAG vs Fine-Tuning vs Long Context

Is the knowledge dynamic (changes > monthly)?
  Yes -> RAG
  No  -> Continue
    |
    Is it style/format/persona (not facts)?
      Yes -> Fine-tuning (SFT or DPO)
      No  -> Continue
        |
        Does total corpus fit in 200k-1M tokens?
          Yes -> Long context with prompt caching (cheaper than RAG at small scale)
          No  -> RAG
            |
            Need factual grounding with citations?
              Yes -> RAG (mandatory for auditability)
              No  -> Hybrid: long context for recent + RAG for archive

Rules of thumb:

Under 500 KB of content: long context with prompt caching beats RAG on latency and quality.
Over 10 MB or changes weekly: RAG wins on cost and freshness.
Between: measure both.

Latency Budget (Production Target: < 3s P95)

Stage	Budget	Optimization
Query embedding	50-150ms	Batch + local model for simple queries
Query rewriting (optional)	300-800ms	Skip for short factual queries
Vector search (top-50)	20-100ms	HNSW with `ef_search` tuned
BM25 search	10-50ms	Parallel with vector search
Fusion (RRF)	< 5ms	In-memory
Reranking (top-50 -> top-5)	100-400ms	Cohere/Voyage API or local BGE
LLM generation	1000-2000ms	Streaming, prompt caching
Total	1500-3500ms

Parallelize embedding + BM25. Skip rewriting for short queries. Cache query embeddings for hot terms.

Production Component Diagram

                    +------------------+
                    |   Ingestion API  |
                    +--------+---------+
                             |
                 +-----------v------------+
                 | Chunker + Embedder     |
                 | (batch worker, queue)  |
                 +-----------+------------+
                             |
         +-------------------+---------------------+
         |                                         |
  +------v------+                          +-------v-------+
  |  Vector DB  |                          | Document Store|
  |  (HNSW)     |                          | (Postgres/S3) |
  +------+------+                          +-------+-------+
         |                                         |
         |    +------------------+                 |
         +----> Retrieval Service <----------------+
              | (hybrid + rerank)|
              +--------+---------+
                       |
              +--------v---------+
              |  LLM Gateway     |
              | (Claude/GPT)     |
              +--------+---------+
                       |
              +--------v---------+
              |   API / UI       |
              +------------------+

Separate the ingestion path from the query path. Never block queries on indexing.

Minimal Python Scaffold (Advanced RAG)

from dataclasses import dataclass
from langchain_anthropic import ChatAnthropic
from langchain_openai import OpenAIEmbeddings
from langchain_qdrant import QdrantVectorStore
from langchain_community.retrievers import BM25Retriever
from langchain.retrievers import EnsembleRetriever, ContextualCompressionRetriever
from langchain_cohere import CohereRerank

@dataclass
class RAGConfig:
    top_k_retrieve: int = 50
    top_n_rerank: int = 5
    alpha: float = 0.5  # dense weight in hybrid
    rerank_model: str = "rerank-english-v3.0"
    llm_model: str = "claude-sonnet-4-5-20250929"

def build_pipeline(docs, cfg: RAGConfig):
    emb = OpenAIEmbeddings(model="text-embedding-3-small")
    vstore = QdrantVectorStore.from_documents(docs, emb, collection_name="kb")
    dense = vstore.as_retriever(search_kwargs={"k": cfg.top_k_retrieve})
    sparse = BM25Retriever.from_documents(docs); sparse.k = cfg.top_k_retrieve
    hybrid = EnsembleRetriever(retrievers=[sparse, dense], weights=[1 - cfg.alpha, cfg.alpha])
    reranker = CohereRerank(model=cfg.rerank_model, top_n=cfg.top_n_rerank)
    retriever = ContextualCompressionRetriever(base_compressor=reranker, base_retriever=hybrid)
    llm = ChatAnthropic(model=cfg.llm_model, max_tokens=1024)
    return retriever, llm

Scaling Patterns

Regime	Index	Strategy
< 100k chunks	HNSW in-memory (FAISS, Chroma)	Single node
100k-10M chunks	Managed HNSW (Qdrant, Pinecone, Weaviate)	Replicate reads
10M-1B chunks	Sharded HNSW + IVF coarse filter	Partition by tenant/namespace
> 1B chunks	Disk-ANN / SPTAG / hierarchical	Custom; hire infra

Namespace per tenant avoids noisy-neighbor retrieval in multi-tenant SaaS.

Anti-Patterns

Anti-Pattern	Fix
Starting with agentic RAG	Start naive, measure, add complexity only when recall < 70%
Treating RAG as a solved box	Every stage needs its own eval; see `rag-evaluation`
Single index for heterogeneous content	Separate indexes per content type with a router
Synchronous indexing in query path	Queue ingestion; queries never block on embedding
No reranking above 10k docs	Reranking recovers 10-30% recall@5 vs raw vector search
Hardcoded `top_k` everywhere	Parametrize; tune with eval set
Storing chunks only in vector DB	Keep source of truth in document DB; vector DB holds refs

rag-architecture