Long Context vs RAG

The Decision Has Shifted

As of 2025:

Claude Sonnet 4.5: 200k tokens native, 1M in extended context mode.
Gemini 2.5 Pro: 2M tokens.
GPT-4o family: 128k typical.
Prompt caching makes re-reading large prefixes ~10x cheaper.

This invalidates old heuristics. RAG is no longer automatic; below certain corpus sizes, long-context is cheaper and better.

Decision Tree

Total corpus tokens?
  < 50k  -> Stuff the prompt. No RAG. No caching needed.
  50k-200k
    Query volume high and prompt is stable? -> Long context + prompt caching.
    Low volume or rapidly changing?         -> Long context, no caching.
  200k-2M
    Single user, interactive? -> Long context (Gemini 2M, Claude 1M).
    High concurrency?         -> RAG.
  > 2M   -> RAG, no choice.

Additional overrides:

Need auditability with citations? -> RAG (citations harder in long context).
Content changes hourly? -> RAG (cache invalidates).
Regulatory requirement to prove what was retrieved? -> RAG.

Cost Math: The Real Numbers

Claude Sonnet 4.5 pricing (approximate 2025, subject to change):

Input: $3 / M tokens
Output: $15 / M tokens
Cache write: 1.25x input
Cache read: 0.1x input (10% of base)

Example 1: 150k token corpus, 10 queries/min

Stuff without caching

Per query: 150k in + 500 out = $0.4575
10/min * 60min = 600 queries/hour = $274/hour

Stuff with caching

Cache write (once / 5min): 150k * 1.25 * 12 (per hour) = 2.25M = $6.75
Cache read per query: 150k * 0.1 = 15k effective = $0.045
Per query: $0.045 + (500 * 0.000015) = ~$0.053
600 queries: $6.75 + 600 * $0.053 = ~$38/hour
Savings: 86%

RAG (5k retrieved tokens per query)

Retrieval: ~$0.0003 (OpenAI text-embedding-3-small)
LLM: 5k in + 500 out = $0.0225
Per query: ~$0.023
600 queries: $14/hour
Cheapest — but infra cost (vector DB, reranker) is amortized

At > 10 queries/min and corpus < 200k tokens, cached long-context is competitive with small-corpus RAG once you include RAG's infra cost (vector DB, reranker API, maintenance).

Example 2: 1M token corpus, 1 query/min

Stuff with caching

Cache write: 1M * 1.25 = 1.25M = $3.75 per 5-min window = $45/hour
Read per query: 1M * 0.1 = 100k = $0.30
60 queries: $45 + 60 * $0.30 = $63/hour

RAG

60 queries * $0.023 = $1.38/hour
RAG wins 45x

Rule: above ~300k tokens, RAG dominates on cost regardless of caching.

Quality: The Lost-in-the-Middle / Context Rot Problem

Research (Liu et al. 2023, Anthropic 2024 needle-in-haystack, Databricks 2024):

Accuracy drops when relevant info is in the middle of a long context.
Claude, GPT-4, and Gemini all show degradation > 100k tokens on multi-fact synthesis tasks (even when needle retrieval is ~100%).
Distractors hurt: 20 irrelevant chunks + 1 relevant is harder than 1 relevant alone.

Task type	Long context strength	RAG strength
Needle retrieval (1 fact)	Strong ~100% up to context limit	Strong if retrieval works
Synthesis across many parts	Degrades above 50-100k	Works — the parts are selected
Comparison across sources	Good if all fit	Better — explicit top-K
Summarization of whole corpus	Unique strength of long context	Cannot — RAG sees only top-K
Reasoning over many entities	Degrades	RAG focuses on the relevant entities

Choose long context when the query needs the whole document. Choose RAG when the query needs a small fraction.

Hybrid: RAG Narrows, Long Context Reads

Use RAG to filter to the top-N relevant documents, then stuff those documents (not chunks) into long context. Best of both:

RAG avoids paying for 90% irrelevant content.
Long context keeps doc-level coherence (no chunking artifacts).

from langchain_anthropic import ChatAnthropic
from anthropic import Anthropic

retriever_client = Anthropic()
llm = ChatAnthropic(model="claude-sonnet-4-5-20250929", max_tokens=2048)

def hybrid_long_rag(question: str, top_doc_ids_k: int = 5):
    # Step 1: RAG at document granularity (not chunk)
    doc_ids = doc_level_retriever.invoke(question)[:top_doc_ids_k]
    full_docs = [document_store.get(did) for did in doc_ids]

    # Step 2: stuff full docs into long context with caching
    total_tokens = sum(count_tokens(d.text) for d in full_docs)
    use_cache = total_tokens > 20_000  # cache breakeven threshold

    system_blocks = [
        {"type": "text", "text": f"[DOC {d.id}]\n{d.text}",
         "cache_control": {"type": "ephemeral"} if use_cache else None}
        for d in full_docs
    ]
    # Filter None cache_control fields
    system_blocks = [{k: v for k, v in b.items() if v is not None} for b in system_blocks]

    resp = retriever_client.messages.create(
        model="claude-sonnet-4-5-20250929",
        max_tokens=1024,
        system=system_blocks,
        messages=[{"role": "user", "content": question}],
    )
    return resp.content[0].text

Doc-level retrieval avoids chunking losses; caching makes re-reads cheap.

Latency Comparison

Setup	TTFT	Full response
RAG (hybrid, reranked)	800-1500ms	2-4s
Long context, first call (cold cache)	3-10s	5-20s
Long context, cached call	800-1500ms	2-5s
Hybrid RAG + long context	1200-2000ms	3-6s

Cold long context is slow. Cached long context is competitive. RAG is consistently fast because retrieval + small-context generation has low variance.

Prompt Caching: When It Helps

Caching helps when:

Prefix is reused > ~3 times within the 5-min TTL.
Prefix is > 1024 tokens (minimum cacheable size).
Prefix is stable between calls.

Caching does not help when:

Each query has a different large context.
Prefix changes frequently (content updates hourly).
Low query rate (< 1 per 5 min).

What Each Excels At

Use case	Choose
Q&A over 50k-token spec document (single user)	Long context + caching
Enterprise KB with 10k documents, 1000 daily users	RAG
Legal discovery across one case (2M pages)	Tiered: RAG to surface + long-context to read
Customer support chatbot	RAG (dynamic content, citations needed)
Code review over a single PR	Long context (whole PR + dependencies)
Code review over entire codebase	RAG + long context hybrid
Research agent needing 50 sources	RAG + agentic
Summarize this one book	Long context
News chatbot	RAG (freshness)

Needle-in-Haystack Is Not Enough

Vendors advertise near-100% recall on needle tests. This is a single-fact retrieval. Real tasks are:

Multi-fact synthesis (20 needles, combined).
Reasoning with distractors.
Dense numerical extraction.

Benchmarks that capture this better: RULER, BABILong, LongBench, HELMET. Consult these, not the marketing chart.

Migration: Starting Point

Starting a new project with a corpus under 300k tokens: begin with long context + caching. It's simpler, has fewer moving parts, no vector DB to run. Add RAG when:

Corpus grows past 300k tokens.
Per-query cost exceeds the RAG alternative.
Quality degrades on synthesis tasks.

Anti-Patterns

Anti-Pattern	Fix
RAG on a 50k-token corpus	Long context + caching is cheaper and better
Long context on a 2M-token corpus	Cost + latency punishes you; use RAG
Trusting needle-in-haystack claims for synthesis tasks	Benchmark on your own tasks
Not enabling prompt caching when prefix repeats	90% cost miss
Same context on every turn without cache	Break it into a stable system + variable suffix; cache the prefix
Ignoring long-context latency on first call	Warm the cache during app startup if possible
Long context + large chunking overhead	Remove chunking; pass whole docs
Hybrid with chunk-level RAG feeding long context	Retrieve at doc level; long context swallows the rest
Hardcoded 200k assumption	Model limits change; pull from SDK
Not measuring	Decide with your own eval set and cost dashboard

long-context-vs-rag