RankGPT: LLM-as-Reranker

Why and When

LLM rerankers win when:

• The query contains instructions ("prefer recent docs", "exclude deprecated APIs").
• The corpus is too niche for pretrained rerankers (internal jargon, proprietary ontology).
• The system needs explanations alongside rankings.
• You have very few labeled examples — an LLM needs zero.

They lose on:

• Latency: 300-2000 ms vs 50-300 ms for cross-encoders.
• Cost: $3-30 per 1k queries vs <$2 for Cohere Rerank.
• Throughput: rate limits cap QPS without batching.

Three Prompting Styles

Listwise is the RankGPT default.

Listwise Reranking with Claude

# pip install anthropic
from anthropic import Anthropic
import json, re

client = Anthropic()
MODEL = "claude-haiku-4-5-20250929"  # cheap, fast; upgrade to sonnet for hard queries

LISTWISE_PROMPT = """You rank passages by relevance to a user query.

Query: {query}

Passages (each starts with its index in brackets):
{passages}

Return a JSON object with a single field "ranking": an array of passage indexes
ordered from most to least relevant. Include every index exactly once.
No other text, no explanations."""

def format_passages(docs: list[str]) -> str:
    return "\n\n".join(f"[{i}] {d}" for i, d in enumerate(docs))

def listwise_rank(query: str, docs: list[str]) -> list[int]:
    msg = client.messages.create(
        model=MODEL,
        max_tokens=1024,
        messages=[{
            "role": "user",
            "content": LISTWISE_PROMPT.format(query=query, passages=format_passages(docs)),
        }],
    )
    text = msg.content[0].text
    match = re.search(r"\{[\s\S]*\}", text)
    data = json.loads(match.group(0))
    ranking = data["ranking"]
    # Safety: fill missing indexes at the end
    seen = set(ranking)
    missing = [i for i in range(len(docs)) if i not in seen]
    return ranking + missing

Sliding Window for Long Candidate Lists

Claude handles > 100 passages in one prompt but prompt length grows quickly. The RankGPT paper uses a sliding window:

[0..19] rank -> keep top 10
[10..29] rank (seeded with previous top 10) -> keep top 10
[20..39] ...

def sliding_rerank(query: str, docs: list[str], window: int = 20,
                   step: int = 10, keep: int = 10) -> list[int]:
    n = len(docs)
    if n <= window:
        return listwise_rank(query, docs)[:keep]

    # Start from the end so the best candidates bubble to position 0
    order = list(range(n))
    start = n - window
    while start >= 0:
        end = start + window
        slice_docs = [docs[i] for i in order[start:end]]
        local = listwise_rank(query, slice_docs)
        order[start:end] = [order[start + idx] for idx in local]
        start -= step
    return order[:keep]

End-to-start scan is the trick: the strongest document reaches the front of the list as windows slide back.

Pairwise (Highest Quality, Most Expensive)

Pairwise calls scale as O(n log n) with merge-sort style comparisons:

PAIRWISE_PROMPT = """Given the query, which passage is more relevant? Answer with "A" or "B" only.

Query: {query}

Passage A:
{a}

Passage B:
{b}

Answer:"""

def pairwise_compare(query: str, a: str, b: str) -> int:
    msg = client.messages.create(
        model=MODEL,
        max_tokens=5,
        messages=[{"role": "user",
                   "content": PAIRWISE_PROMPT.format(query=query, a=a, b=b)}],
    )
    return -1 if msg.content[0].text.strip().upper().startswith("A") else 1

import functools
def merge_sort_rerank(query: str, docs: list[str]) -> list[int]:
    idx = list(range(len(docs)))
    def cmp(i, j):
        return pairwise_compare(query, docs[i], docs[j])
    return sorted(idx, key=functools.cmp_to_key(cmp))

Avoid naive bubble-sort (O(n^2) calls). Use merge-sort or tournament bracket.

Pointwise (Cheapest, Most Parallel)

import asyncio
from anthropic import AsyncAnthropic
aclient = AsyncAnthropic()

POINTWISE_PROMPT = """Rate the relevance of the passage to the query from 0 to 10.
Return a JSON object: {{"score": <number>}}

Query: {query}

Passage: {passage}"""

async def point_score(query: str, passage: str) -> float:
    msg = await aclient.messages.create(
        model=MODEL, max_tokens=40,
        messages=[{"role": "user",
                   "content": POINTWISE_PROMPT.format(query=query, passage=passage)}],
    )
    try:
        return float(json.loads(re.search(r"\{[^}]*\}", msg.content[0].text).group(0))["score"])
    except Exception:
        return 0.0

async def pointwise_rerank(query: str, docs: list[str], top_n: int = 5) -> list[int]:
    scores = await asyncio.gather(*(point_score(query, d) for d in docs))
    return sorted(range(len(docs)), key=lambda i: scores[i], reverse=True)[:top_n]

Run calls concurrently with a semaphore to respect rate limits.

Structured Output for Robustness

Anthropic tool-use forces a JSON shape — no regex needed:

tools = [{
    "name": "submit_ranking",
    "description": "Submit the ranking of passage indexes from most to least relevant.",
    "input_schema": {
        "type": "object",
        "properties": {
            "ranking": {
                "type": "array",
                "items": {"type": "integer"},
                "description": "Passage indexes in descending relevance order",
            },
        },
        "required": ["ranking"],
    },
}]

def listwise_structured(query: str, docs: list[str]) -> list[int]:
    msg = client.messages.create(
        model=MODEL, max_tokens=1024, tools=tools, tool_choice={"type": "tool", "name": "submit_ranking"},
        messages=[{"role": "user",
                   "content": LISTWISE_PROMPT.format(query=query, passages=format_passages(docs))}],
    )
    for block in msg.content:
        if block.type == "tool_use" and block.name == "submit_ranking":
            return block.input["ranking"]
    raise ValueError("No ranking returned")

Always prefer tool-use over regex JSON extraction in production.

Calibration: Force Consistency

LLMs are sensitive to passage order inside the prompt. Two tricks:

1. Shuffle input order each call; rerank yields a relative order, not absolute.
2. Run two passes with different initial orders; fuse with RRF.

import random

def robust_listwise(query: str, docs: list[str], passes: int = 3) -> list[int]:
    from collections import defaultdict
    scores = defaultdict(float)
    for _ in range(passes):
        perm = list(range(len(docs)))
        random.shuffle(perm)
        shuffled = [docs[i] for i in perm]
        ranking = listwise_structured(query, shuffled)
        for rank, local_idx in enumerate(ranking):
            real_idx = perm[local_idx]
            scores[real_idx] += 1.0 / (60 + rank + 1)  # RRF
    return sorted(scores, key=scores.get, reverse=True)

Cost and Latency vs Cohere Rerank

Pick Haiku listwise with sliding window as the default LLM-reranker. Upgrade to Sonnet only for hard queries.

Prompt Caching for Cost Control

The query + passages prompt has a large fixed header (instructions, passages rarely change across a user session). Cache it via Anthropic's prompt caching:

msg = client.messages.create(
    model=MODEL, max_tokens=1024,
    system=[
        {"type": "text", "text": "You rank passages by relevance...",
         "cache_control": {"type": "ephemeral"}},
    ],
    messages=[{"role": "user", "content": LISTWISE_PROMPT.format(...)}],
)

Cache hits cut cost ~90% and latency ~30-50% for repeat calls within 5 minutes.

When to Fall Back

Always wrap LLM reranking with a timeout and fallback:

import asyncio

async def rerank_with_fallback(query, docs, top_n=5, timeout_s=2.0):
    try:
        return await asyncio.wait_for(
            pointwise_rerank(query, docs, top_n), timeout=timeout_s
        )
    except (asyncio.TimeoutError, Exception):
        return list(range(top_n))  # retrieval order

LLM rate limits or transient errors must not break the user-facing query path.

Anti-Patterns

Production Checklist

• Listwise + sliding window default; pairwise only for high-stakes queries
• Claude Haiku as default; Sonnet as opt-in for hard queries
• Tool-use for structured ranking output (no regex fallback)
• Prompt caching enabled on system prompt
• Hard timeout with retrieval-order fallback
• Rate-limit-aware semaphore around async calls
• Rerank quality A/B tested against Cohere Rerank monthly
• Cost per query dashboard (input + cached + output tokens)
• Multi-pass shuffle + RRF for stability on sensitive queries
• Defensive missing-index handling in the parser