ARES Evaluation Framework

What ARES Is

ARES (Stanford, 2024) is an automated RAG evaluation framework. Unlike RAGAS, which uses a large LLM as judge at eval time, ARES trains small classifiers (DeBERTa-size) on synthetic data, then uses prediction-powered inference (PPI) to combine classifier outputs with a small human-labeled set for statistically unbiased estimates.

Core claim: you can get tight confidence intervals on RAG quality metrics with as few as 150-300 human labels, plus thousands of synthetic examples.

Three Scored Dimensions

Each is a binary classifier producing probability scores.

Pipeline

corpus -> synth (q, ctx, a, labels) -> train 3 classifiers
                                           |
                                           v
prod logs (q, ctx, a) -> classifier predictions -> PPI  <- gold (150-300 human labels)
                                                     |
                                                     v
                                    unbiased metric + confidence interval

Setup

pip install ares-ai

Requires a local GPU (~16GB VRAM) for classifier fine-tuning.

Synthetic Query Generation

from ares import ARES

ares = ARES(
    synthetic_query_generator="google/flan-t5-xl",
    document_filepaths=["corpus.tsv"],
    few_shot_prompt_filename="few_shot_prompts.tsv",
    synthetic_queries_filenames=["synth_queries.tsv"],
    documents_sampled=10000,
)
ares.generate_synthetic_data()

The synthetic set contains:

• Relevant positive examples (query + matching context).
• Negative contexts (query + random / near-miss context).
• Hallucinated answers (inject contradictions).
• Unfaithful rewrites.

Training Judge Classifiers

classifier_cfg = {
    "classification_dataset": "synth_queries.tsv",
    "test_set_selection": "gold_test.tsv",
    "label_column": ["Context_Relevance_Label",
                     "Answer_Faithfulness_Label",
                     "Answer_Relevance_Label"],
    "num_epochs": 10,
    "patience_value": 3,
    "learning_rate": 5e-6,
    "training_dataset_path": "synth_queries.tsv",
    "assigned_batch_size": 8,
    "gradient_accumulation_multiplier": 16,
    "model_choice": "microsoft/deberta-v3-large",
}

ares.train_classifier(**classifier_cfg)

Output: three fine-tuned DeBERTa classifiers saved to disk.

Prediction-Powered Inference (PPI)

Human labels are expensive. Pure LLM-judge is cheap but biased. PPI gives unbiased estimates by correcting classifier bias using a small labeled sample.

ppi_cfg = {
    "evaluation_datasets": ["prod_logs_labeled.tsv"],  # has ~200 human labels + classifier preds
    "few_shot_examples_filepath": "few_shot_prompts.tsv",
    "checkpoints": ["ckpt_context_rel.pt",
                    "ckpt_faithfulness.pt",
                    "ckpt_answer_rel.pt"],
    "labels": ["Context_Relevance_Label",
               "Answer_Faithfulness_Label",
               "Answer_Relevance_Label"],
    "gold_label_path": "gold_labels.tsv",
}

scores = ares.evaluate_RAG(**ppi_cfg)
# {
#   "Context Relevance":  {"score": 0.83, "ci_low": 0.79, "ci_high": 0.87},
#   "Answer Faithfulness":{"score": 0.91, "ci_low": 0.88, "ci_high": 0.94},
#   "Answer Relevance":   {"score": 0.77, "ci_low": 0.72, "ci_high": 0.82},
# }

PPI output includes confidence intervals — critical for reporting and A/B tests.

PPI Math (Intuition)

Let f(x) be the classifier prediction, y be the true label on gold set of size n, and classifier predictions on an unlabeled set of size N.

mean_true ~= mean(f over unlabeled) + mean(y - f over gold)
             ^^^^^ biased estimator    ^^^^^ bias correction

CI shrinks as both n and N grow; N is cheap (classifier predictions), so PPI gives near-full-label precision with small n.

Comparison with RAGAS

When ARES Wins

• You evaluate at scale (>100k samples per run) and LLM-judge cost is prohibitive.
• You need confidence intervals for A/B tests or regulatory reporting.
• You have a specialized domain (medical, legal) where generic LLM judges miscalibrate.
• You want reproducible evaluation: fixed classifier weights vs a constantly updating LLM.
• You need offline evaluation without outbound API calls.

When ARES Loses

• You have fewer than ~100 gold labels (PPI loses statistical power).
• You need quick iteration: RAGAS runs in minutes, ARES needs a training phase.
• You don't have GPU access.
• Your judge criteria change weekly — re-fine-tuning is heavier than prompt edits.

Gold Label Collection

ARES expects rows:

query	context	answer	Context_Relevance_Label	Answer_Faithfulness_Label	Answer_Relevance_Label
"..."	"..."	"..."	1	1	0

Label conventions:

• Context Relevance: 1 if context contains info to answer, else 0.
• Answer Faithfulness: 1 if every claim in answer is in context, else 0.
• Answer Relevance: 1 if answer addresses query, else 0.

Get three annotators per item; use majority vote. Report Cohen's kappa > 0.6 before trusting labels.

CI Integration Sketch

# nightly_ares.py
results = ares.evaluate_RAG(**ppi_cfg)
for dim, vals in results.items():
    if vals["ci_low"] < BASELINES[dim] - 0.02:
        print(f"REGRESSION: {dim} dropped to {vals['score']:.3f} (CI {vals['ci_low']:.3f}-{vals['ci_high']:.3f})")
        raise SystemExit(1)

Also see continuous-evaluation skill for full CI patterns.

Refreshing the Judge

Re-train when:

• Domain vocabulary shifts (new product area, new regulations).
• Synthetic data quality improves (better generator LLM available).
• Gold set grows by 2x or more.
• Classifier accuracy on gold drops below 85%.

Do not re-train for cosmetic query wording changes.

Anti-Patterns

Production Checklist

• Synthetic set of 5k-10k queries generated and manually spot-checked
• Gold set of 150-300 examples with 3 annotators and kappa > 0.6
• Three judge classifiers trained; test-set accuracy logged
• PPI evaluation script wired into nightly pipeline
• Confidence intervals reported alongside point scores
• Baseline scores versioned; regression alert < baseline - 2pp
• Classifier retrain schedule documented (quarterly + on-trigger)
• Documented comparison vs RAGAS before adopting in production