Agent Benchmarking

Overview

Vibes don't survive production. This skill picks the right standard benchmark for the agent's domain, builds a custom regression pack from real failures, and wires both into CI so quality is tracked, not assumed.

When to use

Choosing or upgrading a base model
Comparing your agent vs vendor agent
Catching capability regressions across releases
Defending agent quality to stakeholders with numbers

Standard benchmarks (2026)

Domain	Benchmark	Measures
Coding	SWE-bench Verified	Real GitHub issue resolution
Coding (long)	SWE-bench Live	Continuously refreshed issues
Web browsing	WebArena	Multi-step web tasks
Desktop / OS	OSWorld	Real desktop apps, multi-modal
General assistant	GAIA	Tool use, reasoning, multi-modal
Tool use	TauBench	Multi-turn customer-service simulation
Reasoning	HLE (Humanity's Last Exam)	Hard graduate-level reasoning
Long horizon	AgentBench	Composite, multiple env

Pick by domain, not by leaderboard popularity.

Custom regression pack

Standard benchmarks miss your failure modes. Build a regression pack from:

Production bug reports (anonymized)
Postmortems → "agent should have done X"
Red-team finds (see agent-guardrails)
Edge cases discovered during eval

Format: same as your standard benchmark (one task → one expected outcome) so the harness is shared.

Layout

benchmarks/
  swe_verified/        # vendored subset for CI speed
  custom_regression/
    cases/
      <id>/task.yaml
      <id>/expected.json
  runners/
    swe_runner.py
    custom_runner.py
  results/
    <ts>/scores.json

CI integration

# Run on every PR
- name: regression-pack
  run: pytest benchmarks/custom_regression/  # fast, ~5min
  required: true

# Run nightly
- name: swe-bench-verified-50
  run: python benchmarks/runners/swe_runner.py --subset 50
  threshold: pass_rate >= 0.55  # block release if below

# Run weekly
- name: full-swe-bench-verified
  run: python benchmarks/runners/swe_runner.py

Scoring rules

Pass rate: primary metric for binary tasks
Partial credit: only when benchmark defines it (don't invent)
Cost per task: secondary, track alongside pass rate
p95 latency: SLO-relevant
Report all three — pass rate alone hides cost/latency regressions

Vendor comparison protocol

When comparing agents/models:

Same prompts, same tools, same harness
Same retry budget, same timeout
Disclose any prompt tuning per vendor
N runs ≥ 3 with mean ± stddev (LLMs are noisy)
Statistical test (e.g., paired bootstrap) before claiming "better"

Anti-patterns

Benchmark cherry-picking — pick the one your agent wins → never trustworthy
Train-on-test contamination — verify SWE-bench Verified, not full SWE-bench
"We beat GPT-4 on our internal eval" — internal eval is regression pack, not benchmark
Ignoring cost / latency — high pass rate at 10× cost is not progress

Quick start

pip install swebench webarena
python -m swebench.harness.run_evaluation \
  --predictions_path my_agent_preds.jsonl \
  --max_workers 4 \
  --run_id my_agent_v0.4

agent-benchmarking