Add a Runner Eval
Use this skill for the Runner Evals family: a real Runner/provider session against a seeded mock control plane. Product browser/server/database/Daytona coverage belongs in add-product-e2e-eval.
Locate the Paperclip checkout using PAPERCLIP_ROOT when supplied, or
git rev-parse --show-toplevel from a checkout. From outside Git, inspect the
workspace roots (for example ~/paperclipai/paperclip) and verify that the
selected root contains packages/paperclip-runner and tests/runner-e2e.
Locate paperclip-evals using PAPERCLIP_EVALS_ROOT or a discovered sibling;
a worktree's parent directory need not contain that repository. Read
doc/evals.md and packages/paperclip-runner/docs/runner-protocol-live-evals.md,
then inspect the nearest existing case, roster, schema, and report test before
editing. Definitions and authored cases belong in the sibling
paperclip-evals/evals/paperclip-runner; Runner integration, aggregation,
viewer, and publication behavior belongs in packages/paperclip-runner.
Keep the control-plane boundary explicit in names and documentation.
The sibling eval README is the concrete map: cases live under cases/,
company fixtures under fixtures/, runtime/model settings under configs/,
selections under rosters/, and maintained campaign membership under
campaigns/live-direct-full.json. Update inventory/coverage mappings when the
program requires them; a new file alone does not join the maintained campaign.
From the Evals repository root, adapt these provider-free checks to the case
and roster you changed. Run the reliability-plan validator only when that
separate plan changes:
python3 evals/paperclip-runner/tools/eval_program.py validate \
--case evals/paperclip-runner/cases/get-task-context.json \
--config evals/paperclip-runner/configs/live-codex-pinned.json
python3 evals/paperclip-runner/tools/run_live_roster.py validate \
--roster evals/paperclip-runner/rosters/live-mini.json --run-id validate-new-case
python3 evals/paperclip-runner/tools/run_live_campaign.py validate \
--campaign evals/paperclip-runner/campaigns/live-direct-full.json
python3 evals/paperclip-runner/tools/reliability_campaign.py validate \
--plan evals/paperclip-runner/campaigns/paperclip-runner-reliability.json
Use nearby positive and negative grader cases/fixtures to calibrate the new assertion, including malformed or missing evidence where the grader must fail closed. Preserve the existing machine disposition and grade; product, model/provider, grading, and infrastructure labels are analytical annotations, not instructions to rewrite classifiers.
Author one bounded case with a deterministic semantic assertion and an inspectable result. Declare its expected operation, state effect, provider lane/profile, timeout and retry policy, and any required evidence. Do not grade hidden reasoning, infer success from a provider terminal message, or invent conversation/tool evidence. Public output follows the reviewed projection: sanitized fixture conversation and allowlisted tool outcomes may be published; raw trusted artifacts, credentials, secrets, private references, and hidden reasoning may not.
Validate without provider calls first using the commands above and the relevant
report/render validation documented in the Runner docs. When a live run is
authorized, pin the Paperclip commit and exact 40-character
paperclip-evals commit, select the smallest useful roster, and retain the
complete provenance and cost record.
Update authoritative detailed docs when the contract or command changes, then
link from doc/evals.md rather than duplicating the Runner runbook. Keep public
reports immutable and use the reviewed projection; sanitized fixture
conversation and allowlisted tool outcomes may be public, while credentials,
secrets, private references, raw trusted payloads, and hidden reasoning must not
be exposed.