pxi-eval-dataset

Produce a small, well-targeted YAML dataset that drops into evals/pxi/datasets/<name>.yaml, runs through evals/pxi/harness/run_experiment.py, and is scored by deterministic code evaluators under evals/pxi/evaluators/.

The aim is a minimal but representative set of synthetic examples — think unit tests, not a benchmark. 10–50 examples, each covering a distinct dimension. Add more only when a new example tests something no existing example does.

Every example must be scorable by deterministic / heuristic / code logic. Every example must include a non-empty splits: list. Use splits: [regression] for small committed regression suites unless the user explicitly asks for a dev or val dataset; keep regression, dev, and val disjoint.

Workflow

1. Identify the target

Confirm with the user what's under test: a specific PXI tool name (e.g. set_time_range), a skill, or a higher-level behavior. For tools:

• Read the tool definition at src/phoenix/server/agents/toolsets/external/tools/<name>.py — the ToolDefinition's description and parameters_json_schema are what the LLM actually sees.
• Search the phoenix src for any server-side implementation function that backs the tool. (External tools execute browser-side and may have none — that's fine, the docstring + schema are the spec.)
• Read src/phoenix/server/agents/toolsets/__init__.py::build_toolset and src/phoenix/server/agents/toolsets/external/__init__.py::build_external_toolset to learn the availability conditions — which ChatContext must be present for the tool to be exposed (e.g. set_spans_filter only exists when deps.contexts.project is set).

2. Survey the evaluators that currently exist

List evals/pxi/evaluators/ and read each module. For every evaluator, note:

• the evaluator name and @create_evaluator(...) decorator,
• what fields it consumes from expected (e.g. expected.tools.required, expected.tool_call_args[<tool>]),
• its score / label semantics and any helpful failure metadata,
• the class of assertion it supports (tool selection, tool arguments, assistant text, multi-call sequencing, ...).

Also peek at evals/pxi/tests/test_evaluators.py for canonical input shapes.

Do NOT hard-code knowledge of which evaluators exist — re-read the directory every time. The set will grow.

3. Match the target to the evaluators — pause if a gap exists

Decide whether the behavior under test can be scored by what's there today.

• Yes → continue. The chosen evaluator names go in the dataset's evaluators: field (required, top-level). The runner uses ONLY the listed evaluators; unrelated ones are not invoked, so the dashboard stays free of vacuous passes from evaluators that don't apply to this dataset. Available names live in evals/pxi/evaluators/__init__.py (EVALUATORS_BY_NAME).

• No → stop and summarize the gap to the user. Propose the shape of a new evaluator:

  • name (snake_case),
  • the expected.<field> it would read,
  • score / label semantics,
  • whether it's tool-call-shaped, text-shaped, structural, etc.

  Then ask: "Should I add this evaluator before we generate examples?" If yes:

  • implement it under evals/pxi/evaluators/<file>.py,
  • add unit-test coverage to evals/pxi/tests/test_evaluators.py,
  • export it from evals/pxi/evaluators/__init__.py,
  • then continue.

  If no, scope the dataset down to assertions the existing evaluators can score, and tell the user what coverage that costs.

4. Enumerate coverage dimensions

Walk this checklist for the target. For each row, write down which queries you'll add to cover it. Skip rows that don't apply (e.g. booleans on a tool with no boolean field).

• Parameter coverage — every required field, every optional field, every meaningful field combination.
• Value coverage — every enum literal; for strings: empty, whitespace, special characters, very long; for numbers: zero, negative, boundary; for booleans: both polarities.
• Combination coverage — fields that interact (e.g. a filter condition paired with a scope toggle, two args that must agree).
• Negative coverage — queries where the tool should NOT be called.
• Ambiguity coverage — borderline queries that test correctness under uncertainty.

Polarity and difficulty targets are stated once in step 5 below; aim for those across the whole dataset, not within each row.

If you cannot fill at least 10 rows, the target is probably too narrow — confirm scope with the user.

5. Draft queries

For each coverage dimension, write one or more queries that feel like they came from a real Phoenix user. Mix across:

• Voice: imperative ("show me LLM spans"), declarative ("I want to see only errors"), question ("what spans took over 5s?"), fragment ("LLM only").
• Polish: clean prose, terse fragments, casual typos, abbreviations, incomplete sentences.
• Personas: new user setting up Phoenix on a new project; engineer debugging an agent or RAG pipeline; PM exploring trace quality; AI engineer writing evals; annotator marking traces; researcher exploring trends. Sample across them — don't sound like one author.
• Difficulty: ~30% obvious, ~50% moderate, ~20% ambiguous / tricky.
• Polarity: at least 30% negative (tool should NOT be called).

Anti-patterns to avoid:

• Don't paraphrase the tool's docstring.
• Don't use technical jargon a real user wouldn't reach for.
• Don't write 20 minor variations of the same intent.

6. Annotate expected outputs — subprocess annotation

Ground truth must be generated in a fresh subprocess, not inline. When annotation happens in the same context that drafted the queries, the agent builds a prior from its own examples: it anchors on condition patterns it established early, collapses ambiguous cases into overconfident annotations, and fills in tool_call_args based on what earlier examples "look like" rather than what the tool spec actually says. This is context rot — annotation signal weakens as the dataset grows.

The remedy: annotate each example in a separate subprocess whose context contains the full PXI toolset, the author's design intent for the query, and the schema contract. Each annotation is an independent reasoning trace from the toolset spec.

Prepare once, reuse across examples. Before launching annotation subprocesses, extract the full toolset spec once (see section 1 below) and cache it. Every annotation subprocess across the dataset reuses the same toolset block; only the query and design-intent block change per-example.

Subprocess context (four sections, in this order)

1. Full toolset spec — the complete toolset the PXI agent sees at runtime, NOT just the tool under test. For each tool, include the name, description, and parameters_json_schema. The annotator has to decide whether the right answer is the focal tool, a different tool, or no tool at all — and cannot do that without seeing every tool's spec. This is the single biggest hedge against annotator hallucination: with only one tool visible, the annotator will force-fit it to ambiguous or negative queries.

   Source the toolset from build_toolset and build_external_toolset in src/phoenix/server/agents/toolsets/ (and src/phoenix/server/agents/toolsets/external/). The eval harness configures ChatContext such that all external tools are available — see evals/pxi/harness/agent_task.py. List every tool that survives that filtering, not just the focal one.

   Mark the focal tool explicitly at the top of this section (★ FOCAL TOOL: <name> — this dataset evaluates this tool's behavior). The annotator's job is still to reason from the spec for whichever tool best fits the query, but the highlight tells them which tool's evaluator family will judge the answer.

2. Evaluator contract — provide the dataset-specific expected-block schema derived from the selected evaluators (what each expected field means, matching semantics, and which fields may be omitted or expressed as variants). Add the explicit instruction: if you are unsure about a value, omit the key rather than guessing.

3. Query and design intent — the input.query string, plus the dataset author's framing if assigned:

   • difficulty: obvious | moderate | tricky
   • polarity: positive | negative
   • category: <coverage dimension> (the taxonomy from step 4 — parameter, value, combination, negative, ambiguity, plus any finer-grained category like span_kind, status, latency, negative_chitchat, negative_wrong_tool, etc.)
   • Any notes: the author wrote

   Frame this for the annotator as the dataset author's intent, NOT ground truth. The annotator should reason from the toolset spec and may disagree with the author — that disagreement is one of the most valuable signals this protocol produces, because it surfaces mis-classified queries.

4. Task instruction (goal-oriented, adapt only the schema reference):

   "Your goal: determine the correct expected: block for this dataset example — the expected output needed to validate whether the PXI agent took the right actions in response to the user's query, given the full toolset spec, evaluation goal, evaluator set, and expected-block schema.

   Constraints:

   • Follow the expected-block schema provided for this dataset. The schema is determined by the evaluation goal and selected evaluators; do not assume every dataset validates only tool selection or tool arguments.
   • When the schema includes tool expectations, consider every tool in the toolset spec, not just the focal one. The right answer may be the focal tool, a different tool, or no tool at all.
   • Use only values directly implied by the relevant spec and the query. Omit any key whose value the spec doesn't determine — don't invent a plausible default.
   • The author's design intent (difficulty, polarity, category, notes) is a hint, not ground truth. If your reading of the spec contradicts the author's labeled polarity, output what the spec implies and record the disagreement in metadata.notes.
   • Return exactly one best expected: block. If multiple outputs seem valid, choose the single strongest answer and mention the alternatives in your brief reasoning; the orchestrator decides later whether variants belong in the final example.

   Output format (exactly two parts, in this order):

   1. Brief reasoning — one or two sentences explaining which expected output you concluded should be used and why. Just enough that an orchestrator comparing three independent annotations can see where you agree or diverge with the others.
   2. The expected: block as YAML matching the schema reference provided in this prompt. No other prose."

   This frames the task as a goal with constraints, not a procedural recipe. Cross-model fan-out is only useful if each annotator reasons independently — prescribing the exact reasoning steps collapses that diversity. The brief-reasoning requirement gives the orchestrator enough signal to adjudicate disagreements without dictating how each annotator gets there.

Expected-block schema and annotation rules

Before launching annotation subprocesses, define the expected-block schema for this dataset from the evaluation goal and selected evaluators. Include that schema verbatim in every annotation prompt. For example, a dataset that validates tool selection and tool arguments may use expected.tools and expected.tool_call_args, while another dataset may validate response text, refusal behavior, or some other evaluator-specific field.

When the schema includes tool expectations, use these conventions:

• Use tools.required for the must-be-called tool(s).
• Use tools.exact_match: true for strict-no-extras cases.
• For a pure-negative case where no tool should be called, set tools.required: [] and tools.exact_match: true. Do not enumerate every tool in tools.forbidden; the tool list can grow and make that brittle. Do not supply tool_call_args.
• Use tool_call_args[<tool>] only when argument correctness matters; leave it off for "any args are fine" cases — especially ambiguous cases where you want to assert the tool fires but not pin its args.

How many opinions, and which models

Collect three independent opinions for every dataset example: Sonnet, Opus, and Codex. Even obvious and pure-negative examples get the full fan-out so the dataset has a consistent annotation provenance and easy-to-audit agreement metadata.

Cross-model diversity — different families, different training, and (for Codex) different agent runtime — catches single-model biases that seed-level diversity misses.

How each opinion is invoked. The Claude Code Agent tool is Anthropic-only, so the OpenAI-side opinion comes from Codex via a Bash wrapper. Using Codex (rather than a one-shot Chat Completions call) gives the OpenAI annotator the same kind of environment access a Claude subagent has — sandbox-constrained file reads, repo navigation, the ability to double-check its reading of the toolset against the actual source. This keeps the three opinions genuinely symmetric.

The Codex wrapper at .agents/skills/pxi-eval-dataset/scripts/annotate_via_codex.sh reads a prompt on stdin and writes Codex's final message to stdout. It runs codex exec with --sandbox read-only (the annotator can read the repo but cannot modify anything), --ephemeral (no session persistence between invocations), and --skip-git-repo-check. Authentication is whatever Codex is already configured with (codex login status); no OPENAI_API_KEY plumbing required. Example invocation:

cat /tmp/annotation_prompt.txt | .agents/skills/pxi-eval-dataset/scripts/annotate_via_codex.sh
# Or pin a specific model:
cat /tmp/annotation_prompt.txt | .agents/skills/pxi-eval-dataset/scripts/annotate_via_codex.sh --model o3

Run all three opinions in parallel. Send a single message containing two Agent tool calls (Sonnet, Opus) and one Bash tool call (the Codex wrapper) — they will run concurrently. Do not share context between them.

Orchestrator subprocess (required)

After the three annotation subprocesses return, run a fourth fresh subprocess as the orchestrator. Its job is to merge the candidates into a single expected: block, with authority to drop spec-invalid proposals. Use a separate subprocess so the orchestrator sees anonymized candidate blocks and cannot be biased by knowing which model produced which annotation.

Orchestrator subprocess context:

• The full toolset spec (same as the annotation subprocess — focal tool marked, all other available tools included). The orchestrator needs this to judge "spec-valid" claims, especially for negative cases where the right answer is a non-focal tool.
• The query and design intent block (same as the annotation subprocess — difficulty, polarity, category, notes). The orchestrator should also be willing to override the author's polarity if the toolset spec and the majority of annotation opinions disagree with it; flag any such override with metadata.notes: 'orchestrator overrode author polarity'.
• All three candidate expected: blocks (anonymized — do not tell the orchestrator which model produced which proposal; this prevents reputation bias)
• The dataset-specific expected-block schema

Orchestrator decision rules:

• Merge portions of the expected: block where the models agree.
• Drop any value that is invalid under the tool spec, evaluator contract, or dataset-specific expected-block schema. Justify each rejection against a specific clause.
• Where models disagree but multiple outputs are valid and the schema supports variants, include variants only for the specific fields where variants are meaningful.
• Where variants do not make sense for the field, use the majority answer.
• Where there is no valid majority and the schema allows omission, omit the disputed optional field rather than over-constraining the expected output.
• For tool-call argument variants, keep at most three variants. If there are many valid argument shapes, prefer omitting tool_call_args[<tool>] so the evaluator checks tool selection without pinning arbitrary arguments.

The orchestrator must justify any "spec-invalid" rejection against a specific clause of the tool spec. It does NOT vote: it adjudicates against the spec.

Confidence metadata — surface noisy annotations

After orchestration, write the annotation provenance into the example's metadata. This makes noisy / low-confidence ground truth easy to audit later.

metadata:
  category: preset
  annotation:
    agreement: high     # high | medium | low — see rule below
    n_opinions: 3       # number of annotation subprocesses run

Agreement categorization rule:

• high — all three opinions produced equivalent expected: blocks (equal tools.{required,forbidden} lists after sorting, and per-tool tool_call_args dicts equal after applying _normalize_arg_value from evals/pxi/evaluators/tools.py to each value).
• medium — majority agreed; minority differed but the orchestrator was able to merge (as variants) or drop (as spec-invalid).
• low — no majority; the orchestrator either fell back to omit-args or made a judgment call with weak signal.

Why this matters: an agreement: low example may still have a correct ground truth, but it's a candidate for human review — the models hedged, which can indicate either a genuinely ambiguous query or a spec the LLMs interpreted differently. A follow-up audit script can flag these for inspection.

Equality-detection caveat: the heuristic catches DSL clause reordering (because it reuses _normalize_arg_value) but does NOT catch semantically-equivalent rewrites (e.g. latency_ms >= 5000 vs latency_ms > 4999). Such cases will surface as agreement: medium or low, which is arguably the right signal — the model hedged.

Multiple-correct arg shapes: variant lists

When more than one set of arguments is reasonable per the spec, write tool_call_args[<tool>] as a list of dicts instead of a single dict. The observed call passes if any variant matches under the same subset rules as the single-dict form. This lets each variant pin a different combination of keys (e.g. a preset variant vs. a custom variant with startTime):

tool_call_args:
  set_time_range:
    - { timeRangeKey: 1d }                  # preset variant
    - { timeRangeKey: 7d }                  # preset variant
    - { timeRangeKey: custom }              # custom range, key omitted

Use variants for genuinely-ambiguous queries (e.g. "show me recent traces" when only a small number of choices are defensible). Keep the list to at most three variants. If there are many valid choices, omit the argument assertion instead. Do not use variants to paper over agent inconsistency on queries that have one clear correct answer.

Applying orchestrator results back to an existing dataset

When the orchestrator returns adjudicated expected: blocks (or new metadata.annotation blocks during a backfill), prefer targeted Edit operations on the specific examples that change. Do NOT re-serialize the whole YAML file with pyyaml or ruamel.yaml:

• pyyaml.safe_dump strips comments and rewrites indentation (e.g. sequence-indent=2 and unquoted list-item style) — a 38-example file comes back with a 1000+ line diff that obscures the real semantic change.
• ruamel.yaml can round-trip, but only when you pin both preserve_quotes=True AND the exact indent(mapping=M, sequence=S, offset=O) settings of the source file. Mismatched indent settings silently corrupt list-item structure (e.g. child fields land at the list-item column instead of inside the item). Recover requires a full revert.

The safe pattern for bulk merges:

1. Load the orchestrator's results into Python (any YAML library is fine for reading).
2. Build a {example_id: (new_expected, new_metadata)} map.
3. Iterate over each changed example and apply a focused Edit call targeting the example's existing block as old_string and the merged block as new_string. Each Edit's diff stays scoped to the one example.
4. Validate with load_dataset after the batch.

This preserves section-header comments by construction and produces a diff that reviews cleanly.

Section-header comments are convenience navigation only. The load-bearing per-example commentary lives in metadata.annotation.notes and metadata.triage.notes, which round-trip safely through any merge approach. If a bulk rewrite loses section comments, the dataset is still fully self-documenting — no need to reconstruct them.

7. Save and validate

Save to evals/pxi/datasets/<name>.yaml. Then:

# Parse + validate schema:
uv run python -c "from evals.pxi.harness.datasets import load_dataset; load_dataset('<name>')"

# Run the experiment end-to-end against the real PXI agent:
uv run python -m evals.pxi.harness.run_experiment --dataset <name>

Run the experiment end-to-end and triage every failure before calling the dataset done. Don't assume a failed example means a broken agent — eval suites have three plausible failure sources, and mixing them up is the easiest way to ship bad ground truth or chase phantom agent regressions.

Failure triage — three categories

For each failed example, classify the failure into one bucket:

Triage workflow

1. Pull the failed example's id, observed tool calls, and evaluator label from the experiment output (the runner prints these per evaluator).
2. Re-read the focal tool's parameters_json_schema and the relevant evaluator code to decide which category the failure belongs in. The categorization is not always obvious — when genuinely uncertain, prefer "harness/evaluator issue" and read the evaluator first, since a broken evaluator silently corrupts both of the other categories.
3. Record the triage decision per failed example before fixing anything. A short list — {example_id: category} — is enough. Without this list, the fix loop tends to oscillate (fix dataset → evaluator flips on a different example → "fix" evaluator → first example breaks again).
4. Fix in order: harness/evaluator issues first, dataset issues second, leave genuine agent issues for last (or escalate rather than fixing them yourself).
5. Re-run the full experiment after each batch of fixes. Stop when the only remaining failures are genuine agent issues.

Other things to look for during validation

• Examples that all pass trivially (probably duplicate coverage — delete or harden).
• Coverage gaps the agent surfaces (e.g. an enum value not represented in any example).
• Examples flagged annotation.agreement: low in metadata — these are exactly the candidates for human review even if they passed.

For local-only experimentation use the harness env vars (see evals/pxi/harness/README.md). There is no --limit flag — keep the dataset small while iterating, or commit a temporary copy.

YAML gotchas that cost an iteration if you miss them:

• Each example must have splits: [...]. For the normal small regression datasets this skill creates, set splits: [regression]. The harness rejects missing or empty split lists.
• The dataset validator uses ConfigDict(extra="forbid") on EvalDataset, so a typo in a top-level key (example instead of examples) raises rather than being silently ignored.
• Quote any scalar that begins with ", [, {, :, *, &, !, or %, and any value containing a leading/trailing colon or # — e.g. condition: "status_code == 'ERROR'". Plain DSL strings without punctuation can stay unquoted, but quoting consistently is cheaper than diagnosing a parser error.
• Example ids must be unique across the dataset; duplicate ids fail validation with the offending ids listed.

Dataset schema reference

Reference the live datasets in evals/pxi/datasets/ for current dataset shapes, examples, and naming conventions. Do not copy a schema from this skill into prompts. Instead, derive the expected-block schema from the selected evaluators and include that schema in each annotation and orchestration prompt.

Validator is in evals/pxi/harness/datasets.py. Matching semantics in evals/pxi/evaluators/tools.py:

• Subset match on tool_call_args: an observed call passes if it has every expected key with a matching value; extra keys are ignored.
• Variant match when tool_call_args[<tool>] is a list of dicts: the observed call passes if ANY variant matches. Use for genuinely ambiguous queries where multiple arg shapes are equally correct.
• Any-of match when a tool is called multiple times: ANY call can satisfy the expected args.
• Order-independent conjunctions: string values joined with and are normalized to a frozenset, so "span_kind == 'LLM' and latency_ms >= 5000" matches "latency_ms >= 5000 and span_kind == 'LLM'". Pure-or expressions are normalized the same way; mixed and/or fall back to exact string comparison.

Out of scope (today)

• Multi-turn evaluation — the harness invokes the agent once per example. If the behavior under test is conversational, raise that as a separate task before writing examples.