LLM Evaluation Skill

Run LLM pipeline evaluation against gold standard datasets using oracle LLM-as-judge scoring. Measures output quality across weighted dimensions, identifies weak steps, and suggests prompt improvements.

Quick Start

# Full evaluation (all test cases, all steps)
/sc:evaluate

# Quick spot check
/sc:evaluate --cases=case_1,case_2 --steps=1,2,3

# Re-evaluate existing results without re-running pipeline
/sc:evaluate --skip-pipeline

# Generate outputs only (no evaluation)
/sc:evaluate --skip-eval

# Specify judge model
/sc:evaluate --judge-model=gpt-4o

# Dry run to preview plan
/sc:evaluate --dry-run

Behavioral Flow

1. Discover - Find evaluation script, gold standards, and prompt files
2. Configure - Parse scope (cases, steps, model overrides)
3. Execute - Run pipeline on gold standard inputs
4. Evaluate - Score outputs against gold standards via LLM-as-judge
5. Analyze - Identify weak steps, dimension breakdowns, patterns
6. Recommend - Suggest specific prompt improvements for low-scoring steps
7. Report - Generate JSON + Markdown evaluation reports

Flags

Phase 1: Discover Project Structure

Locate evaluation components:

If no standard structure found, ask the user to specify paths.

Phase 2: Configure Scope

Parse arguments to determine:

• Which test cases to run (default: all discovered)
• Which pipeline steps to evaluate (default: all)
• Model overrides for pipeline and judge
• Output directory (default: timestamped)

Create output directory:

OUTPUT_DIR="${output:-eval_runs/$(date +%Y%m%d_%H%M%S)}"
mkdir -p "$OUTPUT_DIR"

Phase 3: Execute Pipeline

Run the pipeline on gold standard inputs:

python <eval_script> \
  --output "$OUTPUT_DIR" \
  --verbose \
  [--cases CASES] \
  [--steps STEPS] \
  [--model MODEL] \
  [--skip-pipeline] \
  [--skip-eval]

API call estimation:

• Pipeline: steps x cases API calls
• Evaluation: scored_dimensions x cases judge calls

For quick validation, suggest running on 1-2 cases with 2-3 steps first.

Phase 4: Evaluate with LLM-as-Judge

For each step output, compare against gold standard using oracle LLM-as-judge:

Evaluation dimensions (customizable per project):

Each dimension has a weight (0.0-1.0) summing to 1.0 per step.

Phase 5: Analyze Results

Read and analyze evaluation report:

1. Overall similarity score across all cases and steps
2. Per-step scores — highlight any below threshold (default: 70/100)
3. Per-case scores — identify consistently weak test cases
4. Dimension breakdowns for weak steps

Score interpretation:

Phase 6: Recommend Improvements

For each step scoring below threshold:

1. Read the current prompt template
2. Read the gold standard output (expected)
3. Read the pipeline output (actual)
4. Compare and identify gaps:
   • Missing instructions that gold standard captures
   • Overly broad instructions causing divergent output
   • Format/structure differences
   • Specificity gaps

Present actionable suggestions:

### Step N: <step_name> (Score: XX/100)

**Weakest Dimension**: <dimension> (XX/100)

**Gap Analysis**:
- Gold standard includes <X> but prompt doesn't instruct it
- Output format diverges: gold uses <format>, output uses <other>

**Suggested Prompt Changes**:
1. Add instruction: "<specific instruction>"
2. Clarify format: "<format guidance>"
3. Add example: "<example output snippet>"

Output Structure

eval_runs/YYYYMMDD_HHMMSS/
  results/                    # Pipeline outputs
    case_1/
      step_01_<name>.md
      step_02_<name>.md
      ...
    case_2/
      ...
  evaluation/                 # Judge scores
    evaluation_report.json
    evaluation_report.md
    per_step_scores.csv
    per_case_scores.csv

MCP Integration

PAL MCP (Optional)

Rube MCP (Optional)

Error Handling

Guardrails

• Always pass --verbose for progress visibility
• Warn about API call counts before full runs
• Suggest quick validation on subset before full evaluation
• Preserve all intermediate outputs for debugging
• Never modify gold standard files

Tool Coordination

• Bash - Run evaluation scripts
• Read - Inspect prompts, gold standards, outputs, reports
• Write - Generate reports
• Grep - Search for patterns in outputs
• PAL MCP - Deep analysis of score gaps