Senior Prompt Engineer

Prompt engineering patterns, LLM evaluation frameworks, and agentic system design. Provides static (deterministic) analysis tools to optimize prompts, evaluate RAG retrieval and generation quality, and validate/visualize agent workflows — plus deep reference libraries of prompt patterns, evaluation metrics, and agent architectures.

Core Capabilities

• Prompt optimization — token counting and cost estimation, clarity/structure scoring, ambiguity and redundancy detection, and generation of optimized prompt versions.
• Few-shot & structured output design — extract/manage few-shot examples, design diverse example sets (simple/edge/complex/negative), and enforce reliable JSON/XML schema outputs.
• RAG evaluation — context relevance, answer faithfulness, groundedness (ROUGE-L), and retrieval metrics (Precision@K, MRR, NDCG) over pre-retrieved contexts.
• Agentic system design — validate agent configs, visualize flows (ASCII/Mermaid), estimate token cost per run, and apply ReAct / Plan-Execute / Tool-Use / multi-agent patterns.
• Pattern library — 10 prompt patterns, evaluation frameworks (A/B testing, benchmarks, human eval), and agent architectures with pseudocode.

When to Use

• Optimizing an existing prompt's performance or reducing token costs.
• Designing prompt templates, few-shot examples, or structured-output workflows.
• Evaluating LLM outputs or RAG retrieval/generation quality.
• Building or validating agentic systems and tool-calling workflows.

Tools

References

Load the reference that matches the task — keep this file lean and pull detail on demand:

• references/tools-and-workflows.md — full tool usage with sample outputs, the prompt-optimization / few-shot / structured-output workflows, common-patterns and command quick references, troubleshooting table, success criteria, and complete per-script parameter/output-format reference. Read when running any tool or executing a workflow.
• references/prompt_engineering_patterns.md — 10 prompt patterns (zero/few-shot, CoT, role, structured output, self-consistency, ReAct, tree-of-thoughts, RAG) with example inputs and expected outputs. Read when choosing or applying a prompt technique.
• references/llm_evaluation_frameworks.md — evaluation metrics, text-generation and RAG-specific scoring, human-eval frameworks, A/B testing, benchmark datasets, and pipeline design. Read when measuring quality or comparing prompts.
• references/agentic_system_design.md — agent architectures (ReAct, Plan-and-Execute, Tool Use, multi-agent, memory/state) and design patterns with pseudocode. Read when building agents or tool-calling systems.

Scope & Limitations

This skill covers:

• Static prompt analysis: token counting, clarity scoring, structure detection, and optimization suggestions
• RAG evaluation: context relevance, answer faithfulness, groundedness, and retrieval metrics (Precision@K, ROUGE-L, MRR, NDCG)
• Agent workflow design: configuration validation, ASCII/Mermaid visualization, and token cost estimation
• Few-shot example extraction and management from existing prompts

This skill does NOT cover:

• Live LLM calls or runtime prompt testing --- all analysis is static/deterministic (see senior-ml-engineer for LLM integration)
• Vector database setup or embedding generation --- RAG evaluator scores pre-retrieved contexts only (see senior-data-engineer for pipeline orchestration)
• Fine-tuning, RLHF, or model training workflows (see senior-ml-engineer for model deployment)
• Production monitoring, A/B test execution, or real-time drift detection (see senior-data-scientist for experiment design)

Integration Points