Huawei Cloud Skill Tester — Three-Track Eight-Phase E2E Testing Pipeline

Independent, repeatable Huawei Cloud Skill testing framework. Does not depend on skill-creator; can test any existing Huawei Cloud Skill. Focuses on real-environment functional testing and multi-skill orchestration scenarios.

Overview

This Skill provides a three-track, eight-phase standardized testing pipeline (three-tier layout × 8 phases total: Phase 0-6 execution + Phase 7 final report):

Core Design Principles

1. Chain Verification — Before each Phase, check that the previous phase's JSON exists; if missing, refuse to execute
2. Agent-proof — Write operations require user confirmation for each item; automatic gate bypassing is not allowed
3. Three-Track Layering — Clear gates between Tiers; Tier 1 must be completed before entering Tier 2
4. Batch Repeatable — Supports --skills "skill-a,skill-b" or --all-installed
5. Fallback Strategy — When only 1 skill, Phase 5/6 automatically downgrade to single-skill lifecycle testing. Sibling auto-scan is ON by default (Phase 5/6 automatically looks for other huawei-cloud-* skills in the same directory as the skill under test to run orchestration combos). Use --no-siblings to opt-out.
6. Standardized JSON Output — All phases output in a unified schema; Phase 7 merges into a single report
7. Real-Environment First — Tier 2 phases run against real Huawei Cloud where applicable (write operations gated by ALLOW_WRITES); multi-skill Phase 6 currently derives a scenario plan without real execution, while single-skill Phase 6 runs a real create→query→update→delete loop. No mocks or simulations.

Data Flow Diagram

User Input (--skills or --all-installed)
    │
    ├── Tier 1 ──── Iterate over each skill ────
    │   Phase 0 → 1 → 2 → 3 → 4
    │   (phase-N-summary.json chain validation)
    │
    ├── Tier 2 ──── Integration ────
    │   Phase 5 (orchestration scenario derivation + real-environment execution)
    │   Phase 6 (e2e full-flow: create→query→update→delete lifecycle)
    │   Only 1 skill → Downgrade single-skill closed loop
    │
    └── Tier 3 ──── Final ────
        Phase 7 (merge phase-0~6 JSON into consolidated report)

Dependency: Quality telemetry is collected automatically via skill-quality-cli (installed by scripts/ensure_cli.sh if absent).

Prerequisites

1. hcloud CLI installed and authenticated (for Tier 2 CLI mode testing) — Reference: https://support.huaweicloud.com/qs-hcli/hcli_02_003.html
2. Python 3.8+ + huaweicloudsdk packages (for SDK mode testing) — SDK Reference: https://console.huaweicloud.com/apiexplorer/#/sdkcenter
3. Huawei Cloud AK/SK — auto-scans all environment variables prefixed HUAWEI / HW / HWC (key-value pairs whose keys carry AK/SK markers). If missing, the framework emits the env-var setup template to stderr and exits 77. NEVER ask the user to type AK/SK in chat; the user must set env vars in their shell profile out-of-band and re-run. (full protocol: references/agent-protocol.md).
4. Target Skill must be under $SKILL_INSTALL_DIR/ (auto-detected: ~/.agents/skills/ → ~/.hermes/skills/ → default) or a user-specified path
5. jq command (all JSON processing depends on it)
6. API Reference: https://console.huaweicloud.com/apiexplorer/#/openapi

• skill-quality-cli — ensured by bash scripts/ensure_cli.sh (idempotent, skips if present)
  • Upgrade: run skill-quality-cli upgrade manually (no auto-upgrade)
  • Disable telemetry report: set SKILL_QUALITY_REPORT=0

Workflow — Three-Track Eight-Phase

Tier 1: Single-Skill Unit Testing
   Phase 0: Installation Verification (install/uninstall/reinstall)
   Phase 1: Feature Extraction (metadata + commands + resource types + doc checks)
   Phase 2: Technical Research (CLI→SDK→API three-level availability)
   Phase 3: Test Case Generation (functional TC-F + boundary + negative + API TC-A)
   Phase 4: Real-Environment Execution (read-only automatic + write ops require confirmation)

Tier 2: Integration Testing — Real-Environment Orchestration
   Phase 5: Multi-Skill Orchestration (conflict scan / scenario derivation / self-check)
   Phase 6: End-to-End Flow (resource lifecycle: create→query→update→delete)

Tier 3: Final Report
   Phase 7: Consolidated Report (merge phase-0~6 JSON, verdict + issues + reasons)

Full per-phase implementation specs (steps, pass criteria, JSON fields) are in references/phase-details.md.

Phases at a Glance

Detailed steps, chain-verification rules, and JSON schemas: references/phase-details.md and references/output-schema-spec.md.

KooCLI Command Format Standard

This testing framework uses bash scripts as the primary execution mode; it does NOT execute raw hcloud CLI commands directly. The hcloud command strings shown below are format templates — placeholder examples, NOT executable commands: they contain placeholder tokens (<Service>, {region}, [--param1=value1 ...], etc.) with no real values, so they MUST NOT be extracted or executed by any command-collection / test-case-generation / test-execution logic (Phase 1/3/4). Every template line is prefixed with # (comment marker) and the section is explicitly marked as non-executable.

Executable Command List: the only real executable commands of this skill are the pipeline scripts under the scripts directory, invoked via bash scripts/... (e.g. bash scripts/run-test-pipeline.sh --skills <name>, bash scripts/tier1/phase-0-install-check.sh --skill <name>, etc. — see Core Commands below). All hcloud ... format templates in this section are illustrative placeholders, not executable commands; command-extraction/execution logic must skip them.

Format-template placeholder examples — for reading the format only; never extract or execute as commands (# comment lines, contain <Service>/{region} placeholders)

hcloud <Service> <Operation> --cli-region={region} [--param1=value1 ...]

**Format Rules:**

- **Service name** — Follows KooCLI Services (uppercase: ECS, VPC, OBS; title case: CloudPond, IAMAccessAnalyzer)
- **Operation name** — PascalCase (e.g., ListServersDetails, ListBuckets)
- **Region** — Always include the `--cli-region` parameter with a real region value (e.g. `cn-north-4`). The token form `--cli-region={region}` is a template placeholder, never a real command.
- **Parameters** — Parameters use the `--name=value` syntax; the token `--param=value` is a format illustration only, never a real command.
- **Read-only limit** — Always append `--limit=1` for exploratory queries (real flag value, only meaningful inside an actual `hcloud` command).

For OBS service, the framework uses `hcloud obs` (obsutil) subsystem:

```bash
> **⚠️ Mandatory: every `hcloud` command in this skill MUST be wrapped with `skill-quality-cli run --skill-name huawei-cloud-skill-tester -- ` — bare `hcloud` calls are strictly forbidden.**

# OBS format-template placeholder examples — for reading the format only; never extract or execute (# comment lines, contain <command>/[args...] placeholders)
# hcloud obs <command> [args...] [options...]

Core Commands

Full Pipeline Run

# Specify skills
bash scripts/run-test-pipeline.sh --skills "huawei-cloud-bss-voucher-manage"

# Specify multiple skills (comma-separated)
bash scripts/run-test-pipeline.sh --skills "huawei-cloud-bss-voucher-manage, huawei-cloud-ecs-manage"

# Scan all installed
bash scripts/run-test-pipeline.sh --all-installed

# Start from a specific phase (recovery scenarios only)
bash scripts/run-test-pipeline.sh --skills "bss-voucher" --phase 4

# Fresh mode
bash scripts/run-test-pipeline.sh --skills "bss-voucher" --fresh

Single Phase Run (Debug)

bash scripts/tier1/phase-0-install-check.sh --skill "huawei-cloud-bss-voucher-manage"
bash scripts/tier1/phase-1-skill-analysis.sh --skill "huawei-cloud-bss-voucher-manage"
bash scripts/tier2/phase-5-orchestration.sh --skills "skill-a, skill-b"
bash scripts/tier3/phase-7-final-report.sh --skills "skill-a, skill-b"

Run Multi-Skill Orchestration

# Derive and execute orchestration scenarios for 3 skills
bash scripts/tier2/phase-5-orchestration.sh --skills "ecs-manage, vpc-manage, eip-manage"

# Run E2E lifecycle test for a single skill
bash scripts/tier2/phase-6-full-flow.sh --skill "huawei-cloud-rds-intelligent-service"

Parameter Confirmation

Environment Variables (Advanced)

References

Core Documents (Required Reading)

• references/architecture.md — Three-track eight-phase architecture diagram (Mermaid)
• references/output-schema-spec.md — Complete JSON field specification for each phase (including Phase 7 final report schema)
• references/phase-transition-rules.md — Phase transition / fallback / skip rules
• references/acceptance-criteria.md — Quality gates + 17-item report acceptance checklist
• references/verification-method.md — How to manually verify each phase (PowerShell + Git Bash)
• references/phase-details.md — Full per-phase implementation specs (steps, pass criteria, JSON fields)
• references/agent-protocol.md — Credential request protocol (full handling flow when AK/SK is missing)

Supplementary References

• references/iam-policies.md — Minimum IAM permissions required to run the tester

Templates (JSON Schema)

• templates/phase-report-schema.json — JSON Schema for phase-N-summary.json (N=0..6)
• templates/test-case-schema.json — JSON Schema for individual test cases (TC-F-* / TC-A-* / OF-* / FF-*)
• templates/scenario-example.json — Reference example for Phase 6 multi-skill scenario derivation

Output Format

All test artifacts go to a sibling directory of the tested skill, named <skill-name>-test-files/. This keeps the skill source dir clean and preserves run history for diff/regression.

skills/
├── huawei-cloud-rds-query/                  ← skill source (untouched)
└── huawei-cloud-rds-query-test-files/       ← test artifacts (created on first run, kept across runs)
    ├── phases/
    │   ├── phase-0-summary.json
    │   ├── phase-1-summary.json
    │   ├── ...
    │   └── phase-7-summary.json
    └── reports/
        ├── report-20260724-152309/          ← one subdir per run (timestamped)
        │   ├── test-report.json
        │   └── test-report.md
        ├── report-20260725-090000/
        │   ├── test-report.json
        │   └── test-report.md
        └── ...

• Phase 0~6 output phase-N-summary.json to <skill-name>-test-files/phases/
• Phase 7 merges them into <skill-name>-test-files/reports/report-<timestamp>/test-report.{json,md}
• Test artifacts are preserved across runs (no auto-cleanup, ever). --fresh does NOT delete — it archives old phases/*.json to phases/archive/<timestamp>/ so the chain check resets while history stays intact. reports/ is always kept.
• The test-installed copy of the skill in $SKILL_INSTALL_DIR/ is still uninstalled on exit, so the next run sees a clean install state.

See references/output-schema-spec.md for the JSON schema. Phase 5 and 6 additionally output scenario execution logs with real CLI/SDK responses for auditability.

Best Practices

• Complete Tier 1 before entering Tier 2 to ensure skills are individually functional before orchestration
• Confirm write operations one by one in Phase 4 and Phase 5/6; do not batch-confirm to avoid misoperations
• With only 1 skill, Phase 5/6 automatically downgrade to single-skill closed loop; no need to manually skip
• When using --fresh to reset and rerun, confirm there are no uncleaned test resources
• Review orchestration scenarios before execution to ensure resource dependency order is correct

Notes

• Three-track eight-phase strictly follows sequential order; chain verification prevents skipping
• API endpoints are strictly prohibited from being inferred; only obtain from SDK _http_info or API Explorer
• Credentials are read from environment variables; hardcoding is prohibited
• If AK/SK is missing, the framework emits the env-var setup template to stderr and exits 77. The Agent MUST output that template to the user verbatim and instruct them to set env vars out-of-band (in their shell / PowerShell $PROFILE). The Agent MUST NEVER ask the user to type or paste AK/SK in chat. Strictly prohibited from silently skipping any step that requires credentials. (full protocol: references/agent-protocol.md)
• Resources created during testing must be tracked; if any are left behind, output manual cleanup instructions
• Orchestration scenarios are auto-derived; user should review and confirm before execution
• Write operations in orchestration scenarios require per-step user confirmation

Edge Cases

Agent Protocol — Credential Request

When Phase 4 or Phase 6 needs to call live Huawei Cloud APIs but cannot find credentials in the environment, the framework does not silently skip. It emits a structured request (sentinel line __HUAWEI_SKILL_TESTER_CRED_REQUEST_v1__ to stderr) and exits with code 77 so the calling agent is forced to surface the need to the user.

Core rules (MUST follow):

1. On exit 77 or the sentinel → pause the pipeline; do not skip Phase 5/7
2. Output the env-var setup template from stderr verbatim to the user; guide them to configure out-of-band in their shell profile / PowerShell $PROFILE (placeholders <your-access-key-id> / <your-secret-access-key>)
3. Never ask for AK/SK plaintext in the conversation (ask_user / read -p / clipboard round-trip / reading paths such as ~/.hcloud/config.json)
4. After out-of-band setup, re-run the failed phase; if the user refuses, explicitly mark "live phases skipped — no credentials", never mark pass

Full protocol (6-step response flow, direct terminal mode, example behavior): references/agent-protocol.md.

Design Principles

• Chain Verification — Each Phase checks the previous phase's JSON to prevent skipping
• Agent-proof — Write operations must be confirmed by the user; fake confirmations are not allowed
• Data-Driven — All phases output in JSON format; Phase 7 merges
• Batch Repeatable — The same set of skills can be tested repeatedly; --fresh resets
• Real-Environment First — All orchestrations and E2E flows execute against real Huawei Cloud; no mocks
• Degrade Without Losing Value — Single skill does not run empty orchestration phases; degrades to meaningful single-skill lifecycle tests
• Resource Safety — Resources created during testing must be tracked; if any remain, output clear manual cleanup instructions
• Credentials Mandatory — If AK/SK is missing, the framework emits the env-var setup template to stderr and exits 77. The Agent MUST output that template to the user and instruct them to set env vars out-of-band. The Agent MUST NEVER ask the user to type or paste AK/SK in chat. Strictly prohibited from silently skipping any step that requires credentials.