Huawei Cloud CCE Alarm Correlation Engine

Overview

Correlate Huawei Cloud AOM active and historical alarms for CCE operations, then convert raw alarm streams into prioritized investigation leads. This skill also manages AOM alarm rules and notification action rules through a strict preview-and-confirm workflow.

Architecture: python3 scripts/huawei-cloud.py dispatcher -> local hcloud (KooCLI) -> AOM/CCE/IAM cloud service operations -> alarm query, alarm correlation, alarm-rule management, notification-rule management, and CCE alarm health inspection.

Execution Method: All Huawei Cloud operations must go through the bundled Python dispatcher. The dispatcher invokes the local hcloud CLI. Direct SDK imports, hand-written API signing, curl IAM flows, openstack commands, or out-of-band cloud access paths are prohibited.

Related Skills:

huawei-cloud-cce-metric-analyzer - CCE, AOM, and cloud-resource metric checks
huawei-cloud-cce-kubernetes-event-analyzer - Kubernetes warning event analysis
huawei-cloud-cce-pod-failure-diagnoser - Pod failure diagnosis
huawei-cloud-cce-node-failure-diagnoser - Node failure diagnosis
huawei-cloud-cce-auto-remediation-runner - Remediation preview and execution

Capabilities:

Query active AOM alarms, historical AOM alarms, and merged active+historical views
Analyze alarm deduplication, severity grouping, burst alarms, attention alarms, and chronic alarms
Inspect CCE alarm health and produce risk items for follow-up diagnosis
Query AOM alarm rules with optional cluster_id filtering
Resolve the AOM Prometheus instance bound to a CCE cluster
Create metric alarm rules and event alarm rules
Create notification action rules from a user-provided SMN topic
Batch configure CCE recommended alarm rules from the cloud-side CCE alarm template
Batch clean CCE alarm rules matching the cloud-side CCE alarm template
Update, delete, enable, and disable AOM alarm rules
Query AOM action rules and mute rules

Typical Use Cases:

"List AOM alarms for this CCE cluster"
"Analyze active and historical alarms for alarm storms"
"Show current alarm rules for cluster <cluster_id>"
"Create CCE recommended alarm rules with this notification rule"
"Clean CCE template alarm rules for this cluster"
"Create a notification action rule using this SMN topic"
"Disable this noisy alarm rule after I confirm"

Prerequisites

1. Runtime Dependencies

Python 3.8+ for the dispatcher and result processing
hcloud (KooCLI) 7.2.2+ in PATH
A local hcloud profile configured with hcloud configure
Run environment checks before first use when available in the skill package

2. Credential Configuration

Valid Huawei Cloud credentials via explicit tool parameters, local hcloud profile, or environment variable fallback
Credential priority for hcloud calls is: explicit tool parameters > local hcloud profile > environment variables
Tools that need project_id resolve it internally where possible: explicit project_id parameter first, then active hcloud profile/IAM project lookup for the target region, then environment fallback

Security Rules:

Never expose AK/SK, tokens, or credential-derived secrets in code, commands, logs, or responses
Never run echo $HUAWEI_AK or echo $HUAWEI_SK
Never write credentials to files
Prefer hcloud profile for normal use
Use IAM users with least privilege

Optional Environment Fallback:

export HUAWEI_AK=<your-ak>
export HUAWEI_SK=<your-sk>
export HUAWEI_REGION=cn-north-4
export HUAWEI_PROJECT_ID=<project-id>
export HUAWEI_SECURITY_TOKEN=<security-token>

3. IAM Permission Requirements

Permission	Purpose
`aom:event:list`	Query active and historical AOM alarms
`aom:alarmRule:list`	Query AOM alarm rules
`aom:alarmRule:create`	Create AOM alarm rules
`aom:alarmRule:update`	Update, enable, or disable AOM alarm rules
`aom:alarmRule:delete`	Delete AOM alarm rules
`aom:actionRule:list`	Query AOM action/notification rules
`aom:actionRule:create`	Create AOM notification action rules
`aom:actionRule:delete`	Delete AOM notification action rules
`aom:muteRule:list`	Query AOM mute rules
`cce:cluster:get`	Resolve CCE cluster and AOM Prometheus binding

Permission Failure Handling:

Show the failed operation and required permission.
Ask the user to grant the missing IAM permission.
Pause mutation work until the user confirms permissions are ready.

Core Commands

All commands use the Python dispatcher script:

python3 scripts/huawei-cloud.py <action> <key=value>...

KooCLI命令格式标准

This skill does not ask users to run raw hcloud commands directly. All cloud operations must use the dispatcher command format:

python3 scripts/huawei-cloud.py <tool-name> key=value key=value

The dispatcher converts tool parameters into standard KooCLI calls:

hcloud <service> <operation> --cli-region=<region> --cli-output=json [--cli-jsonInput=<file>]

Follow these rules for every command:

Use key=value parameters; quote values that contain spaces, >, <, |, JSON, or PromQL.
Do not print or persist AK/SK, security tokens, or generated JSON input files.
Use dispatcher preview output first for R2/R1/R0 tools; add confirm=true only after explicit user confirmation.
Prefer hcloud profile credentials; explicit tool parameters override profile and environment fallback.
See CLI Installation Guide for setup and validation.

1. Alarm Query And Correlation

# Query active + historical alarms in a region
python3 scripts/huawei-cloud.py huawei_list_aom_alarms \
  region=cn-north-4

# Query active + historical alarms for a cluster
python3 scripts/huawei-cloud.py huawei_list_aom_alarms \
  region=cn-north-4 cluster_id=<cluster-id>

# Query current active alarms only
python3 scripts/huawei-cloud.py huawei_list_aom_current_alarms \
  region=cn-north-4 cluster_id=<cluster-id>

# Analyze deduplicated, burst, attention, and chronic alarm groups
python3 scripts/huawei-cloud.py huawei_analyze_aom_alarms \
  region=cn-north-4 cluster_id=<cluster-id>

Do not conclude "no issue" from absence of active alarms alone. Always consider historical alarms when diagnosing a recent or recovered problem.

2. Alarm Rule Query

# Query all alarm rules in a region
python3 scripts/huawei-cloud.py huawei_list_aom_alarm_rules \
  region=cn-north-4

# Query alarm rules related to one CCE cluster
python3 scripts/huawei-cloud.py huawei_list_aom_alarm_rules \
  region=cn-north-4 cluster_id=<cluster-id>

# Resolve the cluster AOM Prometheus instance
python3 scripts/huawei-cloud.py huawei_resolve_cce_aom_prom_instance \
  region=cn-north-4 cluster_id=<cluster-id>

huawei_list_aom_alarm_rules supports cluster filtering by cluster_id only. Do not use cluster_name as a filter.

3. Notification Rules

# List existing action/notification rules
python3 scripts/huawei-cloud.py huawei_list_aom_action_rules \
  region=cn-north-4

# Preview creating a notification action rule from an SMN topic
python3 scripts/huawei-cloud.py huawei_create_aom_notification_action_rule \
  region=cn-north-4 rule_name=auto-cluster-xxx \
  notification_topic_name=<smn-topic-name> \
  notification_topic_urn=<smn-topic-urn>

# Confirm creation
python3 scripts/huawei-cloud.py huawei_create_aom_notification_action_rule \
  region=cn-north-4 rule_name=auto-cluster-xxx \
  notification_topic_name=<smn-topic-name> \
  notification_topic_urn=<smn-topic-urn> \
  confirm=true

Never choose a notification rule automatically. If bind_notification_rule_id is missing for batch alarm creation, list available action rules and wait for explicit user selection, or ask the user to provide an SMN topic and create a notification rule first.

4. Alarm Rule Creation

# Preview creating a CCE template metric alarm rule
python3 scripts/huawei-cloud.py huawei_create_aom_alarm_rule \
  region=cn-north-4 cluster_id=<cluster-id> \
  alarm_item=NodeCPUUsageHigherThanEightyPercent

# Confirm creating a CCE template metric alarm rule
python3 scripts/huawei-cloud.py huawei_create_aom_alarm_rule \
  region=cn-north-4 cluster_id=<cluster-id> \
  alarm_item=NodeCPUUsageHigherThanEightyPercent \
  bind_notification_rule_id=<action-rule-id> \
  confirm=true

# Preview creating an event alarm rule
python3 scripts/huawei-cloud.py huawei_create_aom_event_alarm_rule \
  region=cn-north-4 cluster_id=<cluster-id> \
  event_name="<event-name>"

# Confirm creating an event alarm rule
python3 scripts/huawei-cloud.py huawei_create_aom_event_alarm_rule \
  region=cn-north-4 cluster_id=<cluster-id> \
  event_name="<event-name>" \
  bind_notification_rule_id=<action-rule-id> \
  confirm=true

Metric rules with cluster_id + alarm_item and event rules both use the same CCE template payload path and default naming as batch creation. Pass rule_name only when the user explicitly requests a custom name. Single-rule creation requires explicit bind_notification_rule_id during confirmed execution, same as batch CCE alarm rule configuration. Event alarm rule event_name should reference references/cce-event-list.md; metric alarm_item should use the CCE template alias or rule name.

5. CCE Template Alarm Rules

# Preview batch creating CCE recommended alarm rules
python3 scripts/huawei-cloud.py huawei_configure_cce_aom_alarm_rules \
  region=cn-north-4 cluster_id=<cluster-id> \
  bind_notification_rule_id=<action-rule-id>

# Confirm batch creation
python3 scripts/huawei-cloud.py huawei_configure_cce_aom_alarm_rules \
  region=cn-north-4 cluster_id=<cluster-id> \
  bind_notification_rule_id=<action-rule-id> confirm=true

# Preview cleanup of CCE template alarm rules for a cluster
python3 scripts/huawei-cloud.py huawei_cleanup_cce_aom_alarm_rules \
  region=cn-north-4 cluster_id=<cluster-id>

# Confirm cleanup
python3 scripts/huawei-cloud.py huawei_cleanup_cce_aom_alarm_rules \
  region=cn-north-4 cluster_id=<cluster-id> \
  delete_auto_notification_rule=true confirm=true

huawei_configure_cce_aom_alarm_rules requires explicit bind_notification_rule_id. If not provided, do not return or choose available_notification_rules from this tool; call huawei_list_aom_action_rules separately and present choices to the user.

6. Alarm Rule Mutation

# Preview update
python3 scripts/huawei-cloud.py huawei_update_aom_alarm_rule \
  region=cn-north-4 rule_name=my-rule threshold=80

# Confirm update
python3 scripts/huawei-cloud.py huawei_update_aom_alarm_rule \
  region=cn-north-4 rule_name=my-rule threshold=80 confirm=true

# Preview delete
python3 scripts/huawei-cloud.py huawei_delete_aom_alarm_rule \
  region=cn-north-4 rule_name=my-rule

# Confirm delete
python3 scripts/huawei-cloud.py huawei_delete_aom_alarm_rule \
  region=cn-north-4 rule_name=my-rule confirm=true

# Preview disable or enable
python3 scripts/huawei-cloud.py huawei_disable_aom_alarm_rule \
  region=cn-north-4 rule_id=<rule-id>

python3 scripts/huawei-cloud.py huawei_enable_aom_alarm_rule \
  region=cn-north-4 rule_id=<rule-id>

7. Action And Mute Rule Visibility

# Query mute rules
python3 scripts/huawei-cloud.py huawei_list_aom_mute_rules \
  region=cn-north-4

# Preview deleting an action rule
python3 scripts/huawei-cloud.py huawei_delete_aom_action_rule \
  region=cn-north-4 rule_name=<rule-name>

8. Cluster Alarm Inspection

python3 scripts/huawei-cloud.py huawei_aom_alarm_inspection \
  region=cn-north-4 cluster_id=<cluster-id>

Risk Levels

This skill includes read-only query tools and mutation tools. Mutation tools must be previewed first and must wait for explicit user confirmation before using confirm=true.

Level	Meaning	Execution Guidance
R3	No-risk read-only query or local analysis	May run automatically
R2	Low-risk monitoring configuration change, such as creating alarm or notification configuration without deleting resources or increasing service capacity/cost	Preview first; execute only after explicit user confirmation
R1	Risky monitoring change, such as updating or disabling a rule in a way that may reduce observability	Preview first; require explicit user confirmation and `confirm=true`
R0	Critical monitoring protection removal, such as deleting alarm rules, action rules, or broad cleanup	Require explicit confirmation, impact review, and rollback plan before `confirm=true`

Tool	Operation Type	Risk Level	Description
`huawei_list_aom_alarms`	Query	R3	Query active + historical alarms, merged and deduplicated
`huawei_list_aom_current_alarms`	Query	R3	Query current active alarms only
`huawei_analyze_aom_alarms`	Query + local analysis	R3	Analyze alarm groups, bursts, attention alarms, and chronic alarms
`huawei_aom_alarm_inspection`	Query + local analysis	R3	Inspect cluster alarm health
`huawei_list_aom_alarm_rules`	Query	R3	Query AOM alarm rules; supports optional `cluster_id`
`huawei_resolve_cce_aom_prom_instance`	Query	R3	Resolve the cluster AOM Prometheus instance
`huawei_list_aom_action_rules`	Query	R3	Query action/notification rules
`huawei_list_aom_mute_rules`	Query	R3	Query mute rules
`huawei_create_aom_alarm_rule`	Create	R2	Create an AOM metric alarm rule
`huawei_create_aom_event_alarm_rule`	Create	R2	Create an AOM event alarm rule
`huawei_create_aom_notification_action_rule`	Create	R2	Create a notification action rule from a user-provided SMN topic
`huawei_configure_cce_aom_alarm_rules`	Batch create	R2	Create CCE recommended alarm rules using an explicit existing notification action rule
`huawei_enable_aom_alarm_rule`	Enable	R2	Enable an AOM alarm rule
`huawei_update_aom_alarm_rule`	Update	R1	Update an AOM alarm rule
`huawei_disable_aom_alarm_rule`	Disable	R1	Disable an AOM alarm rule
`huawei_delete_aom_alarm_rule`	Delete	R0	Delete an AOM alarm rule
`huawei_cleanup_cce_aom_alarm_rules`	Batch delete	R0	Delete CCE template alarm rules for the target cluster
`huawei_delete_aom_action_rule`	Delete	R0	Delete a notification action rule

Parameter Reference

Common Parameters

Parameter	Required/Optional	Description	Default
`region`	Required	Huawei Cloud region	`HUAWEI_REGION`
`cluster_id`	Tool-specific	CCE cluster ID; required for cluster-scoped operations	N/A
`ak`	Optional	Explicit AK; highest priority for hcloud calls	profile/env fallback
`sk`	Optional	Explicit SK; highest priority for hcloud calls	profile/env fallback
`project_id`	Optional	Explicit Project ID	hcloud/IAM/env fallback
`confirm`	Mutation only	Execute a previewed mutation when set to `true`	`false`

Alarm Query Parameters

Tool	Required	Optional
`huawei_list_aom_alarms`	`region`	`cluster_id`, time-window/filter params
`huawei_list_aom_current_alarms`	`region`	`cluster_id`, filter params
`huawei_analyze_aom_alarms`	`region`	`cluster_id`, time-window/filter params
`huawei_aom_alarm_inspection`	`region`, `cluster_id`	filter params

Alarm Rule Parameters

Tool	Required	Notes
`huawei_list_aom_alarm_rules`	`region`	Optional `cluster_id`; no `cluster_name` filtering
`huawei_create_aom_alarm_rule`	`region`	Template mode: `cluster_id`, `alarm_item`; manual mode: metric fields. Confirmed execution requires `bind_notification_rule_id`
`huawei_create_aom_event_alarm_rule`	`region`, `cluster_id`, `event_name`	R2; confirmed execution requires `bind_notification_rule_id`; optional `rule_name` overrides template naming
`huawei_create_aom_notification_action_rule`	`region`, `rule_name`, `notification_topic_urn`, `notification_topic_name`	R2; user must provide the topic
`huawei_configure_cce_aom_alarm_rules`	`region`, `cluster_id`, `bind_notification_rule_id`	R2; user must explicitly choose the notification rule
`huawei_cleanup_cce_aom_alarm_rules`	`region`, `cluster_id`	R0; optional `delete_auto_notification_rule=true`
`huawei_update_aom_alarm_rule`	`region`, `rule_name`	R1; update fields are tool-specific
`huawei_delete_aom_alarm_rule`	`region`, `rule_name`	R0
`huawei_disable_aom_alarm_rule`	`region`, `rule_id`	R1
`huawei_enable_aom_alarm_rule`	`region`, `rule_id`	R2

Output Format

See Output Schema for the complete JSON response structure.

Key output fields:

success: command success status
error: failure reason when success=false
preview: mutation preview when confirm=true is not provided
requires_confirmation: whether user confirmation is required
report: alarm-correlation summary
issues: inspection risk items
alarms, rules, action_rules, mute_rules: queried resources

Workflow

See Workflow for the detailed alarm-correlation flow.

Recommended workflow:

Read the alarm name, resource, time window, and severity level from the user.
Default to the last 1 hour unless the user gives a different incident window.
Query active + historical alarms with huawei_list_aom_alarms.
Analyze alarm groups with huawei_analyze_aom_alarms.
Query alarm rules, action rules, and mute rules when notification gaps or suppression are suspected.
Group findings by resource, namespace, node, workload, and alarm type.
Hand off to diagnosis skills for Pod, Node, Network, Storage, or Workload root cause analysis.

Verification

Run read-only checks first:

python3 scripts/huawei-cloud.py huawei_list_aom_alarms region=cn-north-4
python3 scripts/huawei-cloud.py huawei_list_aom_alarm_rules region=cn-north-4
python3 scripts/huawei-cloud.py huawei_list_aom_action_rules region=cn-north-4

For cluster-scoped checks:

python3 scripts/huawei-cloud.py huawei_aom_alarm_inspection \
  region=cn-north-4 cluster_id=<cluster-id>

Mutation verification:

Run the command without confirm=true.
Review preview, affected resources, and expected impact.
Wait for explicit user confirmation.
Re-run with confirm=true.
Verify with the corresponding list/query command.

Best Practices

Use active + historical alarms for diagnosis - active-only views can miss recently recovered issues.
Filter by cluster_id - use exact cluster ID for cluster-scoped alarm rule and alarm analysis tasks.
Preview every mutation - never skip the preview step for R2/R1/R0 tools.
Do not auto-select notification rules - always wait for user selection of bind_notification_rule_id.
Keep notification setup explicit - create notification action rules only from a user-provided SMN topic.
Separate analysis from remediation - recommend remediation, then hand off to remediation skills.
Protect observability - avoid disabling or deleting rules unless impact is understood and confirmed.

Notes

Alarm diagnosis must consider both active and historical alarms; do not conclude the cluster is healthy only because there are no active alarms.
R2/R1/R0 tools must use preview-first execution and wait for explicit user confirmation before applying changes.
Do not automatically select notification rules, SMN topics, clusters, or templates for the user.
Do not modify alarm resources outside the tools provided by this skill.
Use cluster_id for cluster-scoped filtering; cluster names are not accepted by huawei_list_aom_alarm_rules.

Troubleshooting

Symptom	Likely Cause	Action
hcloud command fails	Missing profile, region, or IAM permission	Check `hcloud configure list` and IAM policy
No active alarms	Issue may have recovered or only exists in history	Query active + historical alarms
Batch configure fails without notification rule	`bind_notification_rule_id` is missing	List action rules or create one from an SMN topic, then retry
Cluster alarm rule query returns too many rules	Missing `cluster_id` filter	Re-run with exact `cluster_id`
Project ID errors	Profile/project resolution failed	Provide `project_id` explicitly or fix hcloud profile

Limitations

This skill does not modify CCE, ECS, ELB, EIP, VPC, security group, node, workload, or Kubernetes resources.
This skill must not create, update, or delete mute rules.
This skill must not use out-of-band cloud commands to modify alarm resources.
This skill cannot safely infer a notification rule for the user.
huawei_list_aom_alarm_rules filters by cluster_id, not cluster name.

References

Document	Use
Operation Guide	Detailed parameters, output expectations, verification, diagnosis hand-off, and pitfalls
Workflow	Alarm correlation workflow
Output Schema	Output JSON schema
Risk Rules	Risk boundaries and confirmation rules
CLI Installation Guide	hcloud/KooCLI installation and profile checks
IAM Policies	Required AOM, CCE, and IAM permissions
Verification Method	Functional and mutation verification checklist
Acceptance Criteria	Skill acceptance criteria and test cases
CCE Event List	CCE event names for event alarm rules
Prometheus Metric Alarms	Prometheus metric alarm references

huawei-cloud-cce-alarm-correlation-engine