CAST AI Failure Triage
Overview
Separate observation, connectivity, policy, capacity, and disruption failures before proposing a change. Preserve the failing state, use current component topology, and stop when the evidence requires cloud-provider or CAST AI support access.
Prerequisites
- The exact kube context, cluster, region, time window, and observed symptom
- Read-only access to the
castai-agentnamespace - The declared installation owner: castctl, Terraform, GitOps, or console
Instructions
Step 1: Freeze the symptom
Record expected versus actual behavior, timestamps, workload identity, pending-pod reason, and recent configuration changes. Use Read and Grep on runbooks and IaC to determine whether Cost Monitoring, Node Autoscaling, or Workload Autoscaling is actually enabled.
Step 2: Check installation health
Use Bash(castctl:) for version or non-mutating status commands supported by the installed client. Use Bash(helm:) to inspect releases and values, then Bash(kubectl:*) to inspect workloads, readiness, events, and bounded logs in castai-agent. Do not restart components before collecting evidence.
Step 3: Classify the failure plane
| Plane | Evidence | Likely boundary |
|---|---|---|
| Connection | Agent readiness, outbound failures, console disconnect | Identity, network, or cloud permissions |
| Node scaling | Pending pods, policy bounds, node-template fit | Unsatisfied constraints or maximum CPU boundary |
| Workload scaling | Missing recommendations, policy assignment, metrics | Metrics server, confidence, policy, or unsupported workload |
| Disruption | Eviction denial, PDB events, deferred changes | PDB or selected apply mode |
| Reporting | Missing cost or savings window | Ingestion, baseline, adoption, or pricing configuration |
Step 4: Test one hypothesis
Choose the smallest reversible check. Confirm regional endpoint alignment, effective scaling-policy assignment, metrics availability, supported workload type, node-template constraints, and cloud quota. Treat the deprecated cluster minimum CPU setting as migration debt, not a current control to add.
Step 5: Decide the owner and remedy
Map the evidence to the owning layer. Change repository-managed values only through their source of truth; do not mix console edits into Terraform or GitOps ownership. Escalate with a redacted bundle when the failure is inside the hosted control plane or an undocumented provider response.
Tool Discipline
Use Read and Grep for configuration and runbook evidence. Use Bash(kubectl:), Bash(helm:), and Bash(castctl:*) only for bounded inspection commands. Do not apply, upgrade, restart, connect, disconnect, or expose Secret objects during diagnosis.
Output
- A timestamped symptom and environment summary
- Evidence grouped by failure plane
- One supported root-cause hypothesis with confidence
- A reversible remedy, rollback condition, and escalation owner
Examples
Recommendations are absent because metrics-server is missing, so the remedy belongs to cluster observability. A node remains pending because every approved node template conflicts with its constraints; increasing a global limit without reviewing the workload is not the remedy.
Error Handling
| Failure | Response |
|---|---|
| Kube context is ambiguous | Stop before any cluster command and resolve it |
| Logs include credentials or inventory | Redact locally and do not attach raw output |
| A PDB blocks Immediate mode | Preserve the PDB and evaluate Deferred mode with the workload owner |
| Evidence points to cloud quota | Escalate to the cloud owner with the exact denied dimension |