Vast.ai Workload Recovery Runbook
Overview
Protect people and credentials first, then preserve recoverable data, stop uncontrolled spend, and restore service from an external checkpoint or last-known-good template. Provider states imply different actions and must not be collapsed into retry.
Prerequisites
- Severity, start time, affected instances/endpoints, release, and user impact
- Last verified external checkpoint or Serverless template
- Incident commander, data owner, billing owner, and authority for stop/destroy/rollback
Instructions
Step 1: Stabilize identity and scope
Confirm account/team context, affected IDs, current state, balance, and whether a credential or data exposure is involved.
Step 2: Classify the provider state
Outbid/stopped may retain disk and storage charges; exited is workload failure; scheduling may await a reclaimed GPU; unknown/offline is host heartbeat loss; low credit threatens availability and data.
Step 3: Preserve recovery evidence
Copy reachable artifacts and logs, record template/image identity, and verify the newest external checkpoint before any destroy.
Step 4: Contain cost or exposure
Stop new creates, revoke compromised keys, and use stop or destroy only under the incident's data-versus-cost decision.
Step 5: Restore on a clean target
Resume from the verified checkpoint on a compliant different offer, or roll Serverless back to the last-known-good template.
Step 6: Reconcile and close
Confirm service and output, destroy superseded resources, audit charges, rotate temporary access, and document the gap and prevention action.
Authentication
Use an incident key with the minimum temporary permissions and a short revocation deadline. Keep billing-write, team administration, workload storage, and control-plane authority separated.
Tool Discipline
Use Read and Grep to inspect manifests, configuration, provider output, and existing tests before proposing a mutation. Use Write or Edit only for the approved plan, implementation, test, or redacted receipt; do not create, update, destroy, or fund Vast.ai resources without explicit operator approval.
Output
- State-based incident timeline and containment decision
- Checkpoint/template recovery and service-validation evidence
- Resource cleanup, charge reconciliation, access rotation, and postmortem receipt
Return severity, IDs, state class, last good checkpoint/template, containment, restored target, data gap, spend impact, and closed resources.
Examples
After a host goes offline, the team avoids blind restarts, restores the last external checkpoint on a different verified offer, validates output, and retains the original instance ID for support and billing reconciliation.
Error Handling
| Failure | Response |
|---|---|
| No external checkpoint exists | State the recovery gap explicitly and attempt data salvage only if the host becomes reachable. |
| Balance is zero or negative | Escalate immediately; instances stop and resources may be at risk of deletion. |
| Destruction would erase the only data | Require the incident commander and data owner to make the containment decision. |
| Replacement also fails | Stop churn, compare common image/data/config factors, and escalate with redacted evidence. |