Fly.io Failure Triage
Overview
Classify the failing layer before changing anything. Separate local build failures, release orchestration, Machine lifecycle, Fly Proxy routing, private DNS, volume placement, and application behavior so remediation does not destroy the evidence.
Prerequisites
- App and environment identifiers plus incident start time
- Last known healthy release, image, configuration, and region placement
- Read-only access to health, Machine, release, volume, and log evidence
Instructions
Step 1: Freeze the incident window
Record the symptom, first observation, affected regions, recent release or secret changes, and the last known healthy revision.
Step 2: Validate configuration and release identity
Compare the deployed image and rendered configuration with the intended release. Check process groups, ports, services, health paths, signals, and timeouts.
Step 3: Inspect health and Machine lifecycle
Distinguish a failed health check from a Machine that is stopped, suspended, replacing, or repeatedly restarting. Preserve instance-version information.
Step 4: Check routing and networking
Verify public allocation, Fly Proxy service configuration, listening address, target port, and private .internal DNS semantics. Remember that stopped Machines are absent from AAAA responses.
Step 5: Check regional storage dependencies
Confirm that any attached volume exists in the same region and is mounted at the expected path. Treat a single volume as a single-host durability boundary.
Step 6: Choose the smallest recovery
Prefer rollback, configuration correction, or one bounded Machine action over fleet-wide restart. Reconcile health and image state after recovery.
Authentication
Use a read-only organization token for observation where possible. A deploy-capable identity can change code and therefore reach runtime secrets; keep diagnostic access separate from mutation authority and never paste tokens into incident records.
Tool Discipline
Use Read and Grep to inspect application configuration, deployment evidence, provider documentation, fixtures, logs, schemas, and existing tests before proposing a change. Use Write or Edit only for an approved plan, configuration, implementation, test, or redacted receipt. Do not create, deploy, scale, restart, stop, suspend, destroy, rotate, revoke, expose, or migrate live Fly.io resources without explicit operator approval.
Output
- Layered incident timeline and affected-resource map
- Redacted evidence matrix for config, release, health, Machine, network, and volume state
- Bounded recovery or escalation plan with rollback and verification criteria
Return the target organization, app, environment, region set, Machine or database identifiers, source-contract fingerprint, evidence, unresolved risks, rollback state, and final decision without exposing tokens, secrets, connection strings, or customer data.
Examples
After a release, one region returns errors. The operator verifies the new image, finds the Machine started but failing its HTTP health path, confirms the process listens on the wrong port, rolls back that release, and records the configuration mismatch without restarting healthy regions.
Error Handling
| Failure | Response |
|---|---|
| Evidence is incomplete | Do not guess; identify the missing read surface, owner, and safe collection step. |
| Machine state oscillates | Capture current and version state, logs, exit information, and health before attempting another restart. |
| Volume is unavailable | Stop destructive deployment actions and verify region, attachment, snapshot, and restore options. |