Checkpointed Vast.ai Training Run

Overview

Treat a training run as a recoverable state machine, not an SSH session. Bind code and image identity, select an offer through policy, persist checkpoints outside the disposable root disk, export final evidence, and destroy.

Prerequisites

• Immutable image and code revision with deterministic training command
• GPU, VRAM, disk, reliability, geography, and price policy
• Checkpoint destination, resume test, runtime deadline, and cleanup owner

Instructions

Step 1: Freeze the run manifest

Record code revision, image digest, dataset version, command, seed, expected checkpoint cadence, budget, and output destination.

Step 2: Select and create

Search only verified rentable offers that meet the manifest, record price components, and create one labeled instance. Persist new_contract immediately.

Step 3: Reach readiness safely

Poll structured instance state with a deadline and terminal branches. Confirm image identity, disk headroom, GPU model, and CUDA visibility.

Step 4: Run with external checkpoints

Start the workload so checkpoints are uploaded or copied to durable storage at the declared cadence. A local checkpoint alone is not recovery evidence.

Step 5: Verify completion or resume

Validate artifact checksums and run metadata. For an interruption, provision a replacement from policy and prove resume from the last durable checkpoint.

Step 6: Close the cost boundary

Copy final logs and checksums, destroy the instance, confirm removal, and reconcile actual spend against the manifest.

Authentication

Use a scoped control-plane key for search and instance operations. Give the workload only the storage credential needed for its checkpoint prefix, with no Vast.ai billing or team authority.

Tool Discipline

Use Read and Grep to inspect manifests, configuration, provider output, and existing tests before proposing a mutation. Use Write or Edit only for the approved plan, implementation, test, or redacted receipt; do not create, update, destroy, or fund Vast.ai resources without explicit operator approval.

Output

• Frozen run and offer-selection manifest
• State, checkpoint, resume, artifact, and spend evidence
• Confirmed instance destruction and discrepancy report

Return revision, image digest, offer and instance IDs, checkpoint URI/checksum, terminal result, actual spend, and cleanup confirmation.

Examples

A fine-tuning job checkpoints every ten minutes to a run-specific object prefix; after a simulated interruption, a replacement instance resumes from the last checksum and the original contract is destroyed.

Error Handling

Resources

• First-party source notes
• Instance overview
• Data movement
• Manage instances