Flexport Delivery-Gap Incident Runbook

Overview

Preserve freight correctness first. Freeze uncertain mutations, separate provider delivery from local processing, and reconstruct affected state with documented resource/event reads.

Prerequisites

Incident owner, severity, start time, and affected tenant/workflow
Last known good release and configuration
Bounded reconciliation query and business owners for disputed state

Instructions

Step 1: Contain

Disable automatic bookings and affected writes. Keep authenticated webhook acceptance only if durable enqueue and deduplication remain correct.

Step 2: Bound the window

Identify first/last bad receipt by surface, operation, release, credential alias, and version without examining broad sensitive payloads.

Step 3: Split failure domains

Test OAuth, REST read, MCP connection, webhook ingress, queue processing, and downstream state independently.

Step 4: Reconcile read-only

Use /events or affected resource reads to compare provider state with durable local operation keys. Do not assume Flexport exposes a webhook replay endpoint.

Step 5: Repair deterministically

Reprocess authenticated durable events or apply approved state corrections exactly once; reconcile uncertain creates/bookings before retry.

Step 6: Recover and learn

Restore traffic by cohort, verify the full incident window, rotate exposed credentials if needed, and add a sanitized regression fixture.

Authentication

REST calls authenticate with a cached OAuth 2.0 client-credentials Bearer token using audience https://api.flexport.com, or an explicitly accepted broad API key. Use distinct credentials per workload and never log credentials or tokens. MCP calls use the authenticated connection to https://mcp.flexport.com/mcp and remain subject to each tool's documented account permissions.

Tool Discipline

Use Read and Grep for discovery and evidence. Use Write or Edit only for the approved artifact, code, configuration, test, or receipt described by this workflow; do not make an unapproved Flexport-side change.

Output

Scoped decision or implementation artifact
Redacted operation and validation receipt
Failure, rollback, and follow-up ownership record

Return a machine-reviewable receipt in this shape; adapt the operation values, but never place credentials or provider payloads in it:

surface: rest-v3
operation: shipment-read
decision: approved
outcome: verified
evidence:
  release_sha: recorded-out-of-band
  provider_reference: redacted
rollback_owner: logistics-platform

Examples

A receiver deployment parsed bodies before signature validation and dropped events. The team restores the prior release, queries the bounded event/resource window, deduplicates by durable operation identity, and backfills only missing local transitions.

Error Handling

Failure	Response
Provider reads unavailable	Maintain the mutation freeze and preserve the reconciliation window.
Duplicate business action found	Stop workers and choose one authoritative provider resource.
Event type newly additive	Update tolerant routing and replay only authenticated durable items.
Incident needs sensitive payload	Escalate access and minimize fields rather than copying broad logs.

flexport-incident-runbook