Procore Integration Incident Command
Overview
Protect construction data integrity while restoring service. Separate provider outage, identity failure, tenant-routing defect, permission drift, rate exhaustion, event-delivery gap, and internal deployment regression before choosing recovery actions.
Prerequisites
- Incident commander, severity, affected tenant aliases, start time, and objectives
- Current release, last known good release, queue and reconciliation checkpoints
- Procore status, Integration Health, API activity, and approved communication channels
Instructions
Step 1: Stabilize
Pause risky mutations, preserve durable queues, stop retry storms, and keep read-only evidence collection bounded. Do not delete events or rotate credentials without a matching hypothesis.
Step 2: Classify
Compare Procore status with route-level 401, 403, 404, 422, 429, 503, latency, webhook delivery, and backlog signals. Segment by release, environment, company, project, and operation.
Step 3: Test the boundary
Run one read-only connection proof using the intended principal and company context. Check app connection, DMSA permissions, permitted projects, enabled tools, and rate headers.
Step 4: Mitigate narrowly
Roll back a bad release, repair tenant routing, reauthorize or rotate only when credential evidence supports it, reduce concurrency, or disable the affected mutation path.
Step 5: Recover data
Determine the gap window, replay durable work idempotently, and run REST reconciliation. Validate record counts and state hashes before reopening downstream publication.
Step 6: Settle and learn
Monitor until objectives recover, record root cause and contributing controls, assign follow-ups, and retain only redacted evidence with a deletion schedule.
Authentication
Incident probes use the existing OAuth 2.0 principal unless identity failure is the proven cause. Tokens, client secrets, destination headers, and signed URLs never enter the incident timeline or chat.
Tool Discipline
Use Read and Grep to inspect telemetry, configurations, runbooks, and sanitized logs. Use Write or Edit only for the approved mitigation, recovery artifact, timeline, or receipt; consequential provider changes require the incident commander and administrator.
Output
- Classification, scope, timeline, and mitigation decision
- Gap-window reconciliation and parity evidence
- Root cause, rollback, follow-up, and redacted incident receipt
Return current severity, customer impact, safe action, owner, recovery signal, and next checkpoint.
Examples
Webhook deliveries fail during receiver downtime. The team preserves the queue, fixes ingress, identifies the exact gap window, reconciles changed resources through REST, compares downstream state hashes, and resumes mutations only after parity passes.
Error Handling
| Failure | Response |
|---|---|
| Cause is ambiguous | Keep mutations paused and gather one discriminating read-only observation. |
| Retry storm threatens limits | Stop workers, honor provider headers, and resume through controlled queues. |
| Event gap is unknown | Widen reconciliation to the last trusted checkpoint rather than guessing. |
| Credential exposure is confirmed | Rotate or revoke, invalidate caches, and investigate unknown API activity. |