Multi-Repo CI/CD Discipline

Org-agnostic lessons for working across a portfolio of repos that share reusable workflows. The org-specific workflow catalogs live in the child rules (*/.claude/rules/ci-cd-workflows.md).

Diagnosing a CI Failure — Fetch First

Before investigating why a workflow failed, git fetch and compare local HEAD against origin/main. In a multi-repo portfolio, local checkouts go stale, and the failing workflow file (or its config — release-please-config.json, the manifest, a new .yml) may have been added in commits you haven't pulled. It won't exist in your local tree, so you'll search for a "missing" file that CI is actively running. One git fetch + git log --oneline HEAD..origin/main shows the gap; git pull --ff-only syncs it. Diagnose against what CI actually ran, not a stale snapshot.

Rolling a Workflow Out to a Repo Class

When promoting a self-contained workflow into a shared reusable-*.yml and adding a thin caller to every repo of a class, three things bite if missed:

1. IaC feature flags are the source of truth for the repo set. Don't hand-list repos — derive the class from the gitops/Terraform definitions (e.g. release_please = true entries in repositories.tf are the membership list). Extract them rather than guessing which repos qualify:

   awk '/^    "[^"]+" = \{/ { match($0, /"[^"]+"/); name=substr($0, RSTART+1, RLENGTH-2) } /release_please[ ]*=[ ]*true/ { print name }' repositories.tf
   

2. Land the reusable workflow on the .github repo's main first. Callers pin uses: <owner>/.github/...@main, which resolves at dispatch time — a caller merged or dispatched before the reusable workflow exists on main fails. Merge the .github PR before opening (or at least before merging) the callers. workflow_dispatch-only callers are harmless until triggered, so they can be opened in parallel, but the .github change is the prerequisite.
3. Bulk-create callers via the GitHub API, not local clones. The caller body is identical across repos, and some repos may not be checked out locally. Creating the branch + file + PR through the API is uniform and immune to local working-tree state (stale checkouts, wrong branch, uncommitted changes):

   sha=$(gh api repos/<owner>/$r/git/ref/heads/main --jq .object.sha)
   gh api -X POST repos/<owner>/$r/git/refs -f ref=refs/heads/$BRANCH -f sha=$sha
   gh api -X PUT repos/<owner>/$r/contents/$PATH -f message=$MSG -f content=$B64 -f branch=$BRANCH
   gh pr create -R <owner>/$r --base main --head $BRANCH -a laurigates --title "$MSG" --body "$BODY"
   

   Loop sequentially (not a parallel batch — siblings cancel on the first non-zero exit), and skip a repo if the file already exists on main. Title the caller PR ci: so release-please does not cut a version bump for a CI-only change. When the running gh user is the repo owner, omit --add-reviewer (self-review is rejected 422).

Bulk-Read Through the API Too — a Stale Checkout Produces Wrong Findings

The API-not-clones rule above is usually framed as a write discipline. It is equally a read discipline: a portfolio-wide rg/grep over local checkouts answers a question about whatever each clone last fetched, not about the repos. In a side-by-side layout where most clones are touched rarely, that answer is confidently wrong — and wrong in the direction of "this pattern isn't used anywhere," because a checkout predating an adoption simply lacks it.

Evidence (2026-07, ubuntu-slim runner sweep): a local grep -rl concluded 0 repos called reusable-release-please.yml and all 29 inlined googleapis/release-please-action directly. That statement was made to the user as fact. pal-mcp-server had migrated to the reusable on the remote weeks earlier; the local clone was stale. Nothing about the grep output looked suspicious.

• Read each file from origin in the same pass that patches it: gh api repos/<owner>/<repo>/contents/<path> --jq .content | base64 -d. One fetch per repo is cheap and makes the finding and the edit agree.
• Use the local checkouts only to generate candidates, never to state conclusions. Enumerate broadly locally (it's fast), then confirm every claim against the remote before reporting or acting on it.
• Make the patch function assert its precondition and raise. The stale pal-mcp-server clone was caught only because the sweep's patcher demanded a runs-on: inside the named job and raised "no runs-on found inside release-please job" rather than falling through to a no-op or a mangled file. A silent if found: patch would have skipped it invisibly — and the wrong finding would have stood. Same instinct as git-hazards.md: a green run is not proof the content is right.
• Cross-check the repo set itself against the remote as well (gh repo list <owner> --json name,isArchived,isFork) — a local directory can be a fork, an archived repo, or have no remote at all.

Landing a @main Reusable-Workflow Fix on an Existing PR — Re-Trigger, Don't Re-Run

After you merge a fix to a reusable-*.yml on the .github repo's main, the open PRs whose callers pin @main do not pick it up by re-running the failed check. A GitHub re-run replays the original run against the workflow resolution it was first dispatched with — including the exact commit the @main reusable workflow resolved to at that time. So gh run rerun / "Re-run failed jobs" / the rerun-failed-jobs API all replay the stale reusable workflow and fail identically, even though main is now fixed.

The tell: you read the re-run's log and it shows the old behaviour (the command you just replaced still running), not your fix.

Fix: force a fresh pull_request event, which re-resolves @main at dispatch time and picks up the merged fix. Cheapest first:

gh pr update-branch <n> -R <owner>/<repo>   # fires `synchronize`; no-op-safe if already current

If the branch is already current (update-branch reports "already up to date"), close-then-reopen fires reopened instead:

gh pr close <n> -R <owner>/<repo> && gh pr reopen <n> -R <owner>/<repo>

For a release-please PR, prefer update-branch — closing the release PR is safe (release-please acts on push-to-main/schedule, not PR-close events) but reopening is noisier than a branch update. Avoid pushing to a release-please branch by hand.

There is a second replay trap with the same shape: a pull_request re-run also replays the original event payload, so its sender is whoever opened the PR — a bot for Renovate/release-please PRs. Actions with a human-actor guard (e.g. anthropics/claude-code-action without allowed_bots) therefore fail with "Workflow initiated by non-human actor" on re-run regardless of who clicked re-run. A fresh event doesn't change the sender (still the bot), so the real fix is allowed_bots on the action, not re-triggering — but the same "re-run replays the original payload" mechanic is why clicking re-run yourself doesn't help. Both traps are the same root cause: a re-run is a replay, not a new dispatch.