A Green Release Pipeline Is Not a Shipped Artifact — Go Look For the Artifact

Promoted from the always-loaded release-artifact-verification.md portfolio rule, whose stub keeps the gate lines.

A release "succeeded" when the thing you released is downloadable by someone else. Every intermediate signal — a green workflow badge, a published GitHub release, a version tag, a passing check — is a claim about mechanics, and each can be true while nothing shipped. Verify the artifact at its public address, from outside your own tree, in the same breath you announce the release.

Canonical break (2026-08, pal-mcp-server): publish-pypi had failed on every release since it was added — 10.2.2, 10.3.0, 10.4.0, 10.4.1, 10.4.2 — and the package had never existed on PyPI. A month of releases, each one green where anyone would look.

When to Use This Skill

1. A release is not atomic — the failing job is not the one you watch

release-please (and most release pipelines) cuts the tag and the GitHub release in one job and publishes artifacts in another. When the publish job fails, the tag exists, the GitHub release page renders perfectly, the changelog is updated, and the notification says a release happened. The only red is one job inside a run nobody opens.

• Check, always, per artifact class:

  curl -s -o /dev/null -w '%{http_code}\n' https://pypi.org/pypi/<pkg>/json     # 200
  curl -s https://pypi.org/pypi/<pkg>/json | jq -r .info.version                # the version you cut
  gh api users/<owner>/packages/container/<pkg>/versions --jq '.[0].metadata.container.tags'
  npm view <pkg> version
  

• Look at the run's jobs, not its conclusion, when a release "succeeds":

  gh api repos/<o>/<r>/actions/runs/<id>/jobs --jq '.jobs[] | {name, conclusion}'
  

• Fix the design, not just the release: make the publish job's failure visible — gate the release on it, or add a post-release check that asserts the version resolves publicly. A pipeline that can half-succeed silently will.

2. Verify from outside the source tree, or you may be reading the source

An install check run inside the project directory can resolve the local working copy instead of the artifact you just published, and report a plausible-but-wrong version. python -c puts the cwd on sys.path, so a stale *.egg-info/ or a source package in the repo root shadows the installed distribution.

# Wrong — cwd is the repo root; reads the stale local egg-info
$ .venv/bin/python -c "import importlib.metadata as m; print(m.version('pkg'))"
10.4.1                      # ← the version from ./pkg.egg-info/PKG-INFO

# Right — neutral cwd, and confirm the dist-info the venv actually holds
$ cd /tmp && /path/.venv/bin/python -c "import importlib.metadata as m; print(m.version('pkg'))"
10.4.3
$ ls /path/.venv/lib/python*/site-packages | grep 'pkg.*dist-info'
pkg-10.4.3.dist-info

Generalises past Python: node_modules resolution walks upward, cargo prefers a path override, go honours a replace directive. The rule is to verify in a directory that has no relationship to the source, and to confirm against the installed metadata (dist-info, package.json in node_modules) rather than an import that could have come from anywhere.

The failure is silent and reads as a real result — same family as agent-patterns-plugin:probe-input-integrity: a broken harness and a genuine finding produce identical-looking output.

3. A check that never ran reports pass

Path-filtered CI checks (file-patterns, paths:) complete in 4–6 seconds having analysed nothing, and report success. A sibling PR's green tick is not a control unless it actually executed. Full treatment in github-actions-plugin:ai-review-max-turns § "a check that never ran looks exactly like a pass"; noted here because it is the third way a release looks verified and isn't.

4. A build job that never runs the image publishes a broken one, green

A docker build that succeeds proves the layers assembled, not that anything inside can start. A pipeline that builds and pushes without ever executing the entrypoint will publish an image whose every import fails, report success, and deploy it.

Example: a dependency-bot PR moves a multi-stage image's runtime stage to python:3.14-alpine while the builder stage stays on 3.12. The venv copied between stages keeps its packages in lib/python3.12/, so the 3.14 interpreter drops /app/.venv from sys.path entirely. Every service built from it crash-loops for days, hundreds of restarts each, while build, push, release, and every CI check stay green — nothing in the pipeline ever ran the image.

• Gate the push on a smoke step, not the other way round. Build with load: true, run the entrypoint, then push. A broken image never reaches the registry, so there is nothing to roll back.
• Assert the environment, not just an import. import <pkg> alone can pass for the wrong reason; assert the venv is actually on the path: assert [p for p in sys.path if "/app/.venv/" in p].

docker run needs -i or the check is vacuous. Without it docker does not forward stdin, so a heredoc-fed docker run --rm IMG python - reads an empty script and exits 0 having asserted nothing — the gate then passes on every image, including a deliberately broken one:

docker run --rm -i "$IMAGE" /app/.venv/bin/python - <<'EOF'

This is the same family as §2 and agent-patterns-plugin:probe-input-integrity § an inert stub means the real tool ran: the harness is green while measuring nothing. It was caught only by running the finished gate against a known-bad image and requiring it to fail — do that before trusting any gate you just wrote.

The litmus test

Before saying "released", answer: "What URL would a stranger fetch to get this, and have I fetched it?" If the answer cites a workflow badge, a tag, or a GitHub release page, you have verified the mechanics and not the outcome.

Related

• github-actions-plugin:multirepo-ci-cd — a re-run replays the stale workflow, so re-running a failed publish after merging its fix does not test the fix
• github-actions-plugin:ai-review-max-turns — the skipped-check-reports-pass trap in full
• migration-patterns-plugin:tool-migration-cutover — the same law before the fact: removal is gated on a positive operational signal, not on config being present
• git-plugin:git-merge-hazards — "exit 0 is a claim about mechanics, not content", of which this skill is the release-time instance