Probe Input Integrity — Never Fabricate an Identifier You're About to Test Against

Promoted from the always-loaded ~/.claude/rules/never-fabricate-test-identifiers.md rule, whose stub keeps the gate lines. The body below is that rule's text.

When probing a system — an API endpoint, a URL, a record id, a file path, a ticket number — obtain the identifier from the system itself. Never invent a plausible-looking one. A fabricated identifier turns every subsequent observation into a two-variable experiment ("is the endpoint blocking me, or does this thing simply not exist?"), and the two failure modes are routinely indistinguishable in the response.

When to Use This Skill

The trap

The fabricated value looks real, so it never gets questioned. Its failure then gets attributed to the hypothesis actually under test.

Canonical break (2026-07, a research repo): investigating whether Reddit's .json endpoint still worked, I made up a thread URL — /r/LocalLLaMA/comments/1i6r9rp/deepseek_r1_is_now_available_on_azure — a plausible id and slug, entirely invented. It returned 403 Blocked, as did two further probes against the same fake thread. I reported the endpoint as IP-blocked. The user said "it works in my browser," and the first successful request revealed the truth: the endpoint returned a JSON 404 — the thread had never existed. The real discriminator was the client fingerprint, not the IP, and three "conclusive" probes had been testing a nonexistent resource.

The tell that should have fired: 403 and 404 mean very different things, and I never established which I'd get for a known-good id.

The rule

Get a real identifier from the source before probing. List before you get.

# Wrong — invent a plausible thread, then test against it
curl https://www.reddit.com/r/X/comments/<made-up-id>/<made-up-slug>.json

# Right — pull a real one out of the system first, then probe that
curl -s https://www.reddit.com/r/X/ | grep -o '/r/X/comments/[a-z0-9]*/[a-z0-9_]*/'

Generally: a listing/search call before an item call. gh issue list before gh issue view <n>; Glob/ls before Read; a collection endpoint before an item endpoint.

Always run a known-good control. When a probe fails, repeat it against an identifier you know exists. Without a control you cannot separate "access denied" from "not found" from "wrong shape" — and each implies a completely different fix.

Don't fabricate the subject either — extract the code, don't retype it

The same trap one level up. When you build a harness to exercise code that is awkward to run directly, a retyped copy of that code is not that code. It reads as identical in the transcript, and when it misbehaves the failure is attributed to the code under test rather than to your transcription.

Observed 2026-08 (a ComfyUI node-pack fleet): verifying new just prs / just merge recipes over a 14-repo fleet that happened to have zero open PRs. The harness retyped the kind comparison as [ "$k" = "'release'" ] — literal quotes, matching nothing — and reported PLAN=0 for every kind. That is precisely what "no PR qualifies" looks like. Extracting the shipped loop straight out of the justfile with sed and re-running produced the correct per-kind plans and skip reasons.

• Extract the shipped text, e.g. sed -n '/start/,/end/p' <file>, and feed it the inputs. Never hand-paste the logic under test into the harness.
• When your own target set is empty, borrow a control set. A sweep over zero items is green by construction and asserts nothing. Point the extracted code at real objects from elsewhere that cover every branch — for the recipes above, live PRs from other repos spanning release (bot- and human-authored) / deps / mine / external, plus draft, BLOCKED, UNSTABLE, and failing-checks.

An empty run and a broken harness produce the same output. Only the control tells them apart.

Don't fabricate the environment either — an inert stub means the real tool ran

The third form. The subject is real and the inputs are real, but the harness never took effect: a stubbed CLI placed on PATH that is not executable is skipped by PATH lookup in silence, and the real binary runs instead. Nothing warns. The run completes, prints plausible output, and — if the real tool has side effects — performs them.

Observed 2026-08 (an infrastructure repo): testing a workflow step that posts a PR comment, I wrote a fake gh onto PATH with the Write tool, which does not set the executable bit. The real gh ran, and what was framed as a local test POSTed and then PATCHed a comment onto a live PR. It surfaced only because the output carried a real comment ID where the stub's canned 12345 was expected. Re-running with chmod +x behaved correctly.

• Set the bit and check it. chmod +x <stub> via Bash after any Write, then ls -l. File-creation tools generally do not set the executable bit.
• Assert the stub is the one in effect before trusting any case. command -v <tool> at the top of the harness, or better, have the stub print a sentinel the real tool never would and fail the run if it is absent.
• Make the stub's output impossible to mistake for the real thing — a canned 12345 beats a realistic-looking id, so a discrepancy is visible at a glance rather than plausible.
• Treat "it's only a local test" as a reason for more isolation, not less. That framing is what suppresses the caution normally applied to a command that writes to shared state, and whether the writing tool runs is precisely what is in question.

Same law as the two forms above, one layer out: the harness was green while measuring production.

When it bites

• Diagnosing whether an endpoint, credential, or permission works — exactly where a wrong conclusion triggers a redesign that was never needed.
• Writing tests or repro cases against "example" ids nobody verified exist.
• Citing an issue/PR/commit number from memory instead of looking it up. A plausible-but-wrong #162 is worse than no number: it reads as researched.

Relationship to sibling rules

• verify-upstream-before-patching.md, verify-machine-facts-before-publishing.md — same instinct (consult the authoritative source before acting). This skill covers the inputs to your own diagnostics, not the code you patch or the facts you publish.
• tool-result-traps and harness-tool-errors (Read on a missing path; the rg -r fabrication trap) — siblings in the same family: a confidently-wrong output that looks exactly like a real one.

Rationale

An invented identifier is a silent confound. It costs one extra lookup to obtain a real one, and one extra probe to run a control; skipping both bought a wrong root cause, a confidently-reported non-finding, and a diagnosis that unravelled only because a human said "but it works for me." The model will not catch this on its own — in the transcript, a fabricated value looks exactly as legitimate as a real one.