Skill: Hunt a Prod Error

An opaque error has two halves. Local logs say where (route, status, timing, burst shape). Sentry says what (the exception and stack). You need both before naming a cause; either alone produces a guess.

1. Local logs: where and how often

Desktop server log (OTLP JSON, one record per request):

~/Library/Application Support/com.differentai.openwork/logs/openwork-server.log{,.1}

Engine log: ~/.local/share/opencode/log/opencode.log.

Tally 5xx by route, then compare against the same route family that did not fail. A failure rate on one route and zero on its siblings isolates the code path; a burst of failures in the same millisecond points at a shared event (GC, rollover, socket), not per-request data.

rg '"status":5' ~/Library/Application\ Support/com.differentai.openwork/logs/openwork-server.log* \
  | python3 -c 'import sys,json,re,collections; c=collections.Counter()
for l in sys.stdin:
  a=json.loads(l)["attributes"]; c[(a["method"], re.sub(r"(ses|ws|msg)_[A-Za-z0-9]+", r"\1_*", a["path"]), a.get("error.code"))]+=1
[print(v,k) for k,v in c.most_common()]'

Know the blind spot: the OpenCode proxy catch blocks in apps/server/src/server.ts forward the real exception to Sentry only; the local record carries the synthetic Unexpected server error and no cause.

2. Sentry: what threw

Sentry is connected through OpenWork Connect (search_capabilities → search_issues, get_sentry_resource). Org different-ai-inc, project desktop-app, region https://us.sentry.io.

Filter by the tags the server sets: surface:server and the templated route (for example route:"/workspace/:id/opencode/*"), plus release. Read the top issue's stack, firstSeen, users, and event count. The exact exception message is the string a regression spec must reproduce.

3. Name the regression

Correlate Sentry firstSeen with git log --since=<firstSeen - 2d> on the files in the stack. Confirm the shipped code matches source before reasoning about it:

npx --yes @electron/asar extract-file /Applications/OpenWork.app/Contents/Resources/app.asar server/dist/server.js

4. Reproduce on the shipping runtime

Reduce the mechanism to a script and run it on the runtime the error occurred in (run-tests → Match the runtime). A repro that passes on Bun for a Node/undici bug has proven nothing.

5. File and reply

Linear (team Engineering): symptom with the exact user-facing string, route tally, Sentry issue ids with users and counts, mechanism, repro, regressing PR, proposed fix. Link the Sentry issues as attachments.
Reply in the thread where the error was reported with the issue link and a five-line summary: what fails, how often, who is affected, why, what fixes it.
The fix PR carries a spec that reproduces the exact string on dev and passes on the head (write-a-spec). CI publishes the evidence.

hunt-a-prod-error