macos-incident-postmortem

v2026.09.24

Reconstruct macOS freeze, panic, or reboot from DiagnosticReports and shell history. Use when investigating hangs, panics, watchdog timeouts, jetsam, or thermal throttling.

GitHub
安装命令
npx skhub add laurigates/macos-incident-postmortem
Markdown
SKILL.md

macOS Incident Postmortem

When to Use This Skill

Use this skill when...Use something else when...
GUI froze and you're not sure if the machine rebootedLive-debugging a hung process — use sample / spindump
Investigating recent kernel panics or watchdog timeoutsApplication crashes — open the per-app .crash / .ips directly
Cross-referencing "what was I doing at time T?" against logsActive CPU diagnosis — see launchservices-health
Auditing whether the system is "due for a reboot"Pre-incident hardening — wrong skill, this is forensics

Platform Guard

This skill is macOS-only. /Library/Logs/DiagnosticReports/, kern.boottime, pmset -g log, and the log command are Darwin-specific. Refuse to act if uname -s is not Darwin.

test "$(uname -s)" = "Darwin" || { echo "macos-plugin: not Darwin, refusing"; exit 1; }

Core Expertise

A macOS "incident" can mean any of:

  • A kernel panic (system reset by the kernel)
  • A WindowServer userspace watchdog timeout (the GUI froze and the user power-cycled)
  • A LaunchServices / coreaudiod / mds-stores XPC stall (one daemon dragged the GUI down)
  • A jetsam memory-pressure event (the kernel killed apps to reclaim RAM)
  • A thermal throttle (CPU clamped to base frequency for minutes)
  • A user-initiated force-reboot after a hang (no panic; just lost state)

The first job in a postmortem is distinguishing between actual reboots and GUI hangs. The 2026-04-22 incident that motivated this plugin looked like a crash to the user but last reboot showed no reboot — the kernel was fine, only the GUI stack hung.

The second job is timeline reconstruction: cross-reference Diagnostic Reports, kern.boottime, last reboot, last shutdown, and shell history to answer "what happened around time T?".

Gather the deterministic signals

Run the bundled script to gather every mechanical signal in one pass — reboot-vs-hang detection, per-category DiagnosticReports counts, the CPU-event offender histogram, the jetsam victim list, and the fixed reboot/hang classifier:

bash "${CLAUDE_SKILL_DIR}/scripts/macos-incident-postmortem.sh" --home-dir "$HOME"

Parse STATUS= and ISSUES: from the output. The script emits CLASSIFICATION= (REBOOT_PANIC / REBOOT_CLEAN / HANG_UI / HANG_CPU / HANG_JETSAM / HANG_OR_POWERCYCLE / UNKNOWN) with a CLASSIFICATION_REASON=, plus PANIC_COUNT= / HANG_COUNT= / JETSAM_COUNT= / CPU_RESOURCE_COUNT= etc., LATEST_PANIC= / LATEST_HANG= paths, and CPU_OFFENDERS: / JETSAM_VICTIMS: histograms.

Pass the incident time T as an epoch to enable the classifier (MACOS_PM_INCIDENT_EPOCH=<epoch> bash …), or supply an alternate report source with --reports-dir <path>. With no incident time the classifier reports UNKNOWN rather than guessing — feed it T once you know it.

The reboot/hang decision tree the script encodes:

PatternInterpretation
Boot at/after T with a .panic reportTrue reboot from a kernel panic
Boot at/after T, no panicClean restart (user-initiated or watchdog)
Boot before T with a .hang/.spindump.txtGUI hang — machine did not reboot
Boot before T with a .cpu_resource.diagDaemon CPU storm
Boot before T with a JetsamEvent-*Memory-pressure kill
Boot before T, none of the abovePower loss or hard power-cycle

Diagnostic report category reference

/Library/Logs/DiagnosticReports/ collects everything macOS thinks is worth keeping; the script classifies each by filename suffix:

PatternCategorySeverity
*.panicKernel panicCritical
*.ips (process-specific)Userspace crash report (Apple's modern format)Per-process
*.crashLegacy userspace crashPer-process
*.cpu_resource.diagProcess exceeded CPU threshold (typ. 80% / 90s)Hot daemon
*.wakeups_resource.diagProcess woke the system too oftenPower drain
*.diskwrites_resource.diagProcess wrote too much to diskI/O drain
*.hangUI thread hang detectionGUI freeze
*.spindump.txtSpindump capture from a hangGUI freeze
JetsamEvent-*.ipsKernel killed processes for memory pressureRAM exhaustion

Note: Apple migrated most categories to the .ips extension circa Monterey. Older systems and some categories still produce legacy extensions. The script matches by suffix, not by exact filename.

last reboot reads /var/log/wtmp.X rotated logs. On modern macOS, also check the unified log when wtmp has rotated past the incident:

log show --predicate 'eventType == "stateEvent" AND (event == "boot" OR event == "shutdown")' \
  --last 7d --style syslog

Timeline Reconstruction (judgment)

With the deterministic classification and counts in hand, the remaining steps are the agent's job: read the panic backtrace, name suspect kexts, correlate with logs and shell activity, and write the narrative.

Step 1: Inspect the unified log around T

# Adjust the time window to bracket T
log show --start "2026-04-22 08:15:00" --end "2026-04-22 08:25:00" \
  --predicate 'subsystem == "com.apple.WindowServer" OR process == "launchservicesd" OR process == "coreaudiod"' \
  --style syslog \
  | head -500

Common signatures to grep for:

SignatureMeaning
WindowServer: watchdogUI froze long enough to trip the watchdog
posix_spawn failuresFork/exec storm — usually shell loops or runaway scripts
Jetsam KillingKernel killing processes for memory
_dispatch_*_timeoutDaemon stuck on a synchronous IPC call
Thermal pressureCPU thermal-throttled

Step 2: Inspect the panic / hang report

The script's LATEST_PANIC= / LATEST_HANG= lines give you the file to read. Key fields in a panic report:

FieldMeaning
panic(cpu N caller ...)The instruction that panicked; first line says why
Backtrace (CPU N)Call stack at the time of panic
Mac OS version, Kernel versionOS state
System uptime in nanosecondsUptime at the moment of panic
last loaded kext / loaded kextsLikely third-party suspect

Hang reports are spindump-style: one column per thread of the hung process (typically WindowServer or the offender daemon), with stack traces at sample intervals.

Step 3: Correlate with shell activity

# Most recent zsh history entries (assumes default zsh)
fc -l -t '%Y-%m-%d %H:%M:%S' -100

# Or directly:
tail -50 ~/.zsh_history

The zsh EXTENDED_HISTORY format : <epoch>:<elapsed>;<cmd> lets you grep for commands run within the incident window.

Step 4: Synthesize

Write a one-paragraph timeline including:

  • Type of incident (panic / hang / jetsam / power-cycle)
  • Time T and time-since-boot at T
  • Top suspects (loaded kexts, busy daemons, low-memory processes)
  • What recovery looked like (clean reboot / hard power-cycle / GUI returned on its own)

Common Patterns

The reboot-vs-hang one-liner, per-category report counts, the CPU-event offender histogram, and the jetsam victim list are all produced in one pass by the bundled script (scripts/macos-incident-postmortem.sh — see "Gather the deterministic signals" above). Read CLASSIFICATION=, the *_COUNT= keys, CPU_OFFENDERS:, and JETSAM_VICTIMS: from its output rather than re-running the equivalent shell pipelines. A sudden rise in any category count is a leading indicator before a major hang or panic; offenders/victims that recur are running near their CPU/memory budget.

Correlate with sleep/wake history

pmset -g log | grep -E 'Sleep|Wake|DarkWake' | tail -50

A spurt of DarkWake entries followed by a hang report often means a peripheral or sleep-assertion-holder is the trigger.

Skip List (Common False Suspects)

SuspectWhy it's usually NOT the cause
mds, mds_stores (Spotlight)Heavy I/O is normal after large file changes; rarely panics
clouddHigh network use is normal during iCloud sync
bird (CloudKit)Same as cloudd
Time MachineThrottles itself; almost never the proximate cause
kernel_task at 100%This is thermal management running, not a bug

If one of these is the only thing visible in your timeline, look harder — the real cause is usually a daemon that blocked on one of these, not the daemon itself.

Agentic Optimizations

ContextCommand
All deterministic signals + classifierbash "${CLAUDE_SKILL_DIR}/scripts/macos-incident-postmortem.sh" --home-dir "$HOME"
Classify against a known TMACOS_PM_INCIDENT_EPOCH=<epoch> bash "${CLAUDE_SKILL_DIR}/scripts/macos-incident-postmortem.sh"
Last 5 bootslast reboot | head -5
Last 5 shutdownslast shutdown | head -5
WindowServer log slicelog show --predicate 'subsystem == "com.apple.WindowServer"' --last 1h --style syslog | head -200

Quick Reference

Key paths

PathContents
/Library/Logs/DiagnosticReports/All system-wide reports
~/Library/Logs/DiagnosticReports/Per-user reports (rare; mostly legacy)
/var/log/wtmp.XReboot / shutdown record (read via last)
/var/log/asl/ASL legacy logs (mostly unused in 2026)
/var/db/diagnostics/Unified log binary database

Useful log show predicates

PredicateUse
subsystem == "com.apple.WindowServer"GUI hangs
process == "launchservicesd"LS XPC stalls
process == "coreaudiod"Audio daemon issues
eventType == "stateEvent"Boot/shutdown/sleep
eventMessage CONTAINS[c] "hang"Hang detection events
category == "ttsd"Speech synthesis stalls

Time selectors

SelectorExample
--last <duration>--last 1h, --last 1d
--start <ts> --end <ts>--start "2026-04-22 08:00:00"
--info / --debugInclude lower-priority entries
--style syslogCompact, grep-friendly

Decision Flow

Did `last reboot` advance near time T?
├─ YES → Kernel-level event
│   ├─ *.panic file present? → Panic; read backtrace
│   └─ No panic file → Clean restart (user-initiated or watchdog)
└─ NO → Userspace event
    ├─ *.hang or *.spindump.txt near T? → UI thread hang
    ├─ *.cpu_resource.diag spike near T? → Daemon CPU storm
    ├─ JetsamEvent-* near T? → Memory-pressure kill
    └─ None of the above → Power loss or hard power-cycle

Error Handling

SymptomCauseFix
find: ...DiagnosticReports: Permission deniedSome user-level reports require sudoStick to system-wide; don't sudo unless necessary
last reboot emptywtmp rotated past the incidentUse log show --predicate 'event == "boot"' instead
log show very slow / huge outputDefault predicate is too broadNarrow with --predicate and tighter time range
Reports only go back a few daysApple rotates the diag dir aggressivelyCheck ~/Library/Logs/DiagnosticReports/ for backups; some events only persist as log show entries
Filenames with .ips not .crashModern macOS format changeTreat both as equivalent; same parser tools work

Related Skills

  • launchservices-health — when the timeline points at launchservicesd, dig deeper there
  • kitty-session-persistence — recover terminal state lost during the incident
发现
标签

此技能尚未发布标签。

版本
最新版本元数据

版本

v2026.09.24

发布时间

2026年9月24日

分类

未分类

许可证

MIT

源路径

macos-plugin/skills/macos-incident-postmortem

默认分支

main

最新提交

1668324

Tree SHA

b2d4cc3