kubectl debug - Interactive Kubernetes Debugging

Expert knowledge for debugging Kubernetes resources using kubectl debug - ephemeral containers, pod copies, and node access.

When to Use This Skill

Use this skill when...	Use <sibling> instead when...
Attaching an ephemeral debug container to a running pod with `kubectl debug`	Use kubernetes-operations for general kubectl workflows (apply, get, describe, logs)
Creating a pod copy with a different image or command for interactive troubleshooting	Use helm-debugging when the failure is in template rendering or chart configuration, not the running container
Opening a node-level debug session to inspect host namespaces or filesystems	Use helm-release-recovery when the recovery action is a Helm rollback rather than per-pod debugging

Core Capabilities

kubectl debug automates common debugging tasks:

Ephemeral Containers: Add debug containers to running pods without restart
Pod Copying: Create modified copies for debugging (different images, commands)
Node Debugging: Access node host namespaces and filesystem

Context Safety (CRITICAL)

Always specify --context explicitly in every kubectl command:

# CORRECT: Explicit context
kubectl --context=prod-cluster debug mypod -it --image=busybox

# WRONG: Relying on current context
kubectl debug mypod -it --image=busybox  # Which cluster?

Quick Reference

Add Ephemeral Debug Container

# Interactive debugging with busybox
kubectl --context=my-context debug mypod -it --image=busybox

# Target specific container's process namespace
kubectl --context=my-context debug mypod -it --image=busybox --target=mycontainer

# Use a specific debug profile
kubectl --context=my-context debug mypod -it --image=busybox --profile=netadmin

Copy Pod for Debugging

# Create debug copy
kubectl --context=my-context debug mypod -it --copy-to=mypod-debug --image=busybox

# Copy and change container image
kubectl --context=my-context debug mypod --copy-to=mypod-debug --set-image=app=busybox

# Copy and modify command
kubectl --context=my-context debug mypod -it --copy-to=mypod-debug --container=myapp -- sh

# Copy on same node
kubectl --context=my-context debug mypod -it --copy-to=mypod-debug --same-node --image=busybox

Debug Node

# Interactive node debugging (host namespaces, filesystem at /host)
kubectl --context=my-context debug node/mynode -it --image=busybox

# With sysadmin profile for full capabilities
kubectl --context=my-context debug node/mynode -it --image=ubuntu --profile=sysadmin

Debug Profiles

Profile	Use Case	Capabilities
`legacy`	Default, unrestricted	Full access (backwards compatible)
`general`	General purpose	Moderate restrictions
`baseline`	Minimal restrictions	Pod security baseline
`netadmin`	Network troubleshooting	NET_ADMIN capability
`restricted`	High security environments	Strictest restrictions
`sysadmin`	System administration	SYS_PTRACE, SYS_ADMIN

# Network debugging (tcpdump, netstat, ss)
kubectl --context=my-context debug mypod -it --image=nicolaka/netshoot --profile=netadmin

# System debugging (strace, perf)
kubectl --context=my-context debug mypod -it --image=ubuntu --profile=sysadmin

Common Debug Images

Image	Size	Use Case
`busybox`	~1MB	Basic shell, common utilities
`alpine`	~5MB	Shell with apk package manager
`ubuntu`	~77MB	Full Linux with apt
`nicolaka/netshoot`	~350MB	Network debugging (tcpdump, dig, curl, netstat)
`gcr.io/k8s-debug/debug`	Varies	Official Kubernetes debug image

Debugging Patterns

Network Connectivity Issues

# Add netshoot container for network debugging
kubectl --context=my-context debug mypod -it \
  --image=nicolaka/netshoot \
  --profile=netadmin

# Inside container:
# - tcpdump -i any port 80
# - dig kubernetes.default
# - curl -v http://service:port
# - ss -tlnp
# - netstat -an

Application Crashes

# Copy pod with different entrypoint to inspect
kubectl --context=my-context debug mypod -it \
  --copy-to=mypod-debug \
  --container=app \
  -- sh

# Inside: check filesystem, env vars, config files

Process Inspection

# Target container's process namespace
kubectl --context=my-context debug mypod -it \
  --image=busybox \
  --target=mycontainer

# Inside: ps aux, /proc inspection

Node-Level Issues

# Debug node with host access
kubectl --context=my-context debug node/worker-1 -it \
  --image=ubuntu \
  --profile=sysadmin

# Inside:
# - Host filesystem at /host
# - chroot /host for full access
# - journalctl, systemctl, dmesg

Non-Destructive Debugging

# Create copy, keeping original running
kubectl --context=my-context debug mypod -it \
  --copy-to=mypod-debug \
  --same-node \
  --share-processes \
  --image=busybox

# Original pod continues serving traffic
# Debug copy shares storage if on same node

Key Options

Option	Description
`-it`	Interactive TTY (required for shell access)
`--image`	Debug container image
`--container`	Name for the debug container
`--target`	Share process namespace with this container
`--copy-to`	Create a copy instead of ephemeral container
`--same-node`	Schedule copy on same node (with `--copy-to`)
`--set-image`	Change container images in copy
`--profile`	Security profile (legacy, netadmin, sysadmin, etc.)
`--share-processes`	Enable process namespace sharing (default: true with --copy-to)
`--replace`	Delete original pod when creating copy

Best Practices

Use appropriate profiles - Match capabilities to debugging needs
Prefer ephemeral containers - Less disruptive than pod copies
Use --copy-to for invasive debugging - Preserve original pod
Clean up debug pods - Delete copies after debugging
Use --same-node - For accessing shared storage/network conditions

Cleanup

# List debug pod copies
kubectl --context=my-context get pods | grep -E "debug|copy"

# Delete debug pods
kubectl --context=my-context delete pod mypod-debug

Requirements

Kubernetes 1.23+ for ephemeral containers (stable)
Kubernetes 1.25+ for debug profiles
RBAC permissions for pods/ephemeralcontainers

For detailed option reference, examples, and troubleshooting patterns, see REFERENCE.md.

kubectl-debugging