strace / ltrace

Purpose

Guide agents through tracing system calls with strace and library calls with ltrace — the most effective tools for diagnosing incorrect binary behaviour without a crash or debugger.

Triggers

• "My program behaves incorrectly — how do I trace what it's doing?"
• "How do I find what files a binary is opening?"
• "strace shows ENOENT — how do I interpret it?"
• "How do I trace network calls with strace?"
• "What is ltrace and how does it differ from strace?"
• "How do I trace a running process?"

Workflow

1. Basic strace usage

# Trace all syscalls of a command
strace ./myapp arg1 arg2

# Attach to running process
strace -p 12345

# Trace child processes too (-f = follow fork)
strace -f ./myapp

# Save to file (raw output — not stdout)
strace ./myapp 2> trace.txt

# Most useful: timestamps + summary
strace -t -f ./myapp 2>&1 | head -100

2. Filter by syscall category

# Trace file operations only
strace -e trace=file ./myapp

# Trace network syscalls
strace -e trace=network ./myapp

# Trace specific syscalls
strace -e trace=open,openat,read,write ./myapp

# Trace process management
strace -e trace=process ./myapp

# Trace memory operations
strace -e trace=memory ./myapp

# Trace signals
strace -e trace=signal ./myapp

# Multiple categories
strace -e trace=file,network ./myapp

3. Interpreting common errors

# See return values and errors
strace -e trace=file ./myapp 2>&1 | grep -E "ENOENT|EPERM|EACCES|ENOTSUP"

# Find what file is not found
strace ./myapp 2>&1 | grep 'ENOENT'
# Example output:
# openat(AT_FDCWD, "/etc/myapp.conf", O_RDONLY) = -1 ENOENT (No such file or directory)
# → Config file expected at /etc/myapp.conf

4. Useful strace flags

# Show strings fully (default truncates at 32 chars)
strace -s 256 ./myapp

# Timestamps
strace -t ./myapp     # wall clock time
strace -T ./myapp     # time spent in each syscall
strace -r ./myapp     # relative timestamps

# System call count summary
strace -c ./myapp
# Shows count, time, errors per syscall — great for profiling

# Trace with PIDs in output (for -f)
strace -f -p ./myapp
# Output: [pid 12346] open("/etc/passwd", O_RDONLY) = 3

# Decode numerical arguments
strace -e verbose=all ./myapp

# Print instruction pointer at each syscall
strace -i ./myapp

5. ltrace — library call tracing

# Trace all library calls
ltrace ./myapp

# Trace specific library function
ltrace -e malloc,free,fopen ./myapp

# Trace nested calls (lib → lib)
ltrace -n 2 ./myapp   # indent nested calls

# Trace with syscalls too
ltrace -S ./myapp

# Attach to running process
ltrace -p 12345

# Summary statistics
ltrace -c ./myapp

Typical ltrace output:

malloc(1024) = 0x55a1b2c3d000
fopen("/etc/myapp.conf", "r") = 0
free(0x55a1b2c3d000) = <void>

strace vs ltrace:

6. Practical diagnosis workflows

# Find missing config file
strace -e trace=openat,open ./myapp 2>&1 | grep ENOENT

# Find what network connections are made
strace -e trace=network -f ./myapp 2>&1 | grep connect

# Debug dynamic library loading failures
strace -e trace=openat ./myapp 2>&1 | grep "\.so"

# Find permission issues
strace -e trace=file ./myapp 2>&1 | grep -E "EACCES|EPERM"

# Debug slow startup (find where time is spent)
strace -c ./myapp 2>&1
# Look for high % time in unexpected syscalls

# Watch IPC/shared memory
strace -e trace=ipc,shm ./myapp

# Find what the binary exec's
strace -e trace=execve -f ./myapp

7. seccomp filter debugging

If a program is killed by a seccomp policy, strace reveals which syscall triggered it:

strace -e trace=all ./myapp 2>&1 | tail -5
# Often shows the last syscall before SIGSYS

For strace output patterns and ltrace filtering examples, see references/strace-patterns.md.

Related skills

• Use skills/debuggers/gdb when strace shows the failing location and you need to inspect internals
• Use skills/binaries/elf-inspection to understand what libraries and symbols a binary uses
• Use skills/binaries/dynamic-linking for diagnosing LD_* and library loading issues
• Use skills/profilers/linux-perf for performance profiling (strace overhead is too high for perf)