How to Debug a Program on Linux with strace

When a program fails with no useful error message, strace shows you exactly what it’s actually doing at the system-call level — every file it tries to open, every socket it touches, every signal it receives — often revealing the real cause in minutes when application logs offer nothing.

Step 1: install strace, if it isn’t already present

sudo apt install strace      # Debian/Ubuntu
sudo dnf install strace      # Fedora/RHEL

Step 2: run a command directly under strace

strace ./myprogram

This prints every system call the program makes, in real time, as it runs — a firehose for anything non-trivial, but immediately useful for simple cases.

Step 3: attach to an already-running process

strace -p <pid>

Useful for a program that’s already misbehaving and you don’t want to (or can’t) restart — attaching live shows you what it’s doing from this point forward.

Step 4: filter to a specific category of system calls

strace -e trace=open,openat,read,write ./myprogram
strace -e trace=network ./myprogram

Filtering to just file operations, or just network calls, cuts through the noise when you already have a hypothesis about which category of syscall is relevant.

Step 5: find exactly which file a program can’t find or open

strace -e trace=openat ./myprogram 2>&1 | grep ENOENT

A program failing with a vague “file not found” error, but not telling you which file, is one of strace’s most common and satisfying uses — the syscall trace shows the exact path it tried and failed to open.

Step 6: measure time spent in each system call

strace -T ./myprogram

The -T flag appends the time each system call took — useful for spotting a specific syscall (often a network call or a slow disk read) responsible for an otherwise-unexplained pause.

Step 7: get a summary of syscall frequency and total time

strace -c ./myprogram

Rather than a full line-by-line trace, -c produces a summary table — call counts and total time per syscall — useful for spotting a syscall being called suspiciously often (a classic sign of a retry loop or inefficient polling) without reading a huge trace manually.

Step 8: follow child processes too, if the program forks

strace -f ./myprogram

Without -f, strace only follows the initial process — for anything that forks or execs child processes, -f is necessary to see what those children are actually doing as well.

Step 9: save a large trace to a file rather than reading it live

strace -o trace.log -f ./myprogram

Why strace works even when a program has no logging of its own

strace operates entirely at the kernel/syscall boundary, using ptrace(2) to intercept every system call a traced process makes — this means it works identically regardless of whether the program itself has any logging, error handling, or debug output implemented at all. For a genuinely opaque binary, or a program whose own error messages are misleading or absent, strace is often the only way to see what it’s actually trying to do versus what it’s succeeding at.

Step 10: reach for ltrace when the problem is a library call, not a syscall

ltrace ./myprogram

strace shows kernel system calls specifically — it has nothing to say about calls a program makes into shared libraries (malloc, strlen, a specific libcurl function) that never cross into kernel territory at all. ltrace traces exactly this complementary layer: dynamic library function calls, with their arguments and return values, which is the more useful tool when you already know the problem is happening inside library code (an unexpected argument being passed into a known library function, for instance) rather than at the syscall boundary strace covers. The two tools are frequently used together — strace for “what is this program asking the kernel to do,” ltrace for “what is this program asking its linked libraries to do” — since a single trace covering both would be considerably noisier and harder to read than using the right one for the specific question at hand.

Step 11: understand strace’s real performance cost before tracing something latency-sensitive

Because ptrace intercepts and pauses execution at every single traced syscall, a heavily syscall-bound program can run dramatically slower under strace — sometimes an order of magnitude or more — which matters enormously for diagnosing a timing-sensitive bug (a race condition, a timeout) where slowing the program down changes or hides the very behavior you’re trying to observe. For genuinely performance-sensitive tracing, perf trace (built on the kernel’s lower-overhead tracepoint infrastructure rather than ptrace) or targeted eBPF-based tools cost meaningfully less overhead, though at some loss of strace’s simplicity and universal availability — worth reaching for specifically when strace’s own overhead risks masking the exact problem you’re chasing.

Related:

Sources: