Inside the OOM Killer: How Linux Decides What to Kill When Memory Runs Out

When Linux genuinely runs out of memory — no more can be allocated, and no more can be reclaimed by writing back dirty pages or evicting cache — the kernel doesn’t just fail the next allocation request and let the requesting process handle the error gracefully. Historically, the design choice was different: the kernel selects an existing process and kills it outright, on the theory that failing an allocation and hoping every piece of software handles that failure gracefully is less reliable than the kernel forcibly reclaiming memory by terminating something. This is the OOM (out-of-memory) killer, and its victim-selection logic is worth understanding because it frequently surprises people.

Why overcommit makes this worse than it sounds

Linux’s memory overcommit behavior is a significant part of why the OOM killer exists at all in its current form. By default, Linux allows processes to allocate more virtual memory than the system actually has available, on the assumption that most allocated memory won’t actually be touched (a process might malloc() a large buffer and only ever use a fraction of it). This works fine until enough processes actually do touch enough of what they allocated that physical memory demand exceeds what’s available — at which point the kernel is in a position where memory it already promised to give out can’t actually be backed, and something has to give.

vm.overcommit_memory controls this behavior (0 for the default heuristic-based overcommit, 1 for always allowing overcommit, 2 for stricter accounting that refuses allocations beyond a configured limit rather than allowing indefinite overcommit) — and it’s directly relevant to OOM killer behavior, since a stricter overcommit setting means allocation failures happen earlier and more predictably (individual malloc() calls failing, which well-behaved software can catch) rather than deferring the problem to a later point where the OOM killer has to intervene forcibly.

The badness score: how a victim is actually selected

When the OOM killer activates, it computes a “badness” score (formally oom_score, visible per-process) for every candidate process, and kills whichever process has the highest score. The core factor driving this score is straightforward: how much memory the process is actually using, as a percentage of total available memory — a process using a larger share of memory is considered a more impactful, and therefore more “attractive,” target for reclaiming memory quickly. But this is not the only factor, and the adjustments matter enormously for understanding real-world OOM killer behavior.

Root-owned processes historically received a scoring adjustment making them somewhat less likely to be selected (on the theory that many root processes are core system infrastructure whose loss would be more disruptive than losing an ordinary user process), and per-process oom_score_adj lets an administrator or the process itself bias this scoring directly:

cat /proc/<pid>/oom_score
echo -500 /proc/<pid>/oom_score_adj

Setting oom_score_adj to a negative value makes a process less likely to be selected as a victim (down to -1000, which effectively exempts it from OOM killing entirely); a positive value makes it more likely. This is exactly the mechanism system services use to protect genuinely critical processes (an SSH daemon, for instance — losing remote access during a memory crisis is often worse than losing whatever memory-hungry process actually caused it) from being killed just because they happen to be running when memory runs out.

Why the killed process often “wasn’t the problem”

This is the single most common source of confusion around OOM killer behavior: the process it kills is whichever one scores highest on its badness metric right now, which is not necessarily the process that caused memory pressure to build up in the first place. A memory leak in one long-running service can gradually consume memory over hours, and by the time the OOM killer actually triggers, an entirely unrelated, large, but well-behaved process (a database, a build process, a browser) might have the highest badness score simply because it’s using more memory at that instant — and it gets killed, while the actual leaking process, if it’s smaller or has favorable oom_score_adj tuning, survives.

This is why “the OOM killer killed my important service” investigations should never stop at “why did it pick this process” without also asking “what actually drove the system into an OOM condition in the first place” — the victim and the cause are frequently different processes entirely, and only checking dmesg/kernel logs around the OOM event (which log memory usage for every process at the time of the kill, not just the victim) reveals the actual memory-pressure culprit:

dmesg | grep -A 30 "Out of memory"

cgroup-scoped OOM: a more targeted mechanism

Modern Linux also supports OOM killing scoped to a cgroup (control group) rather than only system-wide — a container or systemd service confined to a cgroup with a memory limit can have its own, contained OOM event that only considers and kills processes within that same cgroup, rather than the system-wide OOM killer potentially reaching outside the cgroup that actually caused the pressure. This is a meaningfully better isolation model for containerized workloads specifically: a single misbehaving container hitting its own memory limit gets its own processes killed, rather than risking the system-wide OOM killer selecting a victim process belonging to a completely different, well-behaved container running on the same host.

The honest tradeoff

The OOM killer is a genuinely blunt instrument dealing with a genuinely hard problem: there’s no universally correct answer to “which process should die to save the system” when memory runs out, and the badness-score heuristic is a best-effort approximation, not a guarantee of picking the “right” victim by whatever definition of right a specific administrator has in mind. oom_score_adj tuning for genuinely critical processes, combined with cgroup-scoped memory limits to contain the blast radius of any one workload’s memory pressure, is the practical way to make OOM killer behavior more predictable — but the underlying mechanism, killing something to reclaim memory forcibly, remains a last resort dealing with a situation that ideally should have been caught earlier via memory limits and monitoring, not relied upon as the primary defense against memory exhaustion.