Skip to content
LinuxDeep Dive Published Updated 6 min readViews unavailable

Linux process_mrelease(): Reclaim Memory from a Dying Process

Use process_mrelease() only with a pidfd after exit begins; interpret partial-reclaim errors carefully and keep victim policy, retries, and normal reaping separate.

process_mrelease() is a narrow Linux memory-management system call for one point in a process supervisor’s lifecycle: a target has already been selected for termination, but its address space may not yet have been torn down. The caller can ask the kernel to reap that dying task’s eligible mappings rather than waiting for every detail of its exit path to finish first.

This is not a way to evict memory from a healthy process, a userspace OOM policy, or a command to kill a process. The supervisor still decides whether termination is justified and sends the fatal signal through its normal policy. process_mrelease() is an optional follow-up intended to make memory pressure recover sooner and more predictably. It neither promises an exact number of bytes nor turns every mapping into reclaimable memory.

The target must already be on its way out

The syscall accepts a PID file descriptor and a flags word. The fd gives the request a stable reference to the process selected by the supervisor; it avoids performing a second lookup using a numeric PID after unrelated work has taken place. The descriptor is not a substitute for safe target selection: if a manager starts from a PID it learned earlier, it should still establish that the process it opened is the intended victim. For a child it creates, retaining a pidfd from creation avoids that lookup window entirely.

Eligibility depends on kernel exit state, not merely on the supervisor having chosen a victim or sent it a signal. The kernel checks that the target has entered an exiting state and still has an address space to reap; a core dump in progress is not eligible. If another process shares that same mm, it too must be dying or exiting. A target that has not entered the required exit state, or a live process in another thread group sharing the mm, makes the address space ineligible. Because eligibility can race with exit teardown, handle the syscall result while the pidfd still refers to the target.

The operation works on the target’s address space, not on a caller-supplied byte range. Its implementation checks that the task will free memory, takes the mmap read lock, and invokes the kernel’s OOM-reaper path. That distinction matters when a service has multiple threads or another process shares the same memory map: reaping is safe only when the address space is in a suitable exit state. The syscall does not let a caller reach into a still-running application’s mappings.

Call through the syscall interface and keep policy separate

There is no portable C library wrapper to assume for this Linux-specific interface. On systems whose build headers define SYS_process_mrelease, a small helper can invoke it through syscall():

#define _GNU_SOURCE
#include <errno.h>
#include <stdio.h>
#include <sys/syscall.h>
#include <unistd.h>

static int release_dying_process(int pidfd)
{
    long result = syscall(SYS_process_mrelease, pidfd, 0U);

    if (result == -1) {
        perror("process_mrelease");
        return -1;
    }
    return 0;
}

The supervisor should call this only after its own policy selected the victim and initiated termination. Keep the pidfd open through the attempt, then close it on every path. In a production manager, compile against target ABI headers that declare the syscall number, record the kernel and errno with the attempt, and treat ENOSYS as a runtime capability failure rather than silently substituting a different operation. Distribution kernels may backport features, so a release string alone is not a reliable probe.

The flags argument is reserved and must be zero. Passing any other value is an invalid request. For this API, the return value is a status, not a reclaimed-byte count. If a caller needs to know whether the system actually recovered pressure, it must observe the relevant memory accounting separately and account for concurrent allocations and reclaim.

Interpret failure as lifecycle information

The most useful outcomes are:

  • EBADF: the descriptor is not a valid pidfd.
  • EAGAIN: the kernel could not reap part of the address space in this attempt. The process may still be exiting, so a supervisor can consider a bounded retry with backoff while it still owns the pidfd. This is not a guarantee that a later attempt will succeed.
  • EINVAL: flags are nonzero, the target has not entered the required exiting state, a core dump is in progress, or another live process shares the same mm. All other processes sharing that mm must also be dying or exiting; do not spin against a live target.
  • ESRCH: the pidfd no longer resolves to a task, or no thread in the target group still has an mm (which can happen during exit teardown). Record that race as a lifecycle outcome, not as permission to look up the numeric PID again.
  • EINTR: the caller’s operation was interrupted. Apply the service’s cancellation and retry rules rather than treating interruption as a memory result.
  • ENOSYS: the running kernel does not implement this syscall, including configurations without the required MMU support. Continue with the normal termination-and-wait path.

In the pinned implementation, MMF_OOM_SKIP can mean that the OOM reaper already finished, could not safely complete its work after bounded attempts, or that exit teardown has made further reaping unnecessary. When this flag is already set, process_mrelease() can return zero without performing a new reclaim pass. Treat this as implementation behavior, not an ABI guarantee or proof that this invocation released memory. Separate “call returned zero” from “this call released N bytes”: the interface reports status, not a byte count. Keep normal waitid()/waitpid() handling, child reaping, service restart, and accounting intact; this syscall does not replace any of them.

Where it fits beside pidfds and cgroup controls

The interfaces solve different problems. A pidfd preserves process identity and is the handle passed to process_mrelease(). The reaper syscall asks the kernel to release memory from a task that is already dying. A cgroup limit or memory.reclaim request operates on a workload boundary and is useful for proactive pressure management; it does not make a live process eligible for process_mrelease().

That split suggests a conservative design. First measure pressure and choose a victim according to explicit policy. Then signal the selected task through its already-held pidfd. Attempt process_mrelease() once, optionally retry only the documented partial-progress case with a short bounded policy, and continue normal exit monitoring. If the call is unavailable or loses the race with process exit, the ordinary kernel exit path remains the fallback. Never turn a failure into a reason to kill a second process without reevaluating the pressure and victim policy.

For tests, use a disposable child that allocates memory, retain its pidfd, and exercise the syscall only after arranging an exit condition. Include the live-process negative case, invalid flags, unsupported-kernel behavior, and the race where the child exits before the attempt. Run memory-pressure experiments in a constrained test environment, not against a production process selected only by a stale PID. A useful metric set is the victim’s state, pidfd identity, request time, syscall result, wait status, and cgroup or system memory counters before and after; do not infer a precise reclaim amount from the syscall return alone.

Related:

Sources:

Comments