Skip to content
LinuxDeep Dive Published Updated 5 min readViews unavailable

Linux membarrier(): Process-Scoped Ordering for Runtimes and JITs

Use Linux membarrier() as a deliberate slow-path synchronization primitive, with capability probing, registration, ordering contracts, and runtime limits.

membarrier() is a Linux system call for runtimes that need a memory-ordering operation to affect more than the calling thread. It can be useful when an application has a deliberately asymmetric protocol: a frequent fast path uses a compiler barrier, while a much rarer slow path asks the kernel to make participating threads pass through an ordering point. The optimization is subtle. A system call is more expensive than an ordinary CPU memory-barrier instruction, and correctness depends on the matching operations on both sides of the protocol.

This is not a general replacement for mutexes, language atomics, or a correct C/C++ happens-before relationship. It is a Linux-specific primitive for a runtime or library that owns and can prove its complete synchronization protocol. RCU libraries and garbage collectors are examples documented by the Linux man page.

Discover the ABI instead of guessing from a kernel version

glibc does not provide a wrapper named membarrier(), so user space calls it through syscall(). The command set is kernel-dependent. MEMBARRIER_CMD_QUERY returns a bit mask of supported commands; the query command itself is not present in that mask. Probe the command that the runtime plans to use rather than assuming that a particular distribution release, architecture, or kernel configuration provides it.

For the process-private expedited operation, Linux documents support since kernel 4.14 and requires the process to register its intent before use. A minimal startup probe is:

#define _GNU_SOURCE
#include <errno.h>
#include <linux/membarrier.h>
#include <stdio.h>
#include <sys/syscall.h>
#include <unistd.h>

static long
query_membarrier(void)
{
    return syscall(SYS_membarrier, MEMBARRIER_CMD_QUERY, 0, 0);
}

static int
register_private_expedited(void)
{
    long supported = query_membarrier();

    if (supported == -1) {
        perror("membarrier query");
        return -1;
    }
    if ((supported & MEMBARRIER_CMD_PRIVATE_EXPEDITED) == 0 ||
        (supported & MEMBARRIER_CMD_REGISTER_PRIVATE_EXPEDITED) == 0) {
        fprintf(stderr, "private expedited membarrier is unsupported\n");
        return -1;
    }
    if (syscall(SYS_membarrier,
                MEMBARRIER_CMD_REGISTER_PRIVATE_EXPEDITED, 0, 0) == -1) {
        perror("membarrier registration");
        return -1;
    }
    return 0;
}

This only establishes that the command is available and registered. It does not prove the application’s protocol is correct. SYS_membarrier, the command constants, and the syscall ABI must also be available in the target build headers. If compiling against old headers, fail the build or provide a deliberately versioned compatibility layer; do not silently substitute an unrelated fence.

What the private expedited command actually orders

After registration, MEMBARRIER_CMD_PRIVATE_EXPEDITED asks the kernel to execute the memory-barrier operation on each running thread belonging to the same process. When it returns, the calling thread has the documented guarantee that running sibling threads have passed through a state where their user-space memory accesses match program order across the call. Threads that are not running are described as de facto being in such a state. The scope is process-private, not every process on the machine.

The operation is useful only as one half of a designed pairing. The documented optimization pattern replaces selected barriers on a heavily used fast side with a compiler barrier and places membarrier() on the uncommon slow side. If the fast side does not perform the matching operation, if a reader observes state outside the protocol, or if a language-level data race is introduced, the syscall does not repair that design. Keep publication state, participating threads, and the exact ordering proof explicit in code review.

The GLOBAL family has a different scope: global commands target threads from all processes, and expedited global commands concern processes that registered. Do not choose a global command as a shortcut for a process-local runtime. Expedited commands avoid blocking but add overhead; measure the real workload and call frequency on the architectures that matter.

Instruction-stream synchronization is a separate command

Linux also documents MEMBARRIER_CMD_PRIVATE_EXPEDITED_SYNC_CORE, available since Linux 4.16, for a stronger case: in addition to the private expedited memory-ordering guarantee, running sibling threads have executed a core-serializing instruction when the call returns. This can matter to runtimes changing executable code, but it does not allocate executable memory, coordinate a patching transaction, enforce write-xor-execute, or make unsafe concurrent code replacement valid. Probe and register its corresponding command separately, and follow the architecture and runtime’s full code-publication protocol.

Do not confuse this with an instruction-cache flush API or assume identical implementation support on every architecture. The Linux man page documents EINVAL when the sync-core operation is not implemented by the architecture. Treat a failed capability or registration probe as a normal fallback path.

Failure handling and performance checks

Useful first checks are small and read-only:

uname -r
grep -E 'CONFIG_(NO_HZ_FULL|SMP)=' /boot/config-$(uname -r) 2>/dev/null

uname records the running kernel release; a distribution may not install a matching config file at that path, so a missing result does not prove a config option is disabled. The authoritative capability test remains MEMBARRIER_CMD_QUERY from the target process. When a syscall fails, log the command, flags, errno, architecture, and kernel release. ENOSYS means the syscall is not implemented by that kernel; EPERM indicates an unregistered private expedited use; EINVAL can mean an invalid or unsupported command/argument, including an architecture without the sync-core command.

Benchmark the whole protocol rather than timing the syscall in isolation. membarrier() is heavier than a CPU memory barrier, so the technique only helps when its uncommon slow path is sufficiently rare to outweigh its cost. Test thread creation and shutdown paths, fallback behavior, and the no-support case. If an ordinary mutex or standard atomic protocol is fast enough, prefer the simpler, language-defined synchronization mechanism.

Related:

Sources:

Comments