Linux futex_waitv: Multi-Address Waiting, Wake Indices, and Safe Retries

futex_waitv() is Linux’s wait-any operation for a bounded array of futex words. Instead of choosing one address and sleeping on it, a thread describes multiple addresses and expected values; the kernel can return when one listed futex is woken. This is useful when one worker can make progress because any of several independent conditions changed, for example when work arrived on one queue or a shutdown flag changed, without creating a separate polling thread for each condition.

Implementation-specific behavior in this article was reviewed against Linux v7.3-rc7 (commit 24a19b953e2a3d0ea5a8c4951a0ab8f32720c907); a later kernel may change internals while preserving the documented ABI.

It is not a general event queue, a replacement for condition variables, or an atomic read of an application’s whole data structure. The vector contains synchronization words, not payloads. Correctness still depends on the caller’s atomic state transitions, memory ordering, object lifetime, and a loop that re-evaluates the real predicate after every return. This article focuses on the newer futex_waitv() ABI; the traditional single-address compare-and-block protocol is covered separately in Linux Futexes in Practice.

The vector describes conditions, not work items

The Linux kernel documentation defines each non-NUMA struct futex_waitv entry with an expected value (val), a userspace address (uaddr), per-entry flags, and a reserved field. This article uses that non-NUMA form and does not set FUTEX2_NUMA; the NUMA extension changes the address layout and is outside this example. The vector is limited to 1–128 entries. Set __reserved to zero; the syscall-level flags parameter must also be zero in the reviewed ABI. Each entry’s flags describe the futex size and sharing mode. The supported size is a 32-bit futex selected with FUTEX2_SIZE_U32; do not infer a 64-bit futex from the structure’s 64-bit address and value fields. The UAPI header retains FUTEX_32 as a historical alias marked “do not use.” It also fixes the structure layout and FUTEX_WAITV_MAX limit for the syscall ABI. (kernel API documentation, UAPI header)

For a process-local word, combine FUTEX2_SIZE_U32 with FUTEX_PRIVATE_FLAG. For a word shared across processes, omit the private flag and put the word in memory both processes actually share. A private futex is keyed under the assumption that the participating threads share one address space; two processes mapping the same backing object at different virtual addresses need the shared form. The ordinary word must remain correctly aligned and mapped for every waiter and waker.

Field or argument	Contract to preserve
`uaddr` (non-NUMA form used here)	Address of the 32-bit futex word; it must remain valid while the syscall accesses it.
`val`	Value that the entry expects at that address before the wait can sleep.
`flags` in each entry	Include `FUTEX2_SIZE_U32`; add the private flag only for threads in the same process.
`__reserved`	Initialize to zero.
syscall `flags`	Pass zero with the current ABI.
`nr_futexes`	Number of entries, from 1 through 128.

The kernel compares each listed word with its expected value while preparing the wait. The documented mismatch result is EAGAIN rather than sleeping on a stale snapshot; in the reviewed implementation, however, an entry already observed as woken during that setup takes precedence and its index is returned. This closes the ordinary check-then-sleep window when the value has changed and remains different at the kernel’s comparison; it is not an event-history check. The ABI compares the current word with val, so it cannot detect an ABA sequence in which a producer changes the word away from the expected value and back before that comparison. When every notification must be observed, use a generation or sequence value with a wraparound policy that prevents reuse during the relevant waiter lifetime, or use a durable predicate, counter, or queue that records pending work. Then re-read the protected predicates after waking. The wait-vector comparison does not publish unrelated application data or make several words an atomic snapshot. (futex_waitv API contract, kernel wait setup and equality check)

A return index is only a hint about which wait entry woke

On a wake, the syscall returns the index of a futex entry that was woken. The return is not a count of work, an ownership transfer, or proof that the associated application condition is still true. If several words are woken, the selected index is not guaranteed to be the lowest index or the most recently signaled one. A producer may signal the word and another thread may consume the work before this waiter runs. Therefore use the index to prioritize what to inspect, not to skip the predicate check.

An event loop for a worker might be structured as follows. This is ABI-oriented pseudocode for the raw five-argument syscall: syscall wrappers and syscall-number macros vary across libc versions and architectures, so compile against the target system’s UAPI headers and wrapper rather than hard-coding a number. Because this example calls the raw syscall, it uses the fixed-layout struct __kernel_timespec from the Linux UAPI. Fill its 64-bit seconds and nanoseconds fields with an absolute deadline from the same clock passed to the syscall; do not pass a 32-bit userspace struct timespec directly. (Linux UAPI time types)

/* Compute once from CLOCK_MONOTONIC plus the desired interval. */
struct __kernel_timespec absolute_deadline = {
    .tv_sec = deadline_sec,
    .tv_nsec = deadline_nsec,
};

for (;;) {
    uint32_t work = atomic_load_explicit(&work_state, memory_order_acquire);
    uint32_t stop = atomic_load_explicit(&stop_state, memory_order_acquire);

    if (stop != 0)
        break;
    if (work != 0) {
        consume_available_work();
        continue;
    }

    struct futex_waitv waiters[2] = {
        { .val = 0, .uaddr = (uintptr_t)&work_state,
          .flags = FUTEX2_SIZE_U32 | FUTEX_PRIVATE_FLAG, .__reserved = 0 },
        { .val = 0, .uaddr = (uintptr_t)&stop_state,
          .flags = FUTEX2_SIZE_U32 | FUTEX_PRIVATE_FLAG, .__reserved = 0 },
    };

    long selected = syscall(
        SYS_futex_waitv,
        waiters,                 /* waiters */
        2,                        /* nr_futexes */
        0,                        /* syscall flags */
        &absolute_deadline,       /* absolute time64 timeout */
        CLOCK_MONOTONIC);         /* clockid */
    if (selected >= 0) {
        /* selected is an entry index; loop and reload both predicates. */
        continue;
    }
    /* syscall() returns -1 and sets errno on failure. */
    if (selected < 0) {
        if (errno == EAGAIN || errno == EINTR)
            continue;         /* re-check state; apply signal policy as needed */
        if (errno == ETIMEDOUT) {
            handle_deadline();
            break;            /* leave this wait loop if the handler returns */
        } else {
            handle_syscall_error(errno);
            break;            /* propagate/finish instead of retrying the same error */
        }
    }
}

The timeout and error handlers in this pseudocode may return. The loop therefore exits after either call; it never immediately reuses an already-expired absolute deadline or retries a persistent error. A real worker can instead return an error or explicitly compute a fresh deadline before rearming.

The producer must update the corresponding state word with the synchronization ordering the data protocol requires, then call the compatible futex wake operation on that same word. A wake without a state transition is merely a prompt to recheck; a state transition without the appropriate wake can leave a sleeping consumer parked. Keep both operations coupled in the higher-level protocol. Do not assume that placing two words in one vector gives the application an indivisible snapshot of both words or a transaction over their associated queues.

Timeouts are absolute and clock-specific

The timeout pointer is optional and, when supplied, refers to an absolute deadline. Choose CLOCK_MONOTONIC for elapsed-time limits that should not move when wall-clock time is adjusted; use CLOCK_REALTIME only when the application deliberately wants a wall-clock deadline. The kernel documentation lists these two clock IDs and specifies a 64-bit timespec ABI. On a 32-bit userspace ABI, use the correct 64-bit kernel timespec representation rather than assuming the platform’s ordinary struct timespec has the required layout. A relative duration must first be converted to an absolute deadline using the same clock ID.

The vector operation has distinct outcomes: a returned nonnegative index identifies a woken entry; a value mismatch can return EAGAIN; a signal interrupts the wait; and an expired deadline is a timeout. The current kernel API prose labels that result -ETIMEOUT, while the reviewed Linux wait implementation returns -ETIMEDOUT; through the usual syscall() wrapper, callers handle the latter as errno == ETIMEDOUT. Keep the wrapper’s return convention separate from the raw kernel convention, and keep timeout, interruption, and wake handling separate so that a deadline does not accidentally become an infinite retry loop. (wait-vector implementation)

Compatibility and failure-oriented tests

futex_waitv() is an optional Linux syscall, not a portable POSIX function. Header availability does not establish that the running kernel implements it, and the kernel self-test explicitly skips the wait-vector tests when the syscall is unsupported. Treat it as a capability: invoke the raw syscall through the target libc’s syscall() interface, handle ENOSYS, and retain a fallback such as a condition variable or a design with fewer independent wait sources. Do not use an architecture-specific syscall number copied from another build machine. The kernel’s own self-test provides a useful starting point for private and shared vectors, wake-index selection, invalid flags, null and unaligned addresses, and an invalid clock ID. It does not test value-mismatch handling, so add a separate case with an expected value that differs from the mapped word. (kernel futex self-test)

Test the transition windows, not just the happy path: change a word before the waiter calls the syscall, change it while the vector is being prepared, wake each vector index independently, wake multiple entries, signal the blocked thread, expire the deadline, and unmap or retire the containing object only after all waiters have exited. For shared mappings, run a producer and waiter in separate processes and verify that both use the shared futex key. Use stress tests and race detectors for the userspace state machine, while remembering that a sanitizer cannot prove kernel ABI behavior.

Use futex_waitv() when the state is naturally represented by a small set of 32-bit futex words and avoiding multiple blocking threads materially simplifies the design. Use epoll for file-descriptor readiness instead: a futex word is not a descriptor and carries no event payload. The narrow ABI is powerful precisely when the application keeps that distinction explicit.

Related:

Sources:

The vector describes conditions, not work items

A return index is only a hint about which wait entry woke

Timeouts are absolute and clock-specific

Compatibility and failure-oriented tests

Comments