Linux io_uring Provided Buffer Rings: Incremental Consumption and CQE Ownership

The existing io_uring overview mentions provided-buffer rings among many parts of the interface. This deep dive isolates one narrower contract: how a buffer descriptor moves between an application and the kernel, what incremental consumption changes, and which of two similarly named CQE flags describes the buffer versus the request.

A provided-buffer ring is a queue of descriptors, not a queue of payload bytes. Each struct io_uring_buf identifies a memory range by address, length, and a 16-bit buffer ID (bid). The data region is allocated separately. A buffer group ID selects the ring for a request; the buffer ID in a completion identifies the chosen descriptor within that group. Keep those two identifiers distinct in application state.

Ownership follows publication and completion

The application initializes a buffer ring, adds descriptors, and advances its tail to publish them. After publication, the kernel may select a descriptor for a read or receive submitted with IOSQE_BUFFER_SELECT and the matching buf_group. A completion with IORING_CQE_F_BUFFER carries the selected ID in the upper 16 bits of cqe->flags. Check that bit before extracting the ID; an error completion without a selected buffer has no usable buffer ID.

With an ordinary provided-buffer ring, selection consumes the whole descriptor even when the particular I/O transfers fewer bytes than its capacity. The completion returns the selected buffer to application ownership. The application processes the reported byte count, waits until any downstream consumer is done with the memory, then adds a descriptor back and advances the tail. It may associate a returned ID with a different address, but its own ID-to-memory bookkeeping must describe the mapping currently published.

Incremental rings change that lifecycle. When registered with IOU_PBUF_RING_INC, a selected large buffer can serve successive partial reads or receives. The kernel advances the usable address and reduces the remaining length as bytes are consumed; the application must track the offset for each active buffer ID as well. IORING_CQE_F_BUF_MORE means the selected buffer is still being used by the kernel and will be named by more completions. Do not recycle, overwrite, or free it while that flag is set. On the normal whole-buffer completion path, IORING_CQE_F_BUFFER without IORING_CQE_F_BUF_MORE means the descriptor has returned to application ownership and can be republished at its original base address and full capacity. Do not treat that flag combination alone as proof of whole-descriptor release for every unlocked, bounded-length receive; the early-commit exception is described below.

Treat ownership as a lease, not just an index. A completion makes its reported range available to the application, but BUF_MORE keeps the underlying allocation leased to the ring for later data. If processing is handed to another thread or asynchronous stage, retain the buffer until both the kernel has released it and the consumer has finished. Avoid giving a buffer back to the ring merely because one CQE was handled.

`F_MORE` describes a different lifetime

IORING_CQE_F_MORE says the parent SQE may produce additional CQEs. For a multishot receive, it indicates that the receive request remains armed. IORING_CQE_F_BUF_MORE instead says that this specific selected buffer ID has more completions associated with it. The first flag is per request; the second is per buffer. Neither substitutes for the other.

Both bits can be set on one CQE. In the pinned Linux 6.12 receive path, buffer-completion flags are collected first and then IORING_CQE_F_MORE is ORed in when the multishot request continues. The practical consequences are easy to miss:

F_MORE set, F_BUF_MORE clear: the multishot request continues, but this buffer has returned to the application and can be recycled after processing.
F_MORE clear, F_BUF_MORE set: this request has ended, but the buffer remains ring-owned; do not recycle it. A later request may continue using it.
Both set: the request continues and the current buffer remains ring-owned.
With F_BUFFER set and neither bit set: this selected buffer has been returned and this request has no further CQEs expected.

An -ENOBUFS completion is different again: if IORING_CQE_F_BUFFER is absent, there is no selected buffer to recycle. Replenish only buffers that the application actually owns, then rearm a stopped multishot request when the application’s backpressure policy permits.

Kernel and library gates are separate

The regular ring-provided-buffer registration API is available since Linux 5.19. Incremental consumption, IOU_PBUF_RING_INC, and its IORING_CQE_F_BUF_MORE ownership signal are available since Linux 6.12. The older IORING_CQE_F_MORE is not a synonym or a version substitute for the incremental flag. Distribution kernels may backport features, so a version string is useful context but not a complete capability probe.

Build with liburing and UAPI headers that define the incremental flag; the enum identifier cannot be detected with a preprocessor #ifdef. At runtime, try registering with IOU_PBUF_RING_INC. If that registration fails, record the negative error returned by liburing and try the same bounded ring without the incremental flag. Use that ordinary ring only if its whole-buffer ownership model fits the application. If registration still fails, choose an explicit per-request destination buffer or the older io_uring_prep_provide_buffers() pool, or fail startup if the feature is required. Do not classify every -EINVAL as “kernel too old”: invalid parameters and platform memory constraints can also reject registration. Validate the group ID, power-of-two entry count, backing memory, policy, and error path separately.

Build a bounded pool and publish it once

The following are illustrative code fragments, not a complete compile-ready program: application callbacks and surrounding event-loop state are intentionally omitted, and each CQE must still be acknowledged by the caller (for example, with io_uring_cqe_seen()) after its state has been processed. The pool fragment fixes 64 entries of 64 KiB each: 4 MiB of payload memory per pool, plus the descriptor ring allocated by liburing. It deliberately caps entries well below the documented maximum of 32,768 and uses a power of two. It assumes a Linux program has already initialized ring with io_uring_queue_init() and uses a liburing version whose headers provide the incremental API.

#define _GNU_SOURCE
#include <errno.h>
#include <liburing.h>
#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>

#define PBUF_ENTRIES 64u
#define BUFFER_BYTES (64u * 1024u)
#define PBUF_GROUP 17

_Static_assert((PBUF_ENTRIES & (PBUF_ENTRIES - 1)) == 0,
               "provided-buffer ring entries must be a power of two");
_Static_assert(PBUF_ENTRIES <= 32768u, "ring exceeds documented maximum");

struct receive_pool {
    struct io_uring_buf_ring *ring;
    unsigned char *storage;
    size_t used[PBUF_ENTRIES];
    int incremental;
};

static int
init_receive_pool(struct io_uring *ring, struct receive_pool *pool)
{
    size_t bytes;
    long page_size = sysconf(_SC_PAGESIZE);
    int err = 0;
    int alloc_err;
    void *storage = NULL;

    memset(pool, 0, sizeof(*pool));
    if (page_size <= 0 ||
        ((unsigned long)page_size & ((unsigned long)page_size - 1)) != 0)
        return -EINVAL;
    if ((size_t)PBUF_ENTRIES > SIZE_MAX / (size_t)BUFFER_BYTES)
        return -EOVERFLOW;
    bytes = (size_t)PBUF_ENTRIES * (size_t)BUFFER_BYTES;

    alloc_err = posix_memalign(&storage, (size_t)page_size, bytes);
    if (alloc_err != 0)
        return -alloc_err; /* posix_memalign returns an error number directly. */
    pool->storage = storage;

    pool->ring = io_uring_setup_buf_ring(ring, PBUF_ENTRIES, PBUF_GROUP,
                                         IOU_PBUF_RING_INC, &err);
    if (pool->ring) {
        pool->incremental = 1;
    } else {
        const int incremental_error = err; /* liburing reports -errno here. */

        fprintf(stderr, "incremental buffer ring unavailable (%d); "
                        "trying ordinary provided buffers\n",
                incremental_error);
        pool->ring = io_uring_setup_buf_ring(ring, PBUF_ENTRIES, PBUF_GROUP,
                                             0, &err);
        if (!pool->ring) {
            free(pool->storage);
            pool->storage = NULL;
            return err < 0 ? err : -EIO;
        }
        pool->incremental = 0;
    }

    const int mask = io_uring_buf_ring_mask(PBUF_ENTRIES);
    for (unsigned i = 0; i < PBUF_ENTRIES; i++) {
        void *addr = pool->storage + (size_t)i * BUFFER_BYTES;

        io_uring_buf_ring_add(pool->ring, addr, BUFFER_BYTES,
                              (unsigned short)i, mask, (int)i);
    }
    io_uring_buf_ring_advance(pool->ring, PBUF_ENTRIES);
    return 0;
}

If the ordinary ring is the selected mode, the code still has a working provided-buffer path, but it must not apply incremental offsets or expect F_BUF_MORE. If neither registration succeeds, use a different receive strategy rather than submitting requests against an absent ring. The setup helper’s error is negative errno; preserve it in logs. A failed registration can reflect more than a missing kernel feature.

The next submission excerpt belongs inside an application function that has access to ring, socket_fd, and RECEIVE_TOKEN; it is not a standalone translation unit. A multishot receive uses buffer selection while one SQE remains armed. With incremental consumption, consecutive CQEs may name the same buffer ID when F_BUF_MORE is set. Reserve a buffer-group ID that is not already registered on this ring:

struct io_uring_sqe *sqe = io_uring_get_sqe(&ring);
if (sqe == NULL) {
    /* Submit or drain the SQ before asking for another entry. */
    return -EAGAIN;
}

io_uring_prep_recv_multishot(sqe, socket_fd, NULL, 0, 0);
sqe->flags |= IOSQE_BUFFER_SELECT;
sqe->buf_group = PBUF_GROUP;
io_uring_sqe_set_data64(sqe, RECEIVE_TOKEN);

The multishot helper requires a zero length, buffer selection, and no MSG_WAITALL; it can stop and require re-arming. Check the CQE flags on every completion, including failures. SQE preparation does not itself guarantee that a particular kernel or sandbox will accept the operation.

Process each CQE without conflating ownership

This handler fragment is deliberately for a SOCK_STREAM receive path. A zero-byte recv() means orderly peer shutdown for a stream; for a datagram socket, a zero-length datagram is a valid record and must not be classified as EOF. consume_bytes() must finish reading or copy the range before the buffer is recycled; if it queues asynchronous work, recycling waits for that work too. rearm_receive() is an application policy decision, not a consequence of the buffer flag. The other named callbacks below are application-specific placeholders; wire them to the program’s connection state and error policy, and acknowledge each CQE in the surrounding completion loop only after this fragment has finished handling it.

One v6.12 ownership detail is easy to miss: io_ring_buffer_select() commits a descriptor early when issue_flags includes IO_URING_F_UNLOCKED or the file cannot be polled. A zero requested length is expanded to the selected descriptor’s available length before this commit. For an incrementally consumed ID, that is the remaining extent, not necessarily its original capacity. The kernel then clears req->buf_list but retains buffer-ring request state (REQ_F_BUFFER_RING) for completion; do not confuse that with the selected-buffer state that has been cleared. For a valid IORING_OP_RECV, the non-pollable-file condition is not the route: io_recv() first requires a socket, and socket files provide a poll operation. The exact SQE shown normally submits under the ring lock, and its ordinary apoll retry is also locked; it does not take the unlocked branch on that normal repeating-multishot path. However, an apoll-aborted fallback can queue it to io-wq, as can a request explicitly forced async. v6.12 does not issue a request while REQ_F_APOLL_MULTISHOT remains set in io-wq: it rejects a non-pollable file; for a pollable nonblocking/NOWAIT file it attempts to arm poll and returns without issuing only if that succeeds (otherwise the request fails); for a pollable blocking file it clears REQ_F_APOLL_MULTISHOT before issuing. Thus the unlocked path can occur for an SQE submitted as multishot, but only as a single-completion fallback rather than an actively repeating multishot issue. That fallback has no F_MORE, and userspace must decide whether to submit a replacement request.

For the exact zero-length IORING_OP_RECV multishot sample, an io-wq fallback expands the length to the descriptor’s full remaining extent and commits that extent before the receive result is known. A short positive result can therefore return IORING_CQE_F_BUFFER without F_BUF_MORE; a zero or a negative result that reaches the completion path can do likewise. This is a full-descriptor release here because the accepted multishot IORING_OP_RECV contract requires len == 0, not because the flag combination universally proves it. Do not generalize that to every error: a forced-nonblocking -EAGAIN path can retry or skip completion instead of posting a CQE. On the ordinary locked poll path, a short positive result commits only the bytes received and can produce F_BUFFER | F_BUF_MORE; a nonpositive result with no earlier partial I/O recycles the selection flags before completion, so it emits no buffer CQE flag. Process any reported buffer ownership before interpreting receive status. The pinned path marks nonpositive completion results as ending the active multishot request, so they do not carry IORING_CQE_F_MORE.

Keep the example’s submission contract narrow. It uses IORING_OP_RECV multishot with sqe->len == 0, msg_flags == 0, and does not enable IORING_RECVSEND_BUNDLE. The v6.12 preparation path rejects a nonzero length for multishot IORING_OP_RECV; the bounded-length case below is therefore a separate one-shot (or otherwise applicable selected-receive) contract, not this example. In v6.12, early commit uses the requested length and then drops req->buf_list. If an incremental ring is selected on that early-commit path with a nonzero length smaller than the descriptor’s remaining length, io_kbuf_commit() advances the descriptor address and reduces its length without advancing the ring head, while the later CQE path can still report F_BUFFER without F_BUF_MORE. A handler that then republishes the full descriptor would duplicate a descriptor that remains at the head. Avoid that bounded-length/unlocked combination on kernels with this v6.12 behavior, or use a separately validated receive strategy; do not infer whole-descriptor release merely from F_BUFFER without F_BUF_MORE in that case.

The code below also deliberately omits MSG_TRUNC. With datagram receives, MSG_TRUNC can make the syscall return the full packet length even though only the selected region was copied. In the original v6.12 bounded, non-multishot path, that returned length could advance an incremental descriptor farther than the bytes copied (the kernel clamps against the descriptor’s remaining length, but that may exceed the requested receive length). The stable fix separates payload bytes consumed from the value returned to preserve MSG_TRUNC semantics; it is present in upstream 6.12.112. The upstream discussion’s separate io_recvmsg_multishot() path clamps its own consumed length; do not generalize that result to IORING_OP_RECV multishot, which follows the raw sock_recvmsg() path. Verify vendor kernels for the backport and keep datagram/MSG_TRUNC accounting out of this stream example unless that fix is present and tested.

static int
handle_receive_cqe(struct receive_pool *pool, struct io_uring *ring,
                   int socket_fd, const struct io_uring_cqe *cqe)
{
    const unsigned flags = cqe->flags;

    if (flags & IORING_CQE_F_BUFFER) {
        const unsigned bid = flags >> IORING_CQE_BUFFER_SHIFT;
        size_t start = 0;

        if (bid >= PBUF_ENTRIES)
            return -EPROTO;

        if (pool->incremental)
            start = pool->used[bid];
        if (cqe->res >= 0) {
            const size_t received = (size_t)cqe->res;

            if (start > BUFFER_BYTES || received > BUFFER_BYTES - start)
                return -EOVERFLOW;
            if (received != 0)
                consume_bytes(pool->storage + (size_t)bid * BUFFER_BYTES + start,
                              received);
            if (pool->incremental)
                pool->used[bid] = start + received;
        } else {
            /* No payload bytes, but the CQE still returns an early-committed buffer. */
        }

        if (flags & IORING_CQE_F_BUF_MORE) {
            if (!pool->incremental)
                return -EPROTO;
            /* The kernel still owns this buffer. Do not add it to the ring. */
        } else {
            /* This selected buffer is back in application ownership. */
            if (buffer_consumer_is_done(bid)) {
                void *base = pool->storage + (size_t)bid * BUFFER_BYTES;
                const int mask = io_uring_buf_ring_mask(PBUF_ENTRIES);

                pool->used[bid] = 0;
                io_uring_buf_ring_add(pool->ring, base, BUFFER_BYTES,
                                      (unsigned short)bid, mask, 0);
                io_uring_buf_ring_advance(pool->ring, 1);
        } else {
            /* Deferred completion must reset used[bid] and republish once. */
            defer_recycle_until_consumer_done(bid);
            }
        }
    } else if (cqe->res > 0) {
        return -EPROTO; /* This receive path expected a selected buffer. */
    }

    if (cqe->res == 0) {
        /* SOCK_STREAM EOF: retire receive state; this is not a retry. */
        mark_peer_eof(socket_fd, (flags & IORING_CQE_F_MORE) != 0);
        return 0;
    }
    if (cqe->res == -ENOBUFS) {
        /* There is no selected bid; defer one rearm until capacity returns. */
        record_buffer_starvation();
        if (flags & IORING_CQE_F_MORE) {
            /* Do not submit a duplicate while an implementation keeps it armed. */
            return 0;
        }
        /* Deduplicate recovery by receive token; submit only one replacement SQE. */
        defer_rearm_until_buffer_capacity(socket_fd);
        return 0;
    }
    if (cqe->res < 0) {
        report_receive_error(cqe->res); /* io_uring uses negative errno */
        if (is_terminal_receive_error(cqe->res)) {
            schedule_connection_teardown(socket_fd);
            return 0;
        }
        if (!(flags & IORING_CQE_F_MORE) &&
            connection_wants_receive(socket_fd))
            return rearm_receive(ring, socket_fd);
        return 0;
    }
    if (!(flags & IORING_CQE_F_BUFFER))
        return -EPROTO; /* A positive selected receive must identify its buffer. */
    if (!(flags & IORING_CQE_F_MORE) &&
        connection_wants_receive(socket_fd))
        return rearm_receive(ring, socket_fd);
    return 0;
}

The fragment keeps two counters separate: used[bid] tracks the byte offset inside one incremental buffer, while F_MORE identifies whether a multishot SQE remains armed. The deferred recycle helper has a precise obligation: after the last consumer releases the bytes, it resets used[bid] to zero, republishes the original base address and full capacity, and advances the shared tail exactly once. This matters especially for an early-committed short positive read, whose F_BUFFER without F_BUF_MORE returns the entire descriptor even though used[bid] was advanced by the received byte count. The immediate path above performs the same reset and republish synchronously. The fragment does not make rearming automatic: EOF marks the receive side complete, -ENOBUFS defers one rearm until buffers are replenished, and terminal errors enter teardown. If EOF or teardown is observed with F_MORE set on a different implementation, those helpers must cancel and drain the tracked request, process any already-posted CQEs, and settle their buffer leases before freeing request state. A production handler also needs explicit shutdown and worker-ownership rules for other outstanding work. Do not recycle a buffer while a send, parser, or other consumer still refers to its bytes. If several workers return buffers, serialize their add-and-advance operations through one publisher or another synchronization protocol; they update the same ring tail.

In the pinned Linux v6.12 io_recv path, buffer selection returns -ENOBUFS when the group has no available descriptor; no buffer was selected for that CQE. The multishot path marks the non-positive result as finished before completing it, so the request is not left armed with F_MORE. The defensive flag check above avoids a duplicate SQE if another implementation does leave its request armed. For the pinned path, the application must wait until its buffer-replenishment policy is met and then submit one new receive.

Backpressure and teardown are part of the protocol

An incremental buffer reduces waste when a stream produces small chunks, but it does not remove memory limits or guarantee progress. Bound both the descriptor count and entries × bytes-per-buffer; account for memory per connection, plus any downstream queue that retains completed ranges. When buffers are not returned quickly enough, the ring can run out and a receive can complete with -ENOBUFS. Pause or re-arm based on measured capacity; do not spin on a depleted group.

On teardown, first stop new submissions and settle or cancel outstanding receives and consumers. Drain the CQEs needed to release active leases. Then unregister and free the descriptor ring with io_uring_free_buf_ring() while its parent io_uring context is still valid. Free the separately allocated payload storage only after the ring no longer references it and no application consumer retains it. If unregistration fails, do not free memory that may still be registered or in use; handle the error before continuing teardown.

Validate the path on both the minimum supported kernel and a kernel that supports incremental consumption. Exercise short receives, multiple CQEs for one ID, simultaneous F_MORE and F_BUF_MORE, an F_MORE-clear multishot termination, -ENOBUFS, registration failure, cancellation, and shutdown with a buffer still leased. Record the kernel, liburing and header versions, selected mode, ring and payload sizes, CQE flags, and fallback reason. Compare against an ordinary ring or explicit receive buffers under the same queue depth and workload; adopt incremental mode only when its memory or throughput benefit justifies the additional per-ID state.

Related:

Sources: