FreeBSD netmap: Memory-Mapped Packet I/O Without a Socket per Packet

A normal socket is the right abstraction for most network programs. The kernel validates protocol state, schedules traffic, copies or maps data as needed, and lets an application think in streams or datagrams. A packet forwarder, traffic generator, software switch, or capture engine has a different workload: it may touch only a few bytes in each frame while paying the syscall and per-packet costs of the full stack millions of times per second.

FreeBSD’s netmap framework attacks that overhead by exposing packet buffers and queue descriptors through memory-mapped rings. An application processes batches in shared memory, then uses a small number of synchronization operations to tell the kernel which slots it consumed or produced. That is a powerful data plane, but it is not a promise that every packet moves without copying or that ordinary network semantics remain unchanged.

Rings mirror the queueing problem

A netmap port contains receive and transmit rings. Each ring has slots that identify a buffer and its current length, plus producer and consumer positions shared with the kernel. Registering a physical interface normally exposes rings corresponding to its hardware queues. Other port types represent the host stack, a pipe, or a VALE software-switch endpoint.

The application maps the netmap region once and walks slots directly. It can inspect an incoming frame, swap buffer indexes to forward it, or write a new frame into an available transmit buffer. The expensive boundary crossing moves from every packet to every useful batch.

This design preserves an important distinction: descriptors and buffers are shared, but ownership changes over time. A slot that belongs to the NIC or kernel cannot be rewritten safely just because its address is mapped in the process. Ring indexes are a synchronization protocol, not bookkeeping that can be updated in any order.

Registration defines what the process owns

The native interface opens /dev/netmap and registers a port with NIOCREGIF. The request selects a name and mode, and the response describes ring and memory layout. Most applications should use the maintained libnetmap helpers instead of duplicating ABI parsing, but the same lifecycle remains visible:

struct nmport_d *port = nmport_prepare("netmap:igc0");
if (port == NULL)
    errx(1, "cannot prepare netmap port");

if (nmport_open_desc(port) < 0)
    err(1, "cannot register netmap port");

/* poll port->fd, process complete batches, then close the descriptor */

Code must validate the negotiated ring counts and slot counts rather than compiling in assumptions from one adapter. Multi-queue programs should assign rings deliberately and avoid having unrelated threads mutate the same ring. A descriptor close returns ownership and tears down that registration, so worker lifetime must not outlive the port object.

Batching is the real optimization

Memory mapping alone does not make a fast program. The saving comes from amortizing synchronization, cache misses, and wakeups across multiple frames. A busy-poll loop can process a bounded burst and then synchronize once. An event-driven loop can use poll() or select() when no ring has work.

A robust loop places limits on both batch size and time spent on one queue. Draining a permanently busy receive ring without a budget can starve transmit completion, control messages, or another queue. Conversely, synchronizing after every slot recreates much of the overhead netmap was designed to remove.

Prefetching headers and keeping per-packet state compact help, but only after correctness. Validate captured lengths before parsing Ethernet, VLAN, IP, or transport headers. Frames can be truncated or malicious. A packet API below the IP stack removes protections as well as overhead.

Buffer swapping can avoid a payload copy

When forwarding between compatible netmap rings, an application can exchange buffer indexes rather than copy the entire frame. It marks the slots so the framework knows their buffers changed, publishes the received length on the transmit side, and advances both rings according to the API.

That is zero-copy within this particular handoff. It does not mean the whole system is copy-free. A program may copy metadata, a NIC may move bytes with DMA, a virtual port may have different constraints, and an application that retains a packet after returning its slot must copy it into storage it owns.

Buffer lifetime is the common trap. Once a receive slot is released, the same storage can immediately hold an unrelated packet. Queuing a bare pointer to another thread without transferring the corresponding buffer ownership creates corruption that appears only under load.

NIC offloads change what frames mean

Checksum, segmentation, receive coalescing, VLAN, and other hardware offloads can make captured or transmitted frames differ from the wire representation an application expects. The netmap manual documents interface-specific limitations and often requires disabling incompatible offloads.

Treat adapter model, driver, firmware, queue count, MTU, and offload state as part of the deployment configuration. Verify checksums and maximum frame sizes with traffic generated outside the host. A benchmark that loops synthetic frames through one port is useful, but it cannot prove on-wire correctness.

Link speed is also not an application flow-control policy. If transmit rings fill, the program must decide whether to retry, queue within a bounded budget, backpressure an upstream source, or drop. Unbounded user-space queues merely move congestion until memory is exhausted.

VALE provides a useful software laboratory

VALE ports connect through an in-kernel software switch. They are valuable for testing forwarders, connecting virtual machines, and building repeatable topologies without assigning one physical NIC per endpoint. Netmap pipes provide another controlled way to connect producer and consumer sides.

These facilities are not security boundaries by themselves. Device permissions determine who can register ports, while the application still owns packet validation, isolation, and policy. Run the processor with the least privilege needed after opening its descriptors, and place management APIs on a separate authenticated path.

Use VALE to test congestion, disconnects, malformed frames, ring wraparound, and process restart. Then repeat essential tests on every production driver because hardware behavior is precisely the part a virtual topology cannot reproduce.

Measure the complete forwarding contract

Packets per second is only one result. Record throughput by frame size, drops at each ring, latency distribution, CPU time by core, interrupt behavior, NUMA placement where relevant, and recovery after a link flap. Confirm that counters reconcile: received, intentionally dropped, forwarded, and still queued should explain the input.

Netmap is most effective when the application has a narrow packet-level job and can own queue semantics explicitly. It replaces repeated stack crossings with shared rings and batches. In return, the program must manage buffer lifetime, fairness, parsing, congestion, and driver-specific behavior with the discipline normally supplied by higher network layers.

Related:

Sources: