FreeBSD bus_dma: Map Constraints, Synchronization, and Safe Teardown

FreeBSD’s bus_dma interface is the kernel’s portable contract for mapping memory that a device will access directly. It hides machine-specific address translation, I/O MMU programming, bounce buffers, and cache synchronization behind tags, maps, and explicit synchronization calls. The central safety rule is simple: a kernel virtual address is not a DMA address. A driver must ask busdma for device-visible segments and obey the ownership transitions around each transfer.

This article follows FreeBSD 15.1 bus_dma(9). Derive constraints from the device specification, bridge, and target-release manual; the pinned iwx code below is an implementation example, not a template.

Model the tag as a hardware contract

A bus_dma_tag_t describes a group of transactions and the restrictions they must satisfy. The root of a driver’s tag hierarchy must inherit from the tag returned by bus_get_dma_tag() so bridge, CPU-memory, and device restrictions are preserved. Driver-created child tags can then tighten those inherited constraints, not relax them.

Translate the hardware manual into the tag fields deliberately:

alignment is the required starting-address alignment.
boundary prevents an individual DMA segment from crossing a specified address boundary. FreeBSD’s documented constraint requires a power-of-two boundary that is at least as large as maxsegsz; zero means there is no boundary restriction.
lowaddr and highaddr define the bus-address window that the device cannot directly access and that busdma must avoid or accommodate. For a device that cannot address above 4 GiB, FreeBSD documents BUS_SPACE_MAXADDR_32BIT as lowaddr and BUS_SPACE_MAXADDR as highaddr; do not replace that constraint with a cast of a 64-bit pointer.
maxsize limits the sum of all segment lengths in one map, nsegments limits the scatter/gather list, and maxsegsz limits an individual segment.

Do not guess these numbers from the host’s RAM size or the width of a C pointer. A driver may need separate tags for a descriptor ring and payload buffers: one device can require tightly aligned descriptors while allowing packet buffers with different alignment or segment limits. Create the highest driver tag with the inherited bus tag as its parent, then derive transaction-specific tags from that shared parent. In FreeBSD 15.1, the old filtfunc arguments to bus_dma_tag_create() must be NULL; the manual says use of filters is no longer supported and returns an error.

The returned bus_dma_segment_t entries are the handoff to the device. ds_addr is the device-visible address and ds_len is its length. A mapping may involve an I/O MMU or bounce pages, so neither a KVA nor a physical-address guess is a substitute. Encode the returned address and length into descriptors using the byte order and field width required by the device specification.

Separate long-lived memory from transient transfers

The manual distinguishes static and dynamic transactions because they have different lifetimes.

A static transaction is a long-lived allocation such as a descriptor ring. bus_dmamem_alloc() returns kernel virtual memory and an associated map. The driver performs the initial load needed to obtain the device-visible segment list, then reuses that mapping for the ring’s lifetime. The map is managed by the allocation routine; do not separately create or destroy it. Before release, stop device access, synchronize as required, unload the mapping, and call bus_dmamem_free().

Dynamic transactions map buffers owned elsewhere, such as a packet or I/O block. Create a pool of maps with bus_dmamap_create(). A map cannot be loaded while it already has an active mapping; after the transfer and required post-synchronization, unload it before returning it to the pool. Keep each map associated with exactly the request whose buffer it represents. This ownership discipline prevents a recycled map from silently referring to memory still in use by hardware.

Treat load callbacks as asynchronous state transitions

bus_dmamap_load() returns without blocking, but that does not mean the mapping callback has necessarily completed. If mapping resources are temporarily unavailable, the manual permits deferring the callback and returning EINPROGRESS. The callback then runs when resources become available. Its segment array is valid only for the duration of the callback, so copy the needed address and length values into the driver’s own descriptor state before returning from it.

Deferred callbacks are serviced FIFO, but later loads for the same tag that need no additional resources can still succeed first. If the device queue requires request order to be preserved, the driver must stall later submissions while an earlier mapping is pending. Do not assume that submitting after bus_dmamap_load() returns preserves order.

BUS_DMA_NOWAIT requests immediate failure instead of deferred mapping when resources are unavailable; the documented resource failure is ENOMEM. EFBIG means the requested size fit maxsize, but the mapping could not fit the tag’s segment constraints. Diagnose it by checking nsegments, maxsegsz, boundary, and the device’s descriptor limit. Splitting a request, using a different buffer strategy, or rejecting it may be appropriate only when the hardware and higher-level protocol allow that change. Loosening a tag to suppress EFBIG can create a descriptor the device cannot consume.

Keep the request, buffer, map, and callback context alive until deferred success or error is delivered. Clean up only on a path that knows whether the mapping became active; returning from the initiating function is not proof that a deferred callback is finished. bus_dmamap_destroy() returns EBUSY for an active map, while bus_dma_tag_destroy() returns it while maps remain associated with the tag. Preserve and diagnose either failure.

Synchronize each ownership transfer

bus_dmamap_sync() expresses transfer direction from the host-memory point of view. For a transmit buffer, the CPU writes memory that the device will read. After the CPU’s last update and before submission, synchronize with BUS_DMASYNC_PREWRITE. After the device has finished reading and before the CPU reuses or releases that memory, use BUS_DMASYNC_POSTWRITE as required by the transaction lifecycle.

For a receive buffer, the device writes memory that the CPU will later read. Prepare the mapping with BUS_DMASYNC_PREREAD before giving it to the device. Only after device completion, synchronize with BUS_DMASYNC_POSTREAD before the CPU examines received bytes. These flags are not interchangeable: their names describe the host’s read or write relative to device access, not the driver’s API call named “read” or “write.” The manual states that omitting the appropriate synchronization makes behavior undefined.

Descriptor rings need the same ownership reasoning: sync CPU-written descriptors before hardware reads them, and use the corresponding pre-read and post-read transitions before the CPU reads status fields written by hardware. Keep descriptor and payload synchronization separate when they use different maps.

BUS_DMA_COHERENT may reduce sync cost but does not remove the requirement to call bus_dmamap_sync(). Busdma synchronization handles its documented cache and CPU-memory-ordering work; it does not define a device’s descriptor ownership protocol, register/doorbell semantics, or completion rules. Apply any additional ordering primitive required by the hardware and bus-space documentation.

Read a real transmit path without cargo-culting it

FreeBSD’s iwx driver at a pinned commit from releng/15.1 illustrates the separation: it writes returned segment addresses and lengths into hardware descriptors, PREWRITE-syncs the payload, command, and descriptor maps, then updates the ring and kicks the device. Its descriptor format, byte order, queue bookkeeping, and kick mechanism are specific to that controller, not a universal sequence.

Review one request end to end: map creation, load and callback, descriptor fill, PRE sync, submission, completion, POST sync, unload, and buffer reuse. At each transition identify the owner and evidence that the device has stopped accessing the memory, including error paths.

Quiesce hardware before freeing maps or memory

Teardown is not simply allocation in reverse. Stop submissions, use the device-specific procedure to drain, halt, or reset DMA, and ensure no operation or deferred callback can reference state being removed. POST-sync completed transfers before CPU reuse; unload dynamic maps, destroy inactive maps, release buffers, then destroy child tags before parents. For static allocations, unload before bus_dmamem_free(). Preserve and diagnose the manual’s EBUSY errors for active maps or tags that still have associated maps.

Test fragmented buffers, segment and address limits, boundary crossings, deferred loads under resource pressure, BUS_DMA_NOWAIT failure, TX/RX reuse, and attach-failure cleanup. Emulators and common x86 hosts can conceal missing cache operations that matter on non-coherent systems; test the supported architectures and devices.

Related:

Sources: