Skip to content
LinuxDeep Dive Published Updated 7 min readViews unavailable

Linux IOMMUFD and VFIO: Establish DMA Ownership with IOAS

Trace VFIO device binding into IOMMUFD, inspect IOAS mapping constraints, and preflight group isolation without unbinding a live device.

Passing a PCI device to a userspace driver is not just a matter of opening a device file. The process must obtain a supported VFIO interface, claim DMA ownership through the kernel, attach the device to an I/O address space, and map only the memory the device is meant to reach. Linux IOMMUFD provides a userspace API for managing I/O page tables; VFIO connects supported devices to that API.

The important boundary is between device ownership and memory mapping. A successful VFIO cdev bind establishes the DMA ownership claim, but the device still needs an attached IOMMUFD page table before userspace DMA mappings are established. An IOAS describes IOVA-to-memory mappings; it is not a promise that every address or device can use the same range.

The API names below follow the current kernel.org IOMMUFD and VFIO documentation reviewed on 2026-10-12. Kernel configuration, architecture, IOMMU driver, device topology, and distribution backports vary. The inventory below is read-only: it does not detach drivers, bind a device, alter IOMMU state, or expose the device to a workload.

Inventory the PCI function and its isolation group

Record the running kernel and inspect one candidate device. Replace the example BDF with the exact PCI address from the system you are evaluating:

uname -r
bdf=0000:03:00.0
dev="/sys/bus/pci/devices/$bdf"

if test ! -d "$dev"; then
    printf 'No PCI function at %s\n' "$bdf" >&2
    exit 1
fi

printf 'Driver: '
readlink "$dev/driver" || printf 'none bound\n'

if test ! -d "$dev/iommu_group"; then
    printf 'No IOMMU group directory for %s\n' "$bdf" >&2
    exit 1
fi

group=$(readlink -f "$dev/iommu_group") || {
    printf 'No readable IOMMU group for %s\n' "$bdf" >&2
    exit 1
}
printf 'IOMMU group: %s\nMembers:\n' "$group"
for member in "$group"/devices/*; do
    test -e "$member" && printf '  %s\n' "$(basename "$member")"
done

if test -c /dev/iommu; then
    ls -l /dev/iommu
else
    printf '%s\n' '/dev/iommu is not present'
fi
if test -d /dev/vfio/devices; then
    ls -l /dev/vfio/devices
else
    printf '%s\n' 'VFIO device cdev directory is not present'
fi

The group listing matters more than the selected function alone. VFIO cdev access still follows IOMMU-group DMA ownership semantics: devices in one group cannot be split between multiple IOMMUFD contexts or simultaneously owned by the host driver and a VFIO bus driver. A sibling function, bridge, or storage device in the group can therefore change whether the assignment is viable. Do not unbind every listed device to see what happens; first identify each function’s role and the host recovery path.

The absence of /dev/vfio/devices means the VFIO device cdev path is not available as inspected. It does not, by itself, prove that no legacy VFIO group/container interface exists. The cdev interface requires the kernel’s VFIO device cdev support and IOMMUFD; it is not a generic replacement for every older VFIO application.

If the running kernel exposes its build configuration, inspect it as supporting evidence:

cfg="/boot/config-$(uname -r)"
if test -r "$cfg"; then
    grep -E 'CONFIG_(IOMMUFD|VFIO_DEVICE_CDEV)' "$cfg" || true
elif test -r /proc/config.gz; then
    zgrep -E 'CONFIG_(IOMMUFD|VFIO_DEVICE_CDEV)' /proc/config.gz || true
else
    printf '%s\n' 'Kernel configuration is not exposed at these paths'
fi

Distribution kernels may store the config elsewhere or omit it. A config line and a device node are preflight clues, not proof that a particular device can be safely attached or that a userspace stack supports it.

Follow the IOMMUFD and VFIO object lifecycle

For the VFIO device cdev flow, userspace opens the device cdev and an IOMMUFD file descriptor, then calls VFIO_DEVICE_BIND_IOMMUFD. The bind is the ownership gate: a successful call claims DMA ownership for the device under the IOMMU group rules. Merely opening the cdev does not give the process full device access.

Next, userspace allocates an IOAS with IOMMU_IOAS_ALLOC and attaches the VFIO device to that address space with VFIO_DEVICE_ATTACH_IOMMUFD_PT. The attachment creates or reuses a hardware page-table object compatible with the IOMMU path. The application maps memory into the IOAS with IOMMU_IOAS_MAP; the mapping associates an IOVA range visible to the device with userspace memory. The device’s own queue or register protocol must still be programmed correctly before it will use that mapping.

Treat the IOAS as a constrained address-space object, not as a flat promise that every IOVA works. After each device attach or detach, query IOMMU_IOAS_IOVA_RANGES and honor the returned ranges and minimum alignment before mapping. The kernel documentation notes that allowed ranges can narrow when devices attach and widen when they detach. If a request fails, preserve the ioctl and errno; errors such as unsupported operation, invalid fields, and unavailable IDs point to different compatibility problems.

The safe conceptual sequence is:

inspect group and driver ownership
open /dev/iommu and the intended VFIO cdev
bind the VFIO device to that IOMMUFD context
allocate an IOAS and attach the device
query allowed IOVA ranges and alignment
map only the buffers required by the device
quiesce the device, stop DMA, then unmap and detach

This is a lifecycle outline, not a pasteable assignment script. Real code must check every open and ioctl result, handle device-specific reset and interrupt behavior, prevent use-after-unmap DMA, and clean up object IDs and file descriptors on every failure path. The kernel’s VFIO documentation includes a C cdev example, but its PCI function, memory mapping, permissions, and device operations are illustrative and must not be copied onto an arbitrary host device.

Preserve the DMA security boundary

An IOMMU can restrict device DMA to mapped IOVA ranges only when the hardware, firmware, kernel driver, and device topology provide the expected isolation. A group is the kernel’s relevant ownership unit for VFIO; it is not necessarily one marketing-named function. If group membership includes another device needed by the host, do not proceed until the assignment design accounts for it.

The no-IOMMU VFIO mode intentionally permits userspace drivers to perform DMA without physical IOMMU protection. It is unsafe for a device exposed to untrusted or faulty code and is not a troubleshooting fallback for a missing IOMMU group, cdev, or mapping. Do not enable it to make an otherwise incompatible passthrough experiment start.

Device assignment also changes host ownership. A display adapter may drive the active console; a storage controller may own the root disk; and network devices may carry the management path. Before a maintenance-window test, identify the exact function, its group siblings, current driver, recovery console, and rollback plan. A successful IOAS bind does not establish that host services remain available or that a guest driver supports the hardware.

For an initial experiment, choose a spare device on a disposable host, not a function required by the running system. Use a kernel/userspace combination documented for that device and inspect the group again after reboot or firmware changes. Avoid writing to hardware registers or issuing DMA until the group, ownership, IOVA limits, mapped buffers, interrupt path, and reset behavior are all understood.

Diagnose failures without changing ownership blindly

If /dev/iommu is absent, check the running kernel and its IOMMUFD configuration, then confirm that the intended device and IOMMU driver are supported. If there is no IOMMU group link, investigate firmware IOMMU enablement, kernel boot policy, and platform topology with the system owner. Do not infer that adding an arbitrary kernel parameter will produce safe isolation.

If the cdev is absent but a legacy group device exists, determine which VFIO API and userspace stack the application actually supports. If VFIO_DEVICE_BIND_IOMMUFD fails, inspect whether another function in the IOMMU group is already owned, whether the driver supports IOMMUFD, and whether the requested API is available in the running kernel. If bind succeeds but attachment or mapping fails, compare the requested IOAS ID, supported IOVA range, alignment, permissions, and device-specific DMA constraints.

Keep the Linux DMA API distinction clear. A kernel driver normally maps buffers with the DMA API for its device. VFIO/IOMMUFD instead mediates a userspace-managed device and its DMA mappings. The two articles describe related but different ownership paths: do not cast a CPU pointer to a DMA address, and do not treat an IOAS as a substitute for driver DMA API calls.

The acceptance record should include kernel and userspace versions, PCI BDF and IDs, current driver, full IOMMU group membership, VFIO interface used, IOMMUFD bind/attach results, queried IOVA ranges, mapping sizes and permissions, and device-specific reset and teardown results. A device is ready only when ownership, DMA scope, host recovery, and workload behavior have all been checked on the exact platform.

Related:

Sources:

Comments