Skip to content
LinuxDeep Dive Published Updated 6 min readViews unavailable

Linux HMM Mirrors: Notifier Invalidation, Range Faults, and Safe Retries

Trace Linux HMM range snapshots from notifier registration through retry-safe device page-table updates, with locking and fault rules made explicit.

Linux Heterogeneous Memory Management (HMM) is kernel infrastructure for drivers that need a device-side view of a process address space or need to migrate pages to device memory. It does not make an arbitrary GPU or accelerator coherent, and it is not a userspace switch that enables shared virtual memory. Hardware-specific page-table updates, invalidation, and device synchronization remain the driver’s responsibility.

For a mirrored range, correctness depends on treating a page-table read as a snapshot that can become stale while the driver works. The current upstream documentation describes a protocol built from an mmu_interval_notifier, an HMM range-fault helper, a driver update lock, and a sequence retry check. The unlocked timeout helper shown below is version-dependent: it is declared by the upstream Linux v7.3-rc7 header but not by the v7.2 header. Check the exact target kernel tree, especially for vendor kernels, before using this interface; older headers require their own supported locking and fault-handling path. The important boundary is between discovering CPU mappings and making corresponding device mappings visible: an invalidation must not race through that boundary unnoticed. This is distinct from registering a userfaultfd handler: HMM consumes kernel page-table state for a device, while userfaultfd delegates selected fault resolution to a userspace manager.

Register the range and define invalidation work

A driver begins by registering an mmu_interval_notifier for the target mm_struct and virtual-address range. Its operations include an invalidate() callback. When CPU page-table changes affect the watched interval, the callback must update the device view, for example by marking an entry read-only or unmapping it, and complete that update before returning. The exact operation depends on what the device can safely do; HMM provides common coordination, not a device-independent page-table writer.

The driver should use one update lock for both sides of the race. The invalidation callback takes this lock while applying its device-side change. The population path takes the same lock before asking whether the HMM snapshot is still valid and before installing device entries. That shared serialization point is why the sequence check must occur under the lock, not earlier in an unlocked gap.

Fault a range, then validate before publishing

For the upstream v7.3-rc7 interface, the normal documented helper for populating device mappings is hmm_range_fault_unlocked_timeout(). It takes the mmap read lock internally and may let handle_mm_fault() drop it while resolving a fault. This matters for VMAs whose fault handlers can release that lock, including userfaultfd-managed ranges. A caller must not hold mmap_read_lock() before calling this helper.

The protocol, with driver-specific setup and error handling omitted, is:

again:
    ret = hmm_range_fault_unlocked_timeout(&range, timeout);
    if (ret != 0)
        goto out;

    take_lock(driver->update);
    if (mmu_interval_read_retry(range.notifier, range.notifier_seq)) {
        release_lock(driver->update);
        goto again;
    }

    /* Install device page-table entries from the validated range snapshot. */
    update_device_page_table(driver, &range);
    release_lock(driver->update);

This is a control-flow sketch, not a complete driver: the range, PFN array, notifier, mm lifetime, locking primitive, cleanup path, and hardware-specific update routine must all be initialized for the target kernel and device. In the documented pattern, the driver lock is the same lock used in invalidate(). HMM records the notifier sequence in the range; the retry call must use that notifier and sequence. If invalidation or a lock drop made the snapshot stale, release the lock and fault again. Only install entries after the under-lock retry check succeeds. The take_lock and release_lock names mirror the documentation’s pseudocode; they are not kernel functions to copy into a driver.

The range flags express what the driver requires, and the distinction between range defaults and per-entry overrides is important for write faults. For a readable snapshot across the range, initialize range.default_flags to HMM_PFN_REQ_FAULT and range.pfn_flags_mask to zero. To request write access for only one entry, retain HMM_PFN_REQ_FAULT as the default, set the mask to HMM_PFN_REQ_WRITE, and put HMM_PFN_REQ_WRITE in that entry’s PFN slot before faulting. This combination requests at least readable mappings across the range and write access at the selected address. Do not set the write mask and entry flag while omitting the default fault request: the upstream header requires HMM_PFN_REQ_WRITE to be combined with HMM_PFN_REQ_FAULT. After the helper succeeds, the PFN flags describe current page-table state, including validity and writability. Requesting access is not a promise that every mapping can be made writable; handle the helper’s error and returned per-page state according to the driver’s policy.

The mapping’s lifetime is another independent requirement. The kernel’s illustrated pattern first obtains a live mm reference with mmget_not_zero() and calls mmput() on its exit path. A notifier registration and a PFN snapshot do not by themselves keep the address space alive for arbitrary later device work. Pair references and notifier teardown with the driver’s request and device-work lifecycle so cleanup cannot outlive the address space it describes.

Bound retries without pretending to cancel a fault

The timeout argument is in jiffies. Passing zero means retry indefinitely. A nonzero budget is checked between retry attempts inside one helper call; it preserves caller policy when repeated notifier invalidations prevent a stable range from being obtained. If that internal retry budget is exhausted before a stable snapshot is found, the documented result is -EBUSY. This timeout does not interrupt an individual page-fault operation already in progress. The helper can also return -EINTR if acquiring the mmap lock is interrupted or a fatal signal is pending during retry handling. The outer goto again in the sketch starts another helper call after a sequence retry, so a driver that needs one overall deadline must also bound that outer loop; the helper’s timeout alone is not a total wall-clock deadline for the complete protocol.

If the caller truly must retain the mmap lock across work outside HMM, the documentation exposes hmm_range_fault() as a different interface. It requires the caller to hold the lock and leaves it held on return, but cannot support VMAs whose fault handlers need to drop it. New callers should prefer the unlocked timeout helper unless they have a specific reason to keep the lock held.

Keep the guarantee narrow

HMM mirrors CPU address-space state and coordinates invalidation; it does not remove the need for device-specific TLB invalidation, memory barriers, DMA operations, or synchronization required by the hardware. The kernel documentation also treats migration to and from device memory as a separate driver operation using migrate_vma_* helpers and device-specific data movement. A successful range fault is therefore not proof that device work is complete or that a page is resident in device memory.

When diagnosing stale or missing mappings, record the notifier range, sequence used for the snapshot, whether the update lock is shared with invalidation, helper return code, and the device update completion point. Repeated -EBUSY indicates that the driver did not obtain a stable snapshot within its retry budget; it is not evidence that the user mapping is permanently invalid. A device mapping that remains stale after successful validation points instead to the driver’s update, invalidation, or hardware synchronization path.

Related:

Sources:

Comments