Windows Directory Change Notifications: ReadDirectoryChangesW Without Missed-State Assumptions

ReadDirectoryChangesW is a low-latency hint stream about changes under an open directory handle. It is useful for keeping an index, cache, or user interface fresh, but it is not a durable transaction log. A consumer that treats each notification as an exactly-once, permanent record will eventually become inconsistent after buffer overflow, process downtime, network interruption, or an ambiguous rename sequence. The robust design is to use notifications to trigger work and periodically reconcile derived state with an authoritative directory enumeration or a journal designed for change tracking.

Open the directory for the operation you intend

The function watches a directory handle, not a pathname string. Obtain that handle with CreateFileW, use FILE_LIST_DIRECTORY access, pass FILE_FLAG_BACKUP_SEMANTICS so a directory can be opened, and select sharing flags compatible with the applications that create, rename, and delete entries. A watcher that omits FILE_SHARE_DELETE, for example, may prevent other processes from renaming or deleting files while the handle remains open. FILE_FLAG_OVERLAPPED selects asynchronous operation; without it, the caller can block until a change or error is reported.

The following is a deliberately small setup fragment. Error handling and handle ownership are part of the real implementation, not optional decoration:

HANDLE directory = CreateFileW(
    path.c_str(),
    FILE_LIST_DIRECTORY,
    FILE_SHARE_READ | FILE_SHARE_WRITE | FILE_SHARE_DELETE,
    nullptr,
    OPEN_EXISTING,
    FILE_FLAG_BACKUP_SEMANTICS | FILE_FLAG_OVERLAPPED,
    nullptr);

if (directory == INVALID_HANDLE_VALUE) {
    const DWORD error = GetLastError();
    // Report the operation and error; do not treat NULL as the failure value.
}

Choose the filter according to the application. File-name and directory-name changes answer a different question from last-write, size, attributes, or security changes. Requesting every possible class increases the amount of work and can create noisy downstream behavior. The bWatchSubtree argument is also a policy decision: recursively watching a large tree can produce substantial event volume, and a root-level watcher should not assume that a reported name is a complete path from the volume root.

Treat the returned buffer as a linked sequence of variable records

Each completion contains one or more FILE_NOTIFY_INFORMATION records. NextEntryOffset advances from one record to the next; zero marks the last record. The file name is a counted UTF-16 sequence whose length is FileNameLength in bytes. It is not guaranteed to be NUL-terminated. Parse within the exact number of bytes reported, validate each offset and name length against the remaining buffer, and never call a string routine that scans for a terminator that is not promised.

An implementation should copy the record data it needs before reusing the I/O buffer. A simple architecture has the completion path validate records, copy names and actions into a bounded work queue, immediately issue the next read, and let separate workers perform slower metadata reads or application updates. This prevents a slow consumer from leaving the kernel-side notification buffer idle while new activity accumulates. The queue itself still needs back-pressure: if it fills, record that fact and schedule a full reconciliation rather than silently dropping work.

The Action field describes operations such as adding, removing, modifying, or renaming a name. A rename commonly appears as an old-name record followed by a new-name record, but an application should not build an irreversible transaction around pairing two adjacent records. If notifications are lost, coalesced, or invalidated by an overflow, the pair may be incomplete. Treat action sequences as observations, not proof that every intermediate filesystem state was seen.

Overlapped I/O and completion paths

For an overlapped handle, provide a valid OVERLAPPED structure and buffer whose lifetimes extend until completion has been observed. A design can use an I/O completion port or a completion routine, but it should choose one completion mechanism consistently. The directory handle can be associated with an I/O completion port, then each completed read is processed from the completion packet; alternatively, a completion routine can be used without that association. A pending operation is not finished merely because the initiating function returned FALSE: GetLastError() == ERROR_IO_PENDING means the operation was accepted asynchronously and its buffer must remain valid.

Use one clearly owned OVERLAPPED per outstanding operation and do not reuse it until that operation has completed or cancellation has been fully observed. Shutdown should stop new reads, cancel pending I/O with the appropriate handle and OVERLAPPED, drain the completion, and only then free the buffer and close the directory handle. Closing memory or the handle while the kernel may still reference an outstanding request turns normal shutdown into a lifetime bug.

The watcher should be rearmed promptly after handling each completion. Avoid doing network calls, expensive hashing, UI work, or large recursive scans inline on a completion thread. Queue a compact event, reissue the read, and let independent workers process it. If the consumer needs a strict ordering guarantee for its own state, add an application-level sequence or serialize its update worker; the notification API itself does not create a multi-file transaction boundary.

Overflow is a rescan condition, not a successful empty update

The notification buffer is associated with the directory handle and its size does not grow dynamically after the first call. If enough changes occur before the buffer is drained, the system can discard the buffered detail. A successful synchronous return with zero bytes can mean that the buffer could not represent the changes; in the asynchronous path, ERROR_NOTIFY_ENUM_DIR signals that the application must enumerate the directory or subtree to compute current state. The result is not “nothing changed.” It is “the event history is no longer complete.”

Make the recovery path explicit:

Mark the watched scope dirty and stop applying assumptions based on the incomplete event sequence.
Enumerate the relevant directory scope and rebuild or compare the derived index.
Replace stale state atomically where possible, or version the index so readers cannot observe a half-rebuilt result.
Re-arm notification monitoring and perform a second reconciliation if there was a gap between the scan and the re-armed watch.
Emit a metric for overflow and reconciliation duration so repeated loss is visible and buffer/worker capacity can be adjusted.

Even a larger buffer only reduces the probability of overflow; it does not make the stream durable. The ReadDirectoryChangesW contract imposes an important network limitation: a buffer larger than 64 KB fails for directories monitored over the network. A remote share can also have different latency and failure behavior from a local NTFS volume. Use a supported journal or a server-side change feed if the application requires a recoverable history across downtime. For NTFS-specific volume tracking, the USN change journal is a distinct facility with its own journal identity, cursor, and wraparound handling; it is not a drop-in assumption that every filesystem or share provides the same semantics.

Reconciliation and correctness boundaries

Notifications can be coalesced by the filesystem and do not promise that a consumer sees each write as a separate record. A “modified” action means the watched object changed in a relevant way; it does not establish that a writer has finished publishing a multi-step file. Applications that consume generated files should use a publishing protocol, such as writing to a temporary name and renaming only after content is complete, then validate the file before indexing it. This separates “the name appeared” from “the content is complete and acceptable.”

Keep path handling defensive. A notification name is relative to the watched directory. Combine it with the canonical watch root using a path library, then enforce the application’s own containment policy before opening it. Do not assume a notification makes a later open race-free: the file may have been renamed, replaced, deleted, or redirected by a reparse point after the event arrived. Open the object and inspect the resulting handle if identity matters. If the watcher monitors untrusted trees, define whether reparse points are followed and prevent traversal outside the intended root.

For FindFirstChangeNotification, the system signals that a selected kind of change occurred but does not provide the changed name or action; the application must enumerate to learn current state. Microsoft documents FindFirstChangeNotification and ReadDirectoryChangesW as alternative notification choices, not APIs to combine on the same design for duplicate certainty. Choose the simpler signaling mechanism when a full refresh is already acceptable; use ReadDirectoryChangesW when action/name details reduce work but still retain reconciliation.

Operational test matrix

Test create, write, close, rename, delete, and directory-tree changes under realistic load. Include a burst large enough to cause loss, slow down the event consumer deliberately, and confirm the full-reconcile path repairs its index. Test a path that is removed or renamed while a request is pending, a share interruption if network paths are supported, and process shutdown while overlapped I/O is outstanding. Verify that every error path closes handles exactly once and that cancellation completions are drained before buffers are released.

Instrument the watcher with counters for issued reads, completed records, parse failures, buffer overflows, queue saturation, rescans, and failed re-arms. A quiet watcher is not necessarily healthy: the process may have stopped monitoring or lost its directory handle. A health check should verify that the watch is active and that a periodic reconciliation can complete within the expected time.

Use ReadDirectoryChangesW as a timely invalidation channel, not as an unquestioned source of truth. Correctness comes from safe buffer parsing, explicit asynchronous lifetime management, visible overflow recovery, and reconciliation with the directory’s actual state.

Related:

Sources: