NTFS USN Change Journal: Incremental File Tracking Without False Guarantees

The NTFS Update Sequence Number change journal gives backup tools, indexers, and synchronizers an efficient way to discover which filesystem objects changed. NTFS appends records to a journal associated with the volume, letting a consumer resume near its previous position instead of walking every directory after each interruption.

The journal is a bounded change feed, not an eternal audit trail. Its records describe reasons and file-reference relationships, not the complete old and new content, final pathname, or user who caused an operation. A correct consumer combines a checkpoint with a journal identifier, detects discontinuity, and falls back to a full scan when its history is no longer trustworthy.

USNs order records within a journal instance

Each record carries an update sequence number, or USN. A consumer saves the position after the last record it incorporated and asks for later records on the next run. FSCTL_QUERY_USN_JOURNAL reports the current journal state, while FSCTL_READ_USN_JOURNAL reads records beginning at a requested USN.

The volume handle for control operations is opened using a path such as \\.\C: with appropriate access. Microsoft documents these journal operations as requiring administrator privileges. A service should run with only the rights it needs, keep the volume handle private, and treat an authorization failure as loss of incremental coverage rather than silently declaring no changes.

USN values give an ordering mechanism for this feed. They are not wall-clock timestamps and should not be merged numerically across volumes. Store the volume identity and checkpoint together.

The journal ID validates the checkpoint

NTFS associates a 64-bit identifier with a journal. It changes the identifier when existing records are or may be unusable, including journal recreation and some volume-version transitions. Microsoft requires consumers to use both the record USNs and the current identifier.

A durable checkpoint should contain at least:

volume identity
UsnJournalID
next USN to read
consumer schema version
time the checkpoint became durable

At startup, query the journal again. If its ID differs from the saved ID, the old checkpoint cannot establish continuity. If the saved position is older than the journal’s first available USN, required records were trimmed. Either condition demands a full reconciliation scan before a new incremental baseline is trusted.

Do not update the checkpoint before the index or backup state it represents is durable. Commit processed changes and the next USN together when possible. After a crash, replaying a few records is safer than advancing past changes that were never committed.

Records identify objects more reliably than paths

A USN record includes a file reference number and a parent reference, a name, reason flags, and versioned metadata. File reference numbers let a consumer correlate an object across some renames without depending exclusively on its old pathname.

They do not provide an immutable global identity. NTFS can eventually reuse file records, and a deleted object’s parent chain may no longer resolve. Maintain a mapping built from a baseline enumeration, validate reference sequence information where exposed by the record format, and expect some events to require targeted rescanning.

Rename activity is represented through old-name and new-name reasons rather than one atomic record containing two full canonical paths. Other changes can occur between those records. Pair only when the ordering and object identity support it, and degrade to remove-plus-add or rescan when correlation is ambiguous.

Reason flags can accumulate

NTFS may accumulate reason flags for repeated changes while a file remains open and emit a close indication when the handle closes. A record can therefore represent a set of reasons rather than one high-level user action. Consumers should test flags with bitwise logic and remain tolerant of reasons they do not need.

A data-overwrite or data-extend reason says data changed; it does not contain the changed bytes. Security, attribute, hard-link, reparse-point, stream, and rename reasons likewise report categories, not a reversible operation log. A backup tool still has to open and copy a stable version of the file, often using a snapshot strategy for application consistency.

Do not assume every record has the same structure version. Windows defines several USN_RECORD versions. Walk returned buffers using each record’s declared length and major version, validate bounds before accessing fields, and skip or reject unsupported versions according to a documented compatibility policy.

The journal is deliberately finite

Journal configuration includes a maximum size and an allocation delta. NTFS trims old data at maintenance checkpoints as the journal grows; actual size can move beyond the target before trimming occurs. A busy volume can overwrite a paused consumer’s required range.

Capacity planning therefore starts with peak change rate and the longest legitimate consumer outage. Monitor the distance between the consumer checkpoint and the oldest available USN. Alert before the gap closes, while there is still time to resume or schedule a full scan.

Increasing journal size extends the window but consumes disk space and does not eliminate discontinuity. Deleting or recreating the journal is disruptive to every dependent indexer and backup consumer. Microsoft’s fsutil usn exposes administrative operations, but they should be treated as planned changes with an inventory of downstream recovery costs.

A baseline and tail need a handoff point

An incremental index begins with a consistent baseline. One practical design queries a journal position, enumerates the volume’s file records with supported control codes, then reads forward from the captured point to incorporate changes that occurred during enumeration. The precise handoff must handle objects created, renamed, and deleted while the baseline is built.

Filesystem snapshots can simplify content consistency but do not remove the need to match journal and volume state. Record which live or snapshot volume produced each identity and USN. Never combine a snapshot’s file tree with an unrelated current journal and claim one continuous view.

After baseline construction, test a matrix of creates, overwrites, truncates, hard links, alternate streams, directory renames, cross-directory moves, deletions while open, and volume dismounts. Stop the consumer long enough to force journal wrap and verify it refuses incremental continuation.

It is not a security audit log

The change journal does not reliably answer who performed an operation, which process did it, what old bytes contained, or whether an access was authorized. Administrators can manage the journal, and finite retention removes history. Use Windows auditing, ETW, application logs, and security controls for attribution according to the threat model.

The journal is excellent at narrowing work: “these file records may have changed since this durable point.” Its consumer must turn that hint into current truth by opening objects safely, handling deletion and rename races, and reconciling after a gap.

A trustworthy USN-based system makes discontinuity visible. It persists journal ID and next USN with its own state, validates both before reading, parses versioned records defensively, watches retention headroom, and has a tested full-scan path. The speedup is real precisely because the system never pretends a bounded feed is evidence it no longer possesses.

Related:

Sources: