Windows ProjFS: Building a Filesystem Projection Provider That Hydrates on Demand

A source-control workspace can contain millions of files even though one build touches only a fraction of them. Downloading and creating every file wastes network, disk, metadata, and antivirus work. Windows Projected File System, or ProjFS, lets a user-mode provider present a large logical tree while materializing directory entries and file bytes only when applications demand them.

ProjFS is a provider API, not a complete synchronization product. Windows mediates access under a virtualization root and calls the provider for enumeration, placeholder metadata, or file data. The provider still owns the backing-store protocol, caching, authorization, conflict rules, integrity, and recovery.

The virtualization root anchors one projected namespace

The provider marks an empty or controlled NTFS directory as a virtualization root, then calls PrjStartVirtualizing() with callbacks and instance options. From then on, ProjFS routes relevant operations under that root to the provider.

Do not adopt an arbitrary directory containing user files. Existing full files, placeholders, tombstones, and provider state have distinct meanings. Installation should verify the volume supports the feature, the root belongs to this provider, and any persisted instance identity matches the expected backing store.

The provider process must remain available for operations that require information not already materialized. Design service startup, crash recovery, and upgrade before exposing the root as a dependable workspace.

Directory enumeration is a restartable protocol

ProjFS begins enumeration, requests one or more batches, and ends the enumeration through separate callbacks. The provider receives an enumeration identifier so it can retain cursor state across calls. A search expression may filter names, and the framework can stop when its output buffer fills.

An enumeration must be stable enough to resume. Sort entries with the comparison behavior ProjFS expects, preserve cursor state until the end callback, and handle a restarted scan. Do not retain an unbounded server response for every abandoned enumeration; attach size and idle limits to the state table.

Directory entries need accurate names, types, sizes, timestamps, and attributes. Incorrect case handling or unstable ordering can cause duplicates and missing results that appear only with particular Win32 enumeration patterns.

Placeholder metadata comes before file bytes

When Windows needs information for an item not yet represented locally, the provider supplies placeholder information. A placeholder records enough metadata for the namespace to behave like a file or directory without downloading all content.

File data arrives through a separate callback. The request identifies a byte range, and the provider retrieves that range from its backing store, writes it through PrjWriteFileData(), and completes the command. Buffer alignment and write constraints documented by ProjFS must be respected.

Do not claim bytes are present before their integrity is known. Verify the backing object identity, range, and expected version, then provide data. A partial network response or checksum mismatch is an operation failure, not an excuse to zero-fill a source file.

Identity keeps metadata and content coherent

ProjFS placeholders can carry provider-defined content identity and provider identity. Use content identity to bind the projected metadata to the exact backing revision whose bytes will later be served. If the remote object changes, update through supported placeholder operations and a conflict policy rather than silently hydrating new bytes under old metadata.

A robust identity includes an immutable object or revision key, not only a pathname. Renames and case changes make names unstable. Keep the encoded identity compact, versioned, and safe to parse because local placeholder data can outlive a provider upgrade.

Before serving a range, revalidate that the request still targets the same placeholder version. Cancellation, local modification, or deletion can race a slow download.

Local modifications change provider authority

An application can modify, replace, rename, or delete projected items. ProjFS distinguishes placeholder state from a hydrated or locally modified full file. The provider must not overwrite user changes simply because its remote catalog still contains an older object.

Register filesystem-operation notifications needed by the product and return veto decisions only where the API permits and the policy is justified. Notifications are not a universal transaction log. Maintain durable product state for uploads or conflicts, and rescan after any gap.

Define whether the tree is read-only, locally writable with later upload, or a disposable generated workspace. Each model requires different handling of dirty files, tombstones, remote deletion, and reset. Expose that model to users instead of surprising them during cleanup.

Callbacks must support concurrency and cancellation

Windows can issue multiple callbacks concurrently. Avoid one global lock around network I/O. Partition state by command, enumeration, or backing object and use bounded queues so a scan cannot starve an interactive file open.

A provider can complete eligible callbacks asynchronously. It returns a pending status, retains the command identifier, and later calls PrjCompleteCommand(). Cancellation may arrive before that completion. The worker must stop or discard results safely and must never complete one command twice.

Every pending request needs a deadline, byte budget, and shutdown behavior. On service stop, reject new work, cancel or finish outstanding downloads, and end virtualization only after callbacks can no longer access destroyed state.

ProjFS is not the Cloud Files API

ProjFS was designed around projected namespaces such as very large source trees. The Cloud Files API has different integration for sync roots, cloud placeholders, user-visible hydration policy, and Windows shell experiences. Choosing one because both use placeholders leads to missing platform behavior.

Use ProjFS when the provider controls a specialized namespace and can implement its own backing semantics. Use the cloud synchronization platform when the product promises a user cloud-drive experience. Review supported Windows editions, optional-feature deployment, and API minimum versions as part of product requirements.

Neither API removes the need to scan untrusted content. A lazily hydrated executable is still executable, and a projected path can be supplied to tools with their own parsing vulnerabilities.

Recovery needs a reconstructable cache

Treat the local projection as a cache plus user modifications, not as the only copy of remote state. Persist enough mapping to distinguish clean placeholders from local work. After a crash, reconcile the root, provider database, and backing revision before accepting destructive cleanup.

Test empty and huge directories, case-colliding remote names, range reads, memory-mapped access, antivirus scans, cancellation at every callback, provider crash during hydration, remote revision change, disk-full conditions, local rename, and Windows reboot. Measure time to first directory listing as well as full hydration throughput.

ProjFS can make an enormous tree feel local because Windows asks only for what a program touches. That illusion is reliable only when the provider treats enumeration as a protocol, binds content to stable identity, respects local state, bounds asynchronous work, and makes every interrupted operation recoverable.

Related:

Sources: