Spotlight Internals: How macOS Indexes and Searches Your Files

Spotlight answering a search across an entire drive in a fraction of a second only looks like magic because the expensive work already happened, continuously, in the background, well before you pressed Cmd+Space. The actual search at query time is just a lookup against a pre-built index — the interesting engineering is in how that index gets built and kept current in the first place.

mds and mdworker: the indexing pipeline

mds (metadata server) is the daemon that owns Spotlight’s index and coordinates what needs (re)indexing. The actual work of extracting metadata from a given file is delegated to mdworker processes — short-lived, sandboxed helper processes, spawned per batch of files, that invoke the right metadata importer for each file type and hand the extracted metadata back to mds.

ps aux | grep mdworker
mdutil -s /

Splitting extraction into separate mdworker processes rather than doing it all inside mds itself is a deliberate isolation choice — a crash or hang parsing one malformed file’s metadata takes down a disposable helper process, not the central indexing daemon everything else depends on.

Metadata importers: format-specific extraction plugins

A metadata importer is a plugin (a .mdimporter bundle) that knows how to extract searchable attributes from one specific file format — an image importer extracts EXIF data and dimensions, a PDF importer extracts text content and author metadata, a source-code importer might extract symbol names. Every file type indexed by Spotlight has (or falls back to) an importer responsible for deciding what’s actually searchable about it.

mdimport -L
# lists every registered metadata importer and the UTIs it handles

mdimport -d2 somefile.pdf
# runs the appropriate importer against a single file and shows what it extracted

Third-party applications can ship their own .mdimporter bundles to make their proprietary file formats’ contents searchable through the same system search — which is why, for instance, some professional creative applications’ project files are meaningfully searchable by content, not just filename.

The index and mdfind

The extracted metadata lands in a per-volume index (historically visible as a .Spotlight-V100 directory at a volume’s root), queried through the same predicate-based query language both the Spotlight UI and the command-line mdfind use directly:

mdfind "kind:pdf invoice"
mdfind "kMDItemAuthorNames == 'Daniel Cosenza'"
mdfind -onlyin ~/Documents "budget"

mdls dumps every metadata attribute Spotlight has indexed for a specific file — useful both for debugging why a file isn’t turning up in search, and for discovering what attributes a given importer actually extracts:

mdls ~/Documents/report.pdf

Excluding content from indexing

Privacy Preferences’ Spotlight pane, or the command-line mdutil, control whether a given volume is indexed at all — useful for external drives holding content you’d rather not have indexed and cached in Spotlight’s metadata store:

sudo mdutil -i off /Volumes/ExternalDrive
sudo mdutil -E /Volumes/ExternalDrive     # erase and rebuild that volume's index

Live updates via FSEvents

Spotlight’s index doesn’t rely on periodic full rescans — it subscribes to FSEvents, the kernel-level filesystem change notification mechanism, so mds learns about a new or modified file essentially as soon as it’s written, and can queue just that file for (re)indexing rather than rescanning entire directory trees on a schedule.

log stream --predicate 'process == "mds"'

Core Spotlight: exposing app-specific data

Beyond files, apps can index their own in-app content (notes, messages, custom records) into Spotlight’s search via the Core Spotlight framework, making content that never existed as a standalone file on disk still show up in system-wide search — the mechanism behind being able to search for an in-app note or a specific contact directly from Spotlight without opening the app that owns that data first.

Why the architecture looks like this

The overall design — a persistent background daemon, disposable per-file worker processes, pluggable format-specific importers, an event-driven trigger via FSEvents instead of polling, and a query-time index lookup instead of a live filesystem scan — is what lets Spotlight answer “find every PDF mentioning this term” against an entire drive in well under a second. All the expensive work (deciding what’s inside every file) happens incrementally, in the background, as files change; the only thing that has to happen fast at query time is looking a term up in a structure that was already built for exactly that.