Skip to content
LinuxFix Published Updated 3 min readViews unavailable

Fixing inotify Watch Exhaustion Without Blindly Raising Kernel Limits

A forensic workflow for Linux inotify exhaustion: identify watch owners, distinguish instances and queue overflow, reduce scope, size limits, and retest.

“No space left on device” from a file-watching tool can occur while the disk has free space. inotify accounts watches, instances, and queued events against kernel limits; one developer tool scanning dependency trees or many per-user services can exhaust them. Raising a sysctl without finding the owner may hide a leak and increase unswappable kernel memory use.

Identify which limit failed

Inspect the current per-user limits:

sysctl fs.inotify.max_user_watches \
       fs.inotify.max_user_instances \
       fs.inotify.max_queued_events

max_user_watches limits the number of watched paths associated with a real user ID. max_user_instances limits inotify instances, usually file descriptors created with inotify_init1(). max_queued_events sizes an instance’s event queue; overflow is reported to the application as IN_Q_OVERFLOW and means it must rebuild state.

The observed error and application logs should determine which resource is involved. A program opening a new instance for every directory has a different bug from one instance legitimately watching a huge tree. Queue overflow under a change storm is not repaired merely by allowing more persistent watches.

Find owners through /proc

inotify descriptors appear in process fdinfo. As root, inventory them without dumping unrelated process environment or secrets:

for f in /proc/[0-9]*/fdinfo/*; do
  grep -q '^inotify' "$f" 2>/dev/null || continue
  printf '%s\n' "$f"
done

Count inotify lines per process and map PIDs to commands with ps. Tools such as editors, language servers, sync clients, container managers, and recursive reloaders are common users, but do not kill by reputation—prove which process and UID owns the descriptors. A container can still charge host kernel resources to the relevant user namespace/accounting context.

Inspect its configured roots. Watching a repository root that includes generated build trees, package caches, .git objects, virtual environments, and node dependencies can multiply demand. Exclude directories that do not affect the workflow, consolidate duplicate watchers, and update a leaking application before changing the host ceiling.

Size a measured increase

If legitimate peak demand remains near the configured limit, calculate headroom from observed counts, concurrent users, and workload growth. Apply a temporary value first:

sudo sysctl -w fs.inotify.max_user_watches=262144

The example is not a universal recommendation. Each watch consumes kernel memory whose size depends on architecture and kernel implementation. Very large limits let a runaway program pin more memory. Monitor slab/kernel memory and place service-level constraints around untrusted multi-tenant workloads.

After a successful test, persist the chosen value in a documented sysctl drop-in such as /etc/sysctl.d/60-inotify.conf, then load it with the distribution’s supported mechanism. Search for competing files that set the same key; last-writer ordering can make a correct value appear to revert.

Account for service identity when comparing observations. The limits are charged per real user ID, so many systemd units running as the same shared account contribute to one pool even when their cgroups are separate. Conversely, a developer’s graphical session may contain several helpers under one UID. Record UID, process, instance count, and watch count together; a container or cgroup name alone cannot explain which ceiling the kernel is enforcing.

Verify recovery semantics

Restart the affected application so it can create a clean instance, then reproduce startup, large-tree traversal, branch switches, dependency installation, and change bursts. Confirm its watch count stabilizes and events are processed. Force or simulate queue overflow in a test environment and verify the application performs a full rescan; once events are dropped, it cannot reconstruct truth from later incremental messages alone.

The durable fix is the smallest combination of reduced scope, corrected ownership/lifecycle, and measured headroom. A higher number is justified only after the watch graph explains where the memory and events go.

Related:

Sources:

Comments