Fixing 'Disk Full' on Linux When df Shows Space Available
df says you have plenty of free space, but every write fails with ENOSPC. Two completely different root causes look identical from the outside — here's how to tell them apart.
An application fails to write, logging (or reporting directly) “No space left on device,” but df -h clearly shows gigabytes of free space on the filesystem in question. This confusing symptom almost always traces back to one of two distinct causes, and telling them apart takes about thirty seconds.
Cause 1: you’re actually out of inodes, not space
Every file, no matter how small, consumes one inode — a fixed-size metadata structure allocated when the filesystem was created. A directory tree with an enormous number of tiny files (a cache directory, a mail spool, session files) can exhaust the available inode count while leaving most of the actual disk space untouched.
df -iFilesystem Inodes IUsed IFree IUse% Mounted on
/dev/sda1 1310720 1310720 0 100% /If IUse% is at or near 100% while df -h’s regular space usage looks fine, this is your problem. Find the directory responsible:
for dir in /var/* /tmp/* /home/*; do
echo "$dir: $(find "$dir" -xdev | wc -l)"
done | sort -t: -k2 -rn | headThere’s no way to add inodes to an existing filesystem without reformatting it with a different inode ratio (mkfs -N) — the practical fix is deleting or relocating the offending small files, not just freeing space.
Cause 2: a deleted file is still open by a running process
On Linux, deleting a file with rm only removes its directory entry — if a process still holds the file open, the underlying disk blocks aren’t actually freed until that process closes the file or exits. A long-running process (a log writer that was never told to reopen its file after log rotation, most classically) can hold gigabytes of “deleted” data that df still counts as used, while ls/du on the directory shows nothing because the directory entry is already gone.
lsof +L1lsof +L1 specifically lists open files with a link count of zero — files that have been deleted but are still held open by a process, which is exactly the signature of this problem. The output shows the PID holding it and, often, an approximate size:
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NLINK NODE NAME
rsyslogd 892 syslog 3w REG 259,1 8589934592 0 1234 /var/log/syslog (deleted)Restarting or signaling that process to reopen its log files (many daemons support SIGHUP for exactly this) releases the space immediately without needing a reboot:
systemctl restart rsyslogTelling the two apart quickly
df -i at or near 100% points to cause 1 (inode exhaustion); df -h showing high usage with lsof +L1 showing large deleted-but-open files points to cause 2. They require entirely different fixes, which is why diagnosing correctly before acting matters — deleting more files won’t help at all if the real problem is an already-deleted file a process is still holding open, and restarting services won’t help if the real problem is inode exhaustion from millions of small files.
Preventing recurrence
For cause 1, application-level log rotation and cache cleanup (logrotate, periodic tmpfile cleanup via systemd-tmpfiles) prevents small-file accumulation from ever reaching inode limits. For cause 2, configuring log rotation to signal the writing process (copytruncate, or a proper reopen signal) rather than just renaming the file out from under it avoids leaving orphaned open file handles in the first place.