Fixing an NFS Mount That Hangs Instead of Failing on FreeBSD

FreeBSD NFS mounts are hard by default. If a server becomes unresponsive, new and outstanding filesystem operations wait until it returns instead of turning an uncertain remote write into an ordinary local error. That behavior protects many workloads from silent partial failure, but it can block shells, service shutdown, filesystem walks, and monitoring. The repair starts with restoring or isolating the failed path, not reflexively converting every mount to soft.

Prove which process and mount are blocked

Do not run find /, df, or broad inventory tools during the incident; they may touch the unavailable mount and create more blocked processes. Start with process and kernel-stack views:

ps axl
procstat -kk PID
mount -t nfs
nfsstat -m

Replace PID with one affected process. A kernel stack containing NFS/RPC wait functions ties the sleep to the client path more reliably than grepping ps for one state letter. nfsstat -m reports effective mount options, including negotiated protocol and transport, which may differ from an administrator’s memory of /etc/fstab.

Identify the exact mount point without dereferencing files inside it. Then stop scheduled jobs, automounters, backups, and health checks that repeatedly touch it. SIGKILL cannot complete while a process is in an uninterruptible kernel wait; sending it repeatedly only marks the process to exit when the operation returns.

Distinguish an unavailable server from a blocked protocol path

Test network reachability without assuming ICMP proves NFS:

route -n get SERVER_IP
ping -c 3 SERVER_IP
nc -vz SERVER_IP 2049
nfsstat -c

Use the server’s real address. A ping reply says nothing about TCP port 2049 or the NFS daemon. For NFSv4, the client uses TCP and normally talks directly to NFS on 2049. NFSv2/v3 mounting can also involve rpcbind and mountd; rpcinfo -p and showmount -e are relevant there but should not be treated as universal NFSv4 health checks.

Inspect client logs for “server not responding” and “is alive again,” then correlate with server and firewall timestamps. On the server, verify exports, nfsd, storage health, thread saturation, RPC/TCP queues, and any backing filesystem pause. A reachable server can still stop answering because its own disk or ZFS pool is blocked.

Packet capture establishes where requests stop:

tcpdump -ni em0 host SERVER_IP and port 2049

Repeated client retransmissions without replies implicate the server, return path, or firewall. Replies reaching the interface but not completing RPC require deeper client/session analysis. No outbound packet may indicate routing, a local firewall, or a client already waiting on state that is not issuing a new request.

Read hard, soft, and intr as data-integrity policies

With the default hard mount, operations keep retrying. This is appropriate for writable home directories, databases, and application data where returning EIO during an ambiguous write can cause software to proceed with an incomplete view.

A soft mount makes filesystem calls fail after the configured retransmit count. The mount_nfs(8) manual warns that NFS ordering and lock behavior make soft and intr unsafe for many versions and workloads. A failed RPC does not always tell an application whether the server committed the operation. Use soft mounts only for explicitly disposable or read-mostly data whose applications are designed to handle I/O errors and uncertain results.

intr permits a delayed call to fail with EINTR when a termination signal arrives. It is not simply “hard mount safety plus Ctrl-C.” The manual recommends nolockd with intr to avoid indeterminate server lock state and strongly recommends hard, non-interruptible mounts for NFSv4.0. NFSv4.1/4.2 sessions handle some ordering cases better, but the documentation still does not call the combination completely correct.

Do not paste this into production merely to stop a hang:

soft,intr,retrans=3,timeo=30

retrans controls round-trip retry count for soft mounts, while timeo is expressed in tenths of a second and is primarily a transport tuning control. Arbitrarily shrinking both can turn transient latency into application errors and retry storms.

Separate boot-time mount retries from active I/O behavior

By default, mount_nfs also retries an initial mount indefinitely, which can stall boot when a noncritical server is absent. The bg option moves retrying to the background after an initial failure; bgnow does so immediately and avoids the first long wait; retrycnt bounds mount attempts. These options affect establishing the mount, not the semantics of file operations after a mounted server disappears.

For optional data, a release-appropriate /etc/fstab policy can combine a hard mount with background initial mounting. Test the exact entry with mount -a in a maintenance context and confirm dependent services do not assume the path exists immediately. Critical cross-mounted servers need special care to avoid boot dependency cycles.

Recover service before forcing an unmount

The cleanest recovery is to restore the server, network, export, and backing storage. Waiting hard operations then complete, pending signals take effect, and applications can close files consistently. Verify the “server is alive again” message and decreasing RPC retransmissions.

If the mount must be detached, stop applications and change directories out of it first. A normal unmount may block because vnodes are busy. Forced unmount is a last resort and can leave applications with errors and uncertain remote state; it does not repair the server. Preserve logs and know what was writing before using it.

After recovery, compare the incident with nfsstat client counters and server telemetry. High retransmissions during ordinary healthy periods may indicate loss, MTU problems, overload, or undersized server resources. Do not tune read/write sizes or timeout solely from one outage caused by a powered-off server.

Design and verify the long-term policy

Classify each mount by data criticality, read/write behavior, boot criticality, protocol version, locking, and how its applications handle EIO or EINTR. Keep hard semantics for data that must not become ambiguous. Use bg/bgnow for optional boot mounts where appropriate. Reserve soft behavior for cases with an explicit failure contract.

Monitor NFS RPC retransmissions, server latency, mount availability, and blocked process counts without having the monitor itself hang indefinitely on the mount. Rehearse server restart, firewall loss, and client shutdown. An NFS hang is often expected policy reacting to an unexpected outage; a durable fix addresses both halves.

Related:

Sources: