FreeBSD rtld Operations: Diagnose Shared-Library Resolution

When a FreeBSD executable reports that a shared object cannot be opened, the message names the final symptom, not necessarily the faulty layer. The object may not be installed, may exist outside the dynamic linker’s search path, may have the wrong ABI, or may be found while one of its own dependencies is missing. ld-elf.so.1, commonly called rtld, loads shared objects and resolves their symbols before the program’s entry point runs. A diagnosis should establish which object was requested, which search rule selected or rejected each candidate, and whether the binary and library belong to compatible ABIs.

This guide concentrates on native ELF runtime resolution. It does not assume that installing a library, changing LD_LIBRARY_PATH, or rebuilding the application is automatically correct. Those actions can hide a packaging defect, affect unrelated processes, or make a service resolve a different object than an interactive shell. Start by collecting the executable identity and its dependency metadata, then make a narrow, reversible change.

Capture the executable and runtime context

Record the exact executable path, owner, mode, architecture, FreeBSD release, and service launch context. A login shell and an rc.d service may have different environment variables, working directories, and ABI constraints.

freebsd-version -kru
file /usr/local/bin/example
stat -f '%N %Su:%Sg %Sp' /usr/local/bin/example
env | sort | grep '^LD_'

Replace the example path with the program that fails. Environment output can contain sensitive configuration; collect it only where authorized and redact it before sharing. For a daemon, inspect the service manager configuration and the real process environment instead of assuming your shell is representative. Do not add loader variables to /etc/rc.conf just because they make one interactive test pass.

The kernel starts the interpreter named by the ELF PT_INTERP header for a dynamically linked executable. The dynamic linker then loads required shared objects, performs relocations, and transfers control to the program. If the binary is static, changing shared-library search paths will not repair its unresolved symbols. Confirm the file is ELF and check its interpreter and dependencies using the release’s supported inspection tools.

Understand search order rather than guessing

FreeBSD rtld(1) documents an ordered search that includes DT_RPATH when DT_RUNPATH is absent, the executable’s DT_RPATH under the documented conditions, LD_LIBRARY_PATH, DT_RUNPATH, the hints file maintained by ldconfig, and built-in system directories such as /lib and /usr/lib. The exact order is significant: an earlier candidate can shadow a later, correct library. A library found in /usr/local/lib is not automatically available merely because it exists there; the hints file or an object run path may be what makes it discoverable.

The first safe comparison is to inspect the current hints paths without changing them:

ldconfig -r
ldconfig -r | grep -F '/usr/local/lib'

ldconfig -r reports the configured hints directories and scans for libraries in those paths. A directory missing from this output is evidence, not proof, that the failing lookup cannot reach it: an embedded run path or an environment variable could also contribute. Conversely, the directory being present does not prove the exact requested library exists there or has the expected ABI.

For a controlled test, set a process-local library path only for a harmless command and remove it immediately from the diagnostic plan:

env LD_LIBRARY_PATH=/opt/example/lib /usr/local/bin/example --version

This is a diagnostic experiment, not a production fix. It can cause a different library to load than the package manager intended. Do not set it globally in a user’s profile or service environment to mask a missing ldconfig registration. Set-user-ID and set-group-ID execution has special restrictions: the loader ignores or unsets several LD_ variables, and ldconfig hints are used for trusted library paths. Never depend on an environment override for privileged programs.

Verify dependency metadata and the actual loaded object

For a trusted, locally installed executable, ldd lists direct and indirect shared-object dependencies and their resolved paths:

ldd /usr/local/bin/example
ldd -a /usr/local/bin/example

Use the exact ldd(1) options supported by the installed release. A successful result means the loader could resolve the dependencies for this invocation’s environment; it does not guarantee that the program’s later dlopen(3) calls will succeed, that symbols match the consumer’s expectations, or that the service runs with the same environment. Compare the command output when run as the service account with a controlled service-level test.

Do not run ldd on an untrusted executable. Historically and on some systems, dependency inspection can execute code from the inspected object. Check the current FreeBSD ldd(1) security notes before analyzing a downloaded binary; use static ELF metadata inspection such as readelf -d from an installed toolchain or a trusted package-analysis workflow instead. A filename is not a trust boundary.

When ldd reports not found, inspect the consumer’s needed-object name and the library’s SONAME before changing paths. A program linked against libfoo.so.2 does not become compatible with libfoo.so.3 simply by adding a symlink. ABI compatibility is a contract between the library’s exported symbols and the consumer. Use the package database to find the owning package and expected version, and reinstall or rebuild the correctly matched package if files are missing or corrupted.

For symbol-resolution failures, distinguish an absent library from an undefined symbol within a library that did load. The former is a path or installation issue. The latter may indicate an ABI mismatch, a library replacement rule, an incomplete upgrade, or an application linked against a different library version. Preserve the full loader error and compare package versions, timestamps, and ldd output before making changes.

Use library maps only for an intentional compatibility case

/etc/libmap.conf can redirect one shared-library name to another, and rtld also supports process-local library-map controls. This is powerful and can affect every matching program on the host. Treat a mapping as compatibility policy with an owner, scope, rollback, and regression test, not as a generic way to make an error disappear.

Before changing a map, find out whether a package created it, which binaries match the source name, and whether the replacement exports the symbols and behavior they expect. A global mapping can break programs that were correct under the original library. Test the target application and a representative non-target application, inspect the resolved paths, and remove the mapping if its behavior is broader than intended. Do not use LD_PRELOAD or a map to inject a library into production as an improvised hot patch.

Repair the right layer

If a required package library exists in the intended standard directory but the hints file is stale, inspect /etc/ld-elf.so.conf and package-provided configuration under /usr/local/libdata/ldconfig. The boot-time service and package scripts normally maintain the hints file. Prefer repairing the package-owned registration or rerunning the supported ldconfig workflow after verifying its input directories. Do not hand-edit /var/run/ld-elf.so.hints; it is generated runtime state.

If a package was partially upgraded, use pkg info, pkg which, and package repository policy to restore a coherent set of files. Avoid copying a shared object from another host without checking release, architecture, ABI, package version, and dependencies. A successful copy can leave an untracked library that disappears at the next upgrade or creates a silent cross-version mixture.

When a local application embeds an obsolete run path, rebuild it with the intended install name or run path using its documented build system. Do not change the host-wide loader configuration to compensate for one incorrectly linked executable. For software in a jail, inspect the jail’s own filesystem and hints file; a host library path is not automatically present inside the jail.

Acceptance checks and operational record

Before declaring the incident repaired, verify the executable’s format and ABI, all direct and indirect dependencies, any dynamically opened modules used by the failing code path, and the service’s actual launch environment. Run the application under its normal account and startup mechanism. Include one functional check that exercises the code path that originally failed; --version alone may never load the problematic module.

Capture the before-and-after ldd output, relevant package versions, hints directory, configuration change, and rollback procedure. Record whether the change was per-process, package-owned, or system-wide. A clean command line in an administrator’s shell is not sufficient acceptance evidence for a daemon started at boot.

The core diagnostic rule is simple: distinguish the dependency name, the search path, the selected file, and ABI compatibility. Keep each observation separate. That prevents a local environment workaround from becoming an undocumented host-wide dependency.

Related:

Sources: