FreeBSD Jails: Lightweight OS-Level Virtualization Done Right

FreeBSD has shipped jails since the 4.x era. A jail extends chroot(2) with kernel-enforced restrictions on processes, credentials, host identity, networking, and visible system state. It is operating-system-level virtualization: jailed processes share the host’s FreeBSD kernel but execute inside a prison identified by a jail ID and a configured set of parameters.

That shared-kernel fact defines both the efficiency and the security boundary. A jail does not boot another kernel and cannot run an arbitrary foreign kernel. Kernel vulnerabilities, host resource exhaustion, unsafe device delegation, and overly broad jail permissions remain host concerns. “Root in a jail” is constrained, but it is not a magic substitute for host patching, least privilege, resource controls, and backups.

The core idea: partitioning, not emulation

A jail is a collection of processes attached to prison parameters. path establishes its filesystem root; host.hostname supplies its jail-visible hostname; address parameters restrict non-VNET networking; and allow parameters selectively delegate operations. The host can enumerate those prisons and enter one without treating it as a virtual machine console:

jls -v
jexec webserver /bin/sh
jls -n -j webserver

Jailed root normally cannot load kernel modules, administer the host, or signal arbitrary host processes. Some operations can be delegated with settings such as allow.mount, allow.raw_sockets, or device rules. Each delegation enlarges the attack surface and should be justified by an application requirement. A mount.devfs line without an appropriate devfs_ruleset can expose more device nodes than the workload needs.

Thick jails: an independently maintained userland

A thick jail contains its own copy of the FreeBSD base userland. The host kernel must remain compatible with that userland, but files below the jail root can be upgraded and configured independently. The current Handbook obtains base.txz from the official release directory and extracts it into a dedicated jail path. Pin the architecture and release intentionally rather than copying the host’s /bin and /lib while they are live.

mkdir -p /usr/local/jails/media /usr/local/jails/containers/webserver
fetch https://download.freebsd.org/ftp/releases/amd64/amd64/15.1-RELEASE/base.txz \
  -o /usr/local/jails/media/15.1-RELEASE-base.txz
tar -xf /usr/local/jails/media/15.1-RELEASE-base.txz \
  -C /usr/local/jails/containers/webserver --unlink
cp /etc/resolv.conf /usr/local/jails/containers/webserver/etc/resolv.conf
cp /etc/localtime /usr/local/jails/containers/webserver/etc/localtime

This example is release-specific, not a command to run unchanged forever. Verify the supported release, checksum policy, architecture, destination dataset, and available space. A jail userland newer than the host kernel is not a supported shortcut around a host upgrade.

The declarative lifecycle belongs in /etc/jail.conf:

webserver {
    path = "/usr/local/jails/containers/webserver";
    host.hostname = "webserver.example.test";
    ip4.addr = "192.0.2.50";
    exec.clean;
    exec.start = "/bin/sh /etc/rc";
    exec.stop = "/bin/sh /etc/rc.shutdown";
    mount.devfs;
    devfs_ruleset = 5;
}

192.0.2.0/24 is documentation space; substitute a real address already routed or configured on the host. Validate configuration and lifecycle from the host with jail -c webserver, service jail start webserver, jls, and service jail stop webserver. Do not start the same named jail through two independent managers.

Building a jail’s filesystem root

Thin jails and the template misconception

A thin jail shares much of a base userland through a controlled template. The Handbook documents both OpenZFS snapshot-based and NullFS-based approaches. They have different update semantics.

A ZFS clone is created from a particular snapshot. Later changes to the origin dataset do not magically flow into existing clones; copy-on-write sharing saves space, but each clone has its own subsequent state. Updating a template requires a deliberate snapshot, clone, promotion, replacement, or jail-specific update workflow.

With NullFS thin jails, read-only base directories can be mounted into each jail while writable configuration and data remain jail-specific. Updating the shared base can update many jails together, but it also couples their compatibility and maintenance window. Mount permissions and the writable overlay layout become part of the security model.

“Thin” therefore describes storage construction, not stronger process isolation. Thick and thin jails use the same kernel jail mechanism. Choose thick jails for independent userland lifecycle and simple failure domains; choose thin jails when shared base management is worth the coupling.

Thin jails and ZFS: don’t duplicate the base system

Host networking versus VNET

An address-restricted jail without VNET uses the host network stack. Parameters such as ip4.addr constrain addresses available to the jail, but routing tables and interfaces still belong to the host. This design is smaller and often sufficient for a service that needs one or two addresses.

A VNET jail gets a virtualized network stack with its own interfaces, routes, sockets, and firewall context. A common topology creates an epair(4), places one endpoint in a host bridge, and moves the other into the jail:

vnetweb {
    path = "/usr/local/jails/containers/vnetweb";
    host.hostname = "vnetweb.example.test";
    vnet;
    vnet.interface = "${epair}b";
    mount.devfs;
    devfs_ruleset = 5;
    exec.clean;

    $epair = "epair50";
    $bridge = "bridge0";

    exec.prestart = "/sbin/ifconfig ${epair} create up";
    exec.prestart += "/sbin/ifconfig ${bridge} addm ${epair}a up";
    exec.start = "/sbin/ifconfig ${epair}b inet 192.0.2.51/24 up";
    exec.start += "/sbin/route add default 192.0.2.1";
    exec.start += "/bin/sh /etc/rc";
    exec.stop = "/bin/sh /etc/rc.shutdown";
    exec.poststop = "/sbin/ifconfig ${bridge} deletem ${epair}a";
    exec.poststop += "/sbin/ifconfig ${epair}a destroy";
}

The bridge must exist and have a valid uplink design. Interface names and addresses must be unique. Cleanup hooks matter because a failed start can otherwise leave orphan epairs or bridge members. The Handbook’s complete VNET example should be adapted as one unit; copying only vnet.interface does not create the host-side topology.

VNET grants network autonomy, not physical separation. Traffic still crosses host interfaces and pfil hooks according to host configuration. Decide where filtering occurs and test inbound, outbound, DNS, default route, MTU, and teardown.

Resource limits: rctl

Jail namespace isolation does not stop a process from consuming host memory, CPU, descriptors, or process slots. RACCT gathers accounting data and RCTL applies rules. Confirm support on the running kernel before promising a limit:

sysctl kern.racct.enable
rctl -h
rctl -a jail:webserver:memoryuse:deny=1G
rctl -a jail:webserver:maxproc:deny=256
rctl

Rules can be made persistent in /etc/rctl.conf. The exact action supported by a resource and whether a limit applies per process or to the aggregate subject are defined by rctl(8). Test enforcement with a disposable jail; a rule accepted syntactically but scoped differently from the capacity model is not a control.

Monitor both consumption and denials. A memory cap can convert host exhaustion into repeated application failures, which is an improvement in blast radius but not service health. Reserve enough host memory for the kernel, ZFS ARC if present, network buffers, and jail management.

Updates, identity, and operational boundaries

Thick jails need base-system security updates as well as package updates. Their userland release and the host kernel must remain compatible. Thin jails need a documented template update procedure and a way to roll back all affected instances. Third-party packages inside each jail still require pkg audit and upgrades regardless of how the base was created.

Use jls -n to record effective parameters rather than trusting only the source configuration. Audit delegated allow.* settings, mounted filesystems, devfs rules, raw sockets, and host datasets visible inside the jail. Application data should live in a dataset with an explicit backup and snapshot policy, not anonymously inside a disposable root.

Jail IDs are runtime identifiers and can change. Automation should address configured jail names or documented identifiers, verify the target with jls, and stop on ambiguity. Destroying a dataset based only on a stale JID is unacceptable lifecycle automation.

Where jails still have an edge

Jails avoid hardware emulation and a per-guest kernel, which makes them efficient for FreeBSD services. That does not make their overhead literally zero: duplicated userlands, VNET stacks, filesystem layers, accounting, and workload contention all consume resources. Measure density with the actual service and failure policy.

Use a jail when a shared FreeBSD kernel is acceptable and a constrained process/userland boundary solves the problem. Use bhyve when the workload needs another kernel, virtual firmware, hardware passthrough, or a VM boundary. In both cases, the host remains a security-critical control plane.

Related:

Sources: