bhyve: FreeBSD's Native Type-2 Hypervisor

Where jails share one FreeBSD kernel across isolated userlands, bhyve runs complete virtual machines with their own kernels. It entered the FreeBSD base system in 10.0-RELEASE and uses the processor’s hardware virtualization facilities instead of translating every guest instruction in software. The current FreeBSD Handbook lists Intel processors with Extended Page Tables, AMD processors with Rapid Virtualization Indexing or Nested Page Tables, and supported ARM64 systems as host platforms. Guest and device support still varies by architecture, so the Handbook and the host’s own manual pages should be checked before designing a production fleet.

That distinction from jails is operationally important. A jail can be extremely dense because it does not emulate a machine or boot another kernel. A bhyve guest costs more memory and has a larger lifecycle, but it can run FreeBSD, OpenBSD, Linux, Windows, or another supported operating system whose kernel is independent of the host. A kernel failure inside the guest does not become a host kernel failure merely because both systems are FreeBSD.

What bhyve actually is

bhyve is deliberately small compared with a full virtualization product. The vmm(4) kernel facility owns the privileged virtualization machinery, while bhyve(8) is the userland process that defines a VM’s CPUs, memory, firmware, and virtual devices. A live VM also has a device below /dev/vmm, which gives administrators another way to distinguish configured disk images from machines that are actually running.

Before loading anything, verify the running host rather than assuming that the hardware or kernel matches an installation checklist:

freebsd-version -kru
uname -m
grep -E 'VT-x|Features2' /var/run/dmesg.boot
kldload vmm
ls -l /dev/vmm

The Handbook specifically describes EPT and UG in the Intel VT-x line and POPCNT in the AMD Features2 line as useful indicators. Those strings are a compatibility check, not a benchmark and not proof that firmware has left virtualization enabled. A missing capability should be resolved in firmware and against vmm(4) documentation instead of worked around with an unverified loader tunable.

Each VM is visible as a normal host process, so ps, top, procstat, and signals remain useful. There is one lifecycle trap: destroying a VM with bhyvectl --destroy --vm=name is equivalent to an abrupt power cut. The Handbook recommends signaling the bhyve process with TERM for an ACPI shutdown when the guest supports it. Destruction is for a guest that is already stopped or irrecoverably wedged.

Building a minimal VM by hand

The clearest way to learn bhyve is to assemble one command line before adopting a manager. The following FreeBSD-oriented example uses bhyveload to place a loader in guest memory, then starts the virtual hardware. Device and disk names are examples; creating or overwriting a zvol is intentionally outside the example.

bhyveload -m 1024 -d /dev/zvol/tank/vms/guest1 guest1

bhyve -c 2 -m 1024 -H -A \
  -s 0,hostbridge \
  -s 2,virtio-net,tap0 \
  -s 3,virtio-blk,/dev/zvol/tank/vms/guest1 \
  -s 31,lpc \
  -l com1,stdio \
  guest1

Each -s slot,device option creates a PCI function at an explicit slot: virtio-net for the network adapter, virtio-blk for the disk, and lpc for legacy devices such as the serial console. -c 2 assigns two virtual CPUs, -m 1024 assigns memory in megabytes, -H allows a virtual CPU thread to yield when the guest is idle, and -A generates ACPI tables. The exact accepted syntax is versioned with the base system, which is why production scripts should be checked with the host’s man bhyve after an OS upgrade.

Explicit slots improve reproducibility: adding a new disk does not have to reorder every existing guest device. They also make mistakes visible. Reusing a slot, changing a disk backend without understanding guest naming, or attaching the installer ISO on every boot can all produce confusing guest behavior. Store the complete command or generated configuration under version control, not in shell history.

Networking: tap and bridge

A virtio-net adapter backed by tap(4) terminates in a host interface. Bridging that interface with a physical NIC places the guest on the same Ethernet segment as the host. The following is a temporary laboratory setup; persistent production configuration belongs in /etc/rc.conf, normally managed with sysrc.

ifconfig tap0 create
ifconfig bridge0 create
ifconfig bridge0 addm em0 addm tap0
ifconfig bridge0 up

Do not assign the host’s IP address to the TAP member. The host address normally remains on the physical interface or, depending on the site’s bridge design, is moved deliberately to the bridge. Packet filtering must also account for bridged traffic and the guest’s exposure: a new VM on a bridge is a new machine on that network, not a process hidden behind the host firewall. ifconfig bridge0 and ifconfig tap0 should be part of the acceptance test, followed by connectivity from both directions.

NAT is another valid design, but it is a firewall and routing configuration rather than a bhyve feature. Keeping that boundary clear prevents a management wrapper from becoming the undocumented owner of host networking.

UEFI guests and vm-bhyve

Modern non-FreeBSD guests are normally booted with UEFI firmware from the bhyve-firmware package. Confirm the installed package and firmware path on the host because packages can provide multiple firmware variants, including framebuffer-capable images:

pkg install bhyve-firmware
pkg info -l bhyve-firmware

One commonly installed path is /usr/local/share/uefi-firmware/BHYVE_UEFI.fd:

bhyve -c 2 -m 2048 -H -A \
  -s 0,hostbridge \
  -s 2,virtio-net,tap0 \
  -s 3,virtio-blk,/vms/linux-guest.img \
  -s 29,fbuf,tcp=0.0.0.0:5900 \
  -s 30,xhci,tablet \
  -s 31,lpc \
  -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \
  linux-guest

The framebuffer listens on TCP port 5900 in this example. Binding it to 0.0.0.0 exposes it on every host address, so a real deployment should bind to a protected management address, filter it, or tunnel it. The XHCI tablet supplies absolute pointer coordinates that make graphical installers usable. Neither device adds authentication to VNC by itself.

For repeated operations, a manager reduces transcription errors. The FreeBSD Handbook lists vm-bhyve, CBSD, bmd, and other tools available through packages and ports. A typical vm-bhyve flow is:

vm create -t debian -s 20G linux-guest
vm install linux-guest debian-installer.iso
vm start linux-guest
vm console linux-guest

Those commands are management-layer syntax, not aliases for base-system utilities. Templates, datastore locations, switches, and autostart behavior belong to that manager’s documentation. Understanding the underlying bhyve command is still valuable when a wrapper reports only that a guest exited.

Device passthrough

PCI passthrough assigns a host PCI function to a guest through the IOMMU. It requires platform support, firmware configuration, and a device that the host can relinquish safely. The ppt(4) driver must claim the function before its ordinary host driver. A traditional loader configuration identifies devices by bus/slot/function:

# /boot/loader.conf: detach the device from host drivers before boot
pptdevs="2/0/0"

bhyve ... -s 6,passthru,2/0/0 ...

The exact identifier must be derived from pciconf -lv, and the host must retain a separate console, storage path, and management NIC. Passing through the only NIC or display device can make a remote host unreachable. IOMMU grouping and device reset behavior also determine whether a device is safe to reuse after a guest exits; passthrough should be tested across cold boot, guest reboot, and forced termination before production use.

Storage, failure boundaries, and observability

A disk image file is easy to copy, while a ZFS zvol provides a fixed-size block device and can participate in ZFS snapshots. Neither automatically produces an application-consistent backup. A snapshot taken while a guest database is writing can be crash-consistent at best unless the application is quiesced or a guest-aware backup protocol is used. The host must also avoid mounting a filesystem that is simultaneously active inside a guest.

Virtio devices normally reduce emulation overhead, but the guest needs the appropriate drivers. AHCI emulation can be useful during installation or for compatibility, while virtio is generally preferred once supported. Changes should be benchmarked with the actual workload rather than inferred from the device name.

When a VM exits, inspect the bhyve exit status, host logs, and guest console together. bhyvectl --get-stats --vm=name can expose virtual-machine statistics on versions that support the requested counters, while ls /dev/vmm confirms whether the kernel still considers the VM instantiated. Cleanup must never blindly destroy every entry below /dev/vmm on a multi-tenant host.

Choosing between bhyve and jails

The practical dividing line is workload architecture. Jails are ideal when sharing the host kernel is acceptable and density matters. bhyve is appropriate when the workload requires another kernel, a VM-specific boot sequence, virtual firmware, or hardware assignment. Neither choice eliminates security engineering: a bhyve process, its management daemon, TAP interfaces, disk permissions, and exposed consoles remain part of the host attack surface.

bhyve being in the base system is valuable because the kernel interface and userland tool are released together. It does not remove the need for capacity planning, guest patching, backups, fencing of management interfaces, or a tested shutdown path. A production-ready VM is the whole system around the bhyve command, not merely a process that reached its login prompt.

Related:

Sources: