Building a Custom FreeBSD Kernel from Source

Most FreeBSD systems should run the GENERIC kernel. It covers common hardware, receives the broadest testing, and works naturally with release update tooling. A custom kernel is justified when a documented compile-time option, an embedded size constraint, driver development, instrumentation, or a site-specific module policy requires it. Removing drivers rarely produces a dramatic performance improvement and can remove the recovery path for replacement hardware.

A kernel build is also a version-integrity exercise. The kernel, its modules, userland tools, and externally built modules must agree on interfaces closely enough to operate together. Building from whatever branch happens to be at /usr/src is not a valid procedure.

Getting the source

Start by recording the installed and running versions and architecture:

freebsd-version -kru
uname -m
uname -K

freebsd-version -kru can reveal a newly installed kernel that is not yet running or a userland at a different patch level. uname -K reports the kernel ABI version used by package compatibility checks. Resolve unexpected differences before choosing source.

FreeBSD’s official Git repository is https://git.FreeBSD.org/src.git. Select the branch or release tag that corresponds to the intended system, following the Handbook’s branch guidance. For example, a supported release system tracks its matching releng/X.Y branch, not main or an arbitrary older example:

git clone --branch releng/15.1 https://git.FreeBSD.org/src.git /usr/src
git -C /usr/src status --short --branch
git -C /usr/src rev-parse HEAD

releng/15.1 is an example for 15.1-RELEASE systems. It is wrong for a 14.x, STABLE, or CURRENT host. Record the source commit with the deployment record so the installed kernel can be reproduced.

Starting from GENERIC

Kernel configurations live below /usr/src/sys/<arch>/conf. Because /usr/src can be replaced, the Handbook recommends storing the custom file elsewhere and symlinking it into the architecture directory:

cp /usr/src/sys/amd64/conf/GENERIC /root/MYKERNEL
ln -s /root/MYKERNEL /usr/src/sys/amd64/conf/MYKERNEL

Use the directory matching uname -m; amd64 is only an example. Keep the custom file in configuration management and comment why every deviation exists.

One maintainable style includes GENERIC and expresses only the difference. Confirm syntax and availability in the matching source tree’s GENERIC, NOTES, and config(5):

# MYKERNEL
include GENERIC
ident   MYKERNEL

# Enable resource limit accounting
options RACCT
options RCTL

# Kernel debugging
options         KDB
options         DDB
options         WITNESS
options         INVARIANTS

include GENERIC keeps the broad baseline, while nooptions and nodevice can remove specific entries. Removal needs stronger review than addition: storage, console, network, crypto, filesystem, and USB drivers may be essential only during recovery. WITNESS and INVARIANTS are diagnostic facilities with substantial overhead and are not routine production tuning switches.

Before compiling, inspect hardware and loaded modules with dmesg, pciconf -lv, kldstat, usbconfig, and the relevant driver manual pages. A driver currently built into GENERIC will not appear as a separately loaded module, so kldstat alone is not an inventory.

Building

The supported top-level build uses buildkernel and installkernel. Parallelism should fit available memory as well as CPU; the Handbook suggests choosing and measuring a reasonable -j value rather than blindly maximizing jobs.

cd /usr/src
make -j4 buildkernel KERNCONF=MYKERNEL
make installkernel KERNCONF=MYKERNEL

buildkernel builds the kernel and modules in the object tree. installkernel installs the new kernel below /boot/kernel and preserves the previous installed kernel as /boot/kernel.old. Every later install overwrites that single kernel.old generation, so also preserve a separately named known-good kernel before repeated experiments.

Kernel modules installed from packages or ports—graphics drivers and VirtualBox modules are common examples—must match the new kernel interfaces. The Handbook documents PORTS_MODULES in /etc/make.conf as one way to rebuild specified ports modules during kernel builds. Alternatively, rebuild or reinstall them deliberately before declaring the boot complete.

reboot

At the loader menu, select the previous kernel. At an interactive loader prompt, follow the version’s documented sequence; one explicit method is:

OK unload
OK load /boot/kernel.old/kernel
OK boot

Confirm /boot/kernel.old/kernel exists before relying on it. Keep boot media and console access for failures in storage or console drivers that occur before remote networking is available.

After a successful boot, verify more than uname -a:

freebsd-version -kru
sysctl kern.conftxt | sed -n '1,80p'
kldstat
dmesg -a | less

Test the boot filesystem, network, firewall, storage, jails, and third-party modules. A kernel that reaches a shell but lacks a backup controller driver is not accepted.

Building world alongside the kernel

If the source revision changed, the Handbook requires a matching buildworld before buildkernel. A complete source update has ordered phases and should follow the Handbook for the exact branch. The high-level shape is:

make -j4 buildworld
make -j4 buildkernel KERNCONF=MYKERNEL
make installkernel KERNCONF=MYKERNEL
shutdown -r now
make installworld
etcupdate
make check-old
make delete-old
shutdown -r now

This is an outline, not a substitute for /usr/src/UPDATING and the Handbook. etcupdate(8) is the current configuration merge tool. Administrators migrating from mergemaster should bootstrap etcupdate’s database as documented before the upgrade.

Obsolete libraries require special care. make check-old-libs identifies them, but delete-old-libs can break installed packages still linked to those libraries. Rebuild or upgrade affected third-party software first, confirm with package and shared-library checks, and only then remove old libraries. The Handbook warns explicitly that programs using a deleted library will stop working.

Do not combine a kernel from main with a RELEASE userland. The Handbook notes that utilities such as ps and vmstat can fail when kernel and userland versions differ. The temporary new-kernel/old-world phase in the documented install sequence is controlled; indefinite branch mixing is not.

Tuning without a rebuild: loader.conf and sysctl

Before compiling, classify the desired change. Loadable modules can be tested with kldload and configured in /boot/loader.conf; boot-time tunables belong in loader configuration when their manual page says so; runtime controls belong in /etc/sysctl.conf; service settings belong in /etc/rc.conf.

# /etc/sysctl.conf
kern.maxfiles=65536

sysctl kern.maxfiles
sysctl kern.maxfiles=65536

The runtime command is temporary; persistence comes from sysctl.conf. Some values can only be set at boot, some are read-only, and some have safety constraints. The variable’s manual page and sysctl -d description are authoritative.

Build reproducibility and rollback discipline

Record /etc/src.conf, /etc/make.conf, KERNCONF, source commit, build command, compiler output, installed-kernel hash, and external module versions. Two kernels built from the same Git commit can differ because build options and local files differ.

Never make /boot/kernel.old the only rollback. A failed second experiment overwrites it with the first failed kernel. Preserve /boot/kernel.good, confirm it is readable from loader, and ensure the boot filesystem has space before installation. ZFS boot environments can preserve matching boot and userland state, but they must be created and tested before the incident.

A custom kernel should have an owner and a reason to exist. When the compile-time requirement disappears, returning to GENERIC reduces local maintenance and aligns the host with the most-tested path.

Related:

Sources: