Skip to content
FreeBSDHow-To Published Updated 7 min readViews unavailable

FreeBSD ATF and Kyua: Run the Base-System Test Suite Safely

Inspect FreeBSD's ATF and Kyua suite, run focused or full checks, preserve reports, and avoid enabling tests with destructive host side effects.

FreeBSD ships a machine-readable test suite, but a successful kyua test command is not a universal proof that a host is healthy. The useful result is a reproducible record of which test programs ran, which were skipped, which failed, what configuration they saw, and which machine and userland produced the result. That discipline matters after an upgrade, when validating a source-built system, and when comparing a hardware-specific regression across hosts.

The names describe different layers. ATF, the Automated Testing Framework, provides libraries and conventions for writing test programs. Kyua is a separate runtime and reporting tool; it loads a test suite definition from a Kyuafile, selects and executes tests, and stores results. FreeBSD’s suite is normally under /usr/tests, with /usr/tests/Kyuafile as its top-level definition. Kyua can also execute framework-less and TAP-compliant programs, so do not assume every test is an ATF test.

The examples below follow the FreeBSD 15.1-RELEASE manuals. Check the manuals and suite installed on the target release before automating these commands; test inventory, build configuration, and available suite metadata can differ between branches and locally built systems.

Confirm the suite and inspect before executing

First make sure the suite exists. If /usr/tests is absent, the FreeBSD tests(7) manual describes building and installing it with WITH_TESTS=yes in /etc/src.conf followed by a system rebuild. That is a source-build decision, not a reason to copy test binaries from a different kernel, userland, architecture, or branch.

On an installed system, list a small portion of the suite before running it:

test -r /usr/tests/Kyuafile
cd /usr/tests
kyua --config none list --verbose bin/cp usr.bin/stat

kyua list reads the suite definition and prints matching test cases; --verbose includes their metadata. The --config none global option deliberately avoids silently inheriting a user or system Kyua configuration during this initial inventory. It does not make test programs harmless or sandbox them. If a filter does not match, check the installed Kyuafile and test names rather than assuming the subsystem has no tests.

Capture the host identity alongside the output: release and architecture, kernel and userland update level, the origin of /usr/tests, Kyua version, account used, and any Kyua configuration. This metadata helps distinguish a reproducible OS regression from a suite built from another tree or an unsupported platform-specific case.

Start with a narrow test selection

Use the names shown by the installed suite and begin with a bounded selection. The FreeBSD manual documents bin/cp and usr.bin/stat as valid filters:

cd /usr/tests
kyua --config none test bin/cp usr.bin/stat

Kyua creates a new results file for each run. The default store is ~/.kyua/store/, and run logs are normally in ~/.kyua/logs/. Do not discard those artifacts after seeing the terminal summary: they are the evidence needed to compare failures, rerun a case, or hand a useful reproduction to a maintainer.

Inspect the recorded run even when the test command returns a non-zero status:

cd /usr/tests
kyua --config none report --verbose

The test command returns a failure status when an executed test fails or a supplied filter matches no test. A successful shell exit is therefore not the only signal to record, and a skipped test is not a pass. Keep the verbose report, Kyua log, and results database together under the same run identifier in your change record or CI artifact store. If you ran the suite as root, its default results live under root’s home; run the corresponding report with access to that same results store or pass the specific results file.

Understand privilege and isolation boundaries

Kyua runs tests in a controlled environment, including separate processes and temporary working directories; ATF cases also receive per-case execution and cleanup. This improves repeatability and helps clean ordinary test artifacts. It is not equivalent to running an untrusted test suite inside a VM or a security sandbox. FreeBSD’s own suite has configuration switches for tests that can change system-wide state or use destructive resources, which is an explicit boundary on what harness cleanup can guarantee.

Do not solve a permission-related result by running the entire suite with sudo. Some tests require root, but other tests are intended for an unprivileged user and Kyua metadata can make the runner drop privileges for them. Inspect the selected test metadata and run only the required root-dependent case with the necessary authority. Root also changes the home directory used for default logs and results, so keep the execution identity consistent with the account that produces the report.

Before a broad run, inspect both /etc/kyua/kyua.conf and any user-specific ~/.kyua/kyua.conf. In the FreeBSD suite, allow_devfs_side_effects permits tests that can destroy and recreate semipermanent device nodes, including disk device nodes. allow_sysctl_side_effects enables tests that change globally significant sysctls; the tests attempt to undo those changes during cleanup. The disks variable supplies devices for tests that need destructive disk access, and the manual explicitly says those tests are not required to preserve the data on them. fibs supplies routing tables that tests may modify, while allow_network_access enables tests that reach the network attached to the test host.

Treat these as host-impact controls, not routine coverage toggles. Do not point disks at a production or valuable disk. Do not enable global side-effect settings on a multi-tenant or production host merely to increase a pass count. For destructive device or global-state coverage, use an isolated disposable machine and explicitly provision expendable devices and network paths. With defaults, some tests that require extra configuration are skipped; report that honestly instead of describing a partial run as a complete validation.

Run the full suite with a controlled plan

After reviewing the installed test tree and configuration, the documented whole-suite invocation is:

kyua test -k /usr/tests/Kyuafile

For a repeatable post-upgrade check, use a maintenance or staging window, retain the full output and Kyua artifacts, and compare like with like: same architecture, release, test-tree provenance, account, configuration, and hardware where possible. A suite can expose a hardware-specific problem, but it only tests behavior represented in that release’s suite and enabled by the host’s configuration. It cannot establish that every driver, workload, third-party package, or production path is correct.

If one test fails, first confirm the selected test exists and that prerequisites or configuration are satisfied. Re-run the smallest failing filter, preserve its kyua report --verbose output and log, and compare the result against the same release and hardware baseline. A failure is evidence to investigate, not by itself proof that a kernel regression caused it. Likewise, an all-green result means the tests that actually ran passed under those conditions; it is not a security certification or a substitute for service-level checks, backup validation, or application tests.

A practical acceptance record

For each run, store a short manifest beside the results:

  • FreeBSD release, architecture, kernel and userland identity, and whether /usr/tests came from that same installation or a source build.
  • Kyua version, command line, execution account, selected filters, and the effective configuration file or --config none choice.
  • Counts of passed, failed, skipped, and otherwise incomplete cases from the report, without merging skipped cases into passes.
  • The ~/.kyua/store/ result database and matching ~/.kyua/logs/ file, with access restricted if logs contain sensitive host paths or data.
  • Hardware identifiers and any explicitly enabled network, disk, devfs, sysctl, or FIB-related test permissions.

This record makes a later comparison meaningful. Without it, two summaries with different test selection, privilege, hardware, or safety configuration can look comparable while describing different experiments.

Related:

Sources:

Comments