Skip to content
LinuxDeep Dive Published Updated 7 min readViews unavailable

Linux sched_ext: Preflight and Inspect a BPF Scheduler Safely

Inspect sched_ext kernel support, active BPF scheduler state, fallback signals, and cgroup semantics before testing a scheduler policy.

Linux sched_ext is an extensible scheduler class whose scheduling policy is supplied by BPF programs. It lets a userspace scheduler load a set of BPF callbacks instead of changing the kernel’s built-in scheduler, but that flexibility is easy to misread as a harmless desktop tuning switch. A loaded policy may schedule all ordinary tasks, depending on its flags. Before benchmarking one, establish whether the running kernel supports the class, whether a scheduler is already active, what task scope it selected, and how it handles controls such as CPU cgroups and nice weights.

The kernel’s safety behavior is useful, but it is not a substitute for staging. If the BPF scheduler exits, the kernel detects an error or a runnable task stalls, or an operator invokes SysRq-S, tasks return to the fair-class scheduler. That fallback protects system integrity; it does not guarantee that a latency-sensitive workload meets its service objective during a scheduler failure. Test in a disposable VM with the same kernel build and workload before considering a wider rollout.

Check the kernel build before installing a scheduler

The upstream kernel documentation identifies CONFIG_SCHED_CLASS_EXT as the option that enables sched_ext. Its documented configuration also includes BPF, BPF system calls, BPF JIT, BTF debug information, and the listed JIT-on settings. Distribution kernels may ship a config file, expose it through /proc/config.gz, or omit it entirely. Check what the running kernel reports rather than inferring support from a package name:

uname -r
grep -E 'CONFIG_(SCHED_CLASS_EXT|BPF|BPF_SYSCALL|BPF_JIT|DEBUG_INFO_BTF|BPF_JIT_ALWAYS_ON|BPF_JIT_DEFAULT_ON)' \
  "/boot/config-$(uname -r)" 2>/dev/null || true
zgrep -E 'CONFIG_(SCHED_CLASS_EXT|BPF|BPF_SYSCALL|BPF_JIT|DEBUG_INFO_BTF|BPF_JIT_ALWAYS_ON|BPF_JIT_DEFAULT_ON)' \
  /proc/config.gz 2>/dev/null || true
test -r /sys/kernel/btf/vmlinux && echo "kernel BTF is readable"

These checks are evidence, not a universal build test. An unreadable config file does not prove an option is disabled, and a config line does not prove that a particular scheduler can load against the running kernel. CONFIG_DEBUG_INFO_BTF is relevant because BPF schedulers commonly use kernel type information; userspace loaders and BPF verification can impose additional requirements. Match the scheduler project’s compatibility guidance to the exact kernel and BPF toolchain you plan to run.

Read active state without changing it

The sysfs interface is a safe starting point for determining whether a scheduler is currently registered. The kernel documentation shows these read-only files for the root scheduler:

for f in \
  /sys/kernel/sched_ext/state \
  /sys/kernel/sched_ext/root/ops \
  /sys/kernel/sched_ext/enable_seq \
  /sys/kernel/sched_ext/root/events; do
    if test -r "$f"; then
        printf '\n--- %s ---\n' "$f"
        cat "$f"
    else
        printf '\nnot readable: %s\n' "$f"
    fi
done

state reports the current sched_ext state; root/ops names the root scheduler when present; and enable_seq is a monotonically increasing count of scheduler activations since boot. A value of zero means no BPF scheduler has been loaded since this boot, not that the kernel lacks sched_ext support. The events file provides diagnostics such as CPU-selection fallbacks, dispatch behavior, and time spent in bypass mode. Interpret counters by their names and the documentation for the running kernel; a nonzero event is not automatically a failure, while a repeated fallback or bypass trend may be worth investigating.

For an individual process, inspect /proc/<pid>/sched for the ext.enabled field. That answers whether the selected task is on sched_ext; it does not identify every property of the root policy or prove that the scheduler is improving the workload. Pair per-task inspection with /sys/kernel/sched_ext/root/ops and the scheduler’s own status and logs. If these files do not exist, capture the kernel version and config evidence before trying to load anything.

Understand how much policy the BPF program owns

Scope is controlled by the scheduler implementation. When a BPF scheduler is loaded without SCX_OPS_SWITCH_PARTIAL, the kernel documentation says SCHED_NORMAL, SCHED_BATCH, SCHED_IDLE, and SCHED_EXT tasks are scheduled by sched_ext. With partial switching enabled, only tasks explicitly assigned SCHED_EXT use that class; ordinary, batch, and idle tasks stay on the fair-class scheduler. Read the implementation’s flags and deployment instructions before assuming your benchmark process is the only task affected.

The scheduling callbacks also have non-obvious semantics. select_cpu() is a placement hint and can wake the selected CPU, but the final decision happens later and the kernel can ignore an invalid CPU choice. Schedulers place tasks into dispatch queues (DSQs), commonly per-CPU local queues or a global queue, and can define custom DSQs. A policy that changes queueing and placement may affect fairness and latency beyond the one process being measured. A “simple” example scheduler is a way to understand these mechanisms, not a ready-made production policy.

Do not assume normal CPU controls automatically retain their familiar effect. The fair scheduler enforces cgroup CPU settings such as cpu.max, cpu.weight, and cpu.idle. For sched_ext, the kernel communicates those settings to the BPF scheduler through callbacks, and each BPF scheduler is responsible for implementing the intended semantics. The same applies to task nice values, which are converted to weights and reported to the scheduler. Check the loaded policy’s documentation and source before relying on cgroup quotas, weights, or nice as guardrails.

Use a matching kernel tree in a disposable VM

The upstream source tree includes example schedulers under tools/sched_ext. Its documentation demonstrates building those examples with make -C tools/sched_ext and running tools/sched_ext/build/bin/scx_simple from a root prompt. Treat that as a controlled lab sequence, not as a command to try on a work laptop or shared server: use the privileges required by the kernel and loader in a disposable VM, because loading the example changes scheduling behavior while it runs. Match the VM’s kernel, BTF data, and toolchain to the target, and take a recoverable snapshot first.

In that VM, record the state, ops, enable_seq, and events values before the test. Build the example from the source tree corresponding to the running kernel, then start it only after confirming the guest is disposable and can be recovered from the VM console. Observe the scheduler process output and the sysfs state while it is active. The kernel documentation shows Ctrl-C unregistering the example scheduler; after it exits, confirm the state returns to the expected inactive condition and that enable_seq has advanced. Compare this with the kernel’s own automatic fallback behavior, but do not manufacture a stalled workload just to test recovery.

SysRq keys have different roles: SysRq-D triggers a debug dump and does not stop the BPF scheduler; SysRq-S aborts sched_ext and returns tasks to the fair class. Do not confuse a diagnostic dump with a rollback. If a scheduler fails, inspect its userspace exit reason, kernel messages, sysfs event counters, and the source-level compatibility requirements before repeating the test.

Measure policy, not a convenient graph

Benchmark a scheduler against the same kernel configuration and workload using repeated runs, a known CPU affinity, and a documented cgroup setup. Track throughput and latency distributions alongside CPU utilization, migrations, run-queue behavior, and sched_ext events. Keep the tested task set explicit: a policy with full switching can affect services and background work that are not part of the benchmark. If the scheduler implements its own interpretation of cgroup weight or nice, report that policy rather than describing it as ordinary fair-scheduler behavior.

Treat kernel-version compatibility as part of the result. The BPF callbacks, helpers, and scheduler flags are closely tied to kernel interfaces; rebuild and revalidate the scheduler when changing kernel versions rather than assuming an old binary remains suitable. Keep the kernel documentation, scheduler source revision, BPF object, and runtime logs together with the benchmark. That lets another operator distinguish a policy result from a toolchain or kernel mismatch.

The safe workflow is read-only preflight first, kernel-matched disposable testing second, and performance claims only after confirming task scope, scheduler state, and cgroup semantics. sched_ext makes experimentation possible without replacing the kernel, but the active BPF policy still participates in a system-wide scheduling contract.

Related:

Sources:

Comments