Fixing 'Panic: Solaris Assert' Style ZFS Kernel Panics on FreeBSD
What a Solaris-assert-style ZFS panic actually means, why it almost always points to genuine pool corruption rather than a kernel bug, and how to safely get the system back up.
ZFS’s codebase carries a legacy of VERIFY/ASSERT-style internal consistency checks inherited from its Solaris origins, and when one of them fires on FreeBSD, it panics the kernel with a message that looks like a Solaris kernel dump rather than a typical FreeBSD panic — something like panic: solaris assert: <expression>, file: ..., line: .... The instinct is to read this as a mysterious kernel bug. In the overwhelming majority of real cases, it means ZFS detected an internal data structure that violated an invariant it depends on for correctness, and it panicked deliberately rather than continuing to operate on pool metadata it no longer trusts.
Why ZFS panics instead of just logging an error
This is a deliberate design choice, not a missing error-handling path. ZFS’s entire correctness model depends on metadata consistency — block pointers, checksums, transaction group state — being exactly what the code expects at every step. If an assertion catches a violation of that consistency, continuing execution risks writing further corrupted state on top of already-corrupted state, potentially turning a recoverable problem into an unrecoverable one. Panicking immediately, before any more writes happen, is ZFS choosing “stop now” over “possibly make this much worse.”
Reading the actual panic message
The panic string names the specific assertion that failed and the source file/line, which is the single most useful piece of information for figuring out what’s actually wrong — different assertions correspond to different failure categories. A dva.c assertion typically points at bad block-pointer/allocation state on a vdev. Something in zfs_znode.c points at directory-entry or inode-equivalent metadata. Assertions in arc.c relate to the adaptive replacement cache’s own internal bookkeeping, which is more likely to indicate a genuine software bug (since ARC state is memory-resident, not persisted-and-potentially-corrupted disk data) than on-disk corruption.
Before doing anything else, get the full panic string and, if the system produced a crash dump, its location:
grep -i panic /var/log/messages
ls -la /var/crash/A textdump or kernel core dump under /var/crash is worth preserving even if you’re not planning to debug it with kgdb yourself — it’s the artifact that would let someone else (or you, later, with more context) determine whether this was corruption versus a genuine ZFS bug worth reporting upstream.
The immediate priority: don’t make it worse
The single most important thing after a ZFS-related panic is to resist the urge to immediately force-import or repeatedly reboot into the same state hoping it resolves itself. If the panic happens again on the next boot at the same point (e.g., during pool import as part of normal startup), each reboot is another opportunity for ZFS to attempt writes against metadata it already flagged as inconsistent.
Boot into single-user mode or interrupt the boot loader to reach a prompt where you can control exactly when (and whether) the pool imports:
boot -sFrom single-user mode, do not simply run zpool import -a reflexively. Instead:
zpool import(with no pool name) lists importable pools and their reported state — ONLINE, DEGRADED, FAULTED, UNAVAIL — without actually importing anything. This tells you, before committing to an import, whether ZFS itself already believes the pool has a problem beyond the assertion that triggered the panic.
Importing read-only first
If the pool shows as importable at all, import it read-only before attempting a normal read-write import:
zpool import -o readonly=on -f <poolname>A read-only import lets you inspect the pool’s actual state — zpool status -v, checking for reported errors on specific vdevs or files — without risking any further writes that could interact badly with whatever inconsistency triggered the original panic. If the pool imports cleanly read-only and zpool status doesn’t show alarming vdev-level errors, that’s a reasonably strong signal the underlying pool structure is intact and the assertion may have caught something more localized or transient.
Scrubbing before trusting the pool again
Once you have a read-write import that doesn’t immediately re-panic, run a full scrub before resuming normal use, not after:
zpool scrub <poolname>
zpool status -v <poolname>A scrub walks every block and validates it against its checksum, which is exactly the class of verification that would surface latent corruption the original assertion failure hinted at but didn’t fully characterize. If the scrub completes with zero errors, that’s meaningful evidence the pool’s actual data is intact even though something briefly violated an internal invariant. If it reports checksum errors on specific files or, worse, on metadata, that confirms real corruption, and recovery shifts to restoring the affected files from backup rather than trusting the pool to have self-healed.
When this points at hardware, not ZFS
A single isolated assertion failure that doesn’t recur, on a pool that scrubs clean afterward, is often the tail end of a transient hardware event — an unclean shutdown mid-write, a brief storage-controller hiccup, a RAM error flipping a bit in an in-memory structure right before it was flushed. A recurring assertion failure, especially one that happens again shortly after a clean import and scrub, points somewhere more persistent — failing storage hardware, a memory problem (worth running memtest86+ or equivalent before assuming it’s software), or, less commonly, a genuine ZFS bug worth reporting to the FreeBSD or OpenZFS project with the exact panic string and, ideally, the crash dump. Treating a recurring panic as “just reboot it” rather than as a symptom to actually diagnose is how a small, contained problem turns into real data loss.