Diagnosing a Linux Service That Fails Silently from a Permission Denial

A service fails to start, its own log output is unhelpful (“permission denied” with no further detail, or nothing at all), and a manual check of file ownership and Unix permission bits shows everything looks correct. This is the classic signature of a mandatory access control denial — SELinux or AppArmor blocking something that ordinary Unix permissions would otherwise allow.

Step 1: rule out the obvious first

Before suspecting SELinux/AppArmor, confirm the mundane things actually check out:

ls -la /path/to/resource
id serviceuser

If ownership, standard permission bits, and the service’s running user all look correct and it still fails, that’s your cue to check the MAC layer next rather than continuing to stare at ls -la output.

Step 2: check for recent SELinux denials

ausearch -m avc -ts recent

An AVC (Access Vector Cache) denial in this output, timestamped around when the service failed, is close to a smoking gun. audit2why translates the cryptic denial into a human-readable explanation:

ausearch -m avc -ts recent | audit2why

Step 3: check whether SELinux is even the layer in question

getenforce

If this returns Enforcing, SELinux is actively blocking based on policy; Permissive means it’s only logging what would be blocked (useful for testing, but not itself the cause of a real failure); Disabled rules SELinux out as a cause entirely, pointing you toward AppArmor or something else instead.

Step 4: the label mismatch you’re most likely to hit

The single most common SELinux-related failure isn’t a policy that’s genuinely too restrictive — it’s a file that was moved, copied, or restored from backup and now carries the wrong security context relative to what the policy expects for its location:

ls -Z /var/www/html/index.html
restorecon -Rv /var/www/html/

restorecon resets a path’s SELinux label back to the policy-defined default for its location — often the actual fix, rather than needing to write a new policy exception at all.

Step 5: for AppArmor systems, check the profile’s mode and log

aa-status
journalctl -k | grep -i apparmor

aa-status shows every loaded profile and whether it’s in enforce or complain mode. A denial logged via the kernel log (journalctl -k, filtered for AppArmor entries) names the exact path or capability that was denied, similar in spirit to an SELinux AVC entry.

Step 6: generating a fix rather than guessing

For SELinux, audit2allow can generate a custom policy module permitting exactly what was denied — useful for understanding scope, though it’s worth reviewing rather than blindly applying, since an overly broad generated rule defeats the purpose of having MAC enforcement at all:

ausearch -m avc -ts recent | audit2allow -M mypolicy
semodule -i mypolicy.pp

For AppArmor, put the profile in complain mode, exercise the failing operation, then let aa-logprof review the resulting log and interactively decide which accesses should become permanent allow rules:

aa-complain /usr/sbin/myservice
# ...trigger the failing operation...
aa-logprof

The one habit that prevents most of this confusion

Whenever a service fails with no useful application-level explanation and the standard Unix permissions genuinely look correct, checking ausearch -m avc -ts recent (or the AppArmor kernel log equivalent) should be step two, not step twenty — it turns a mystery failure into a specific, named denial almost immediately, rather than requiring a slow process of elimination across every other possible cause first.

Why the same symptom can have entirely different tooling on different distributions

SELinux (the default on RHEL, Fedora, and CentOS-derived systems) and AppArmor (the default on Ubuntu, Debian, and SUSE) are not two configurations of the same mechanism — they’re separate MAC implementations with genuinely different models. SELinux labels every file and process with a security context and evaluates access against a large, centrally-defined policy; AppArmor instead attaches a profile directly to a specific executable’s path, listing exactly which files, capabilities, and network operations that one program may use, with no system-wide labeling scheme at all. This is precisely why the diagnostic commands above split into two separate tracks rather than one — ausearch/audit2allow have no AppArmor equivalent that works the same way, and aa-status/aa-logprof have no direct SELinux equivalent, because the two systems represent policy in fundamentally incompatible internal structures despite solving the same underlying problem. Knowing which one a given distribution actually ships by default, before reaching for either toolset, saves the time otherwise spent running the wrong diagnostic command against a system that isn’t using that particular MAC implementation at all.

Related:

Sources: