Diagnosing a Linux Service That Fails Silently from a Permission Denial
A service won't start, logs are unhelpful, and the files look correctly owned. SELinux and AppArmor enforce a second, invisible layer of permissions — here's how to check it.
A service fails to start, its own log output is unhelpful (“permission denied” with no further detail, or nothing at all), and a manual check of file ownership and Unix permission bits shows everything looks correct. This is the classic signature of a mandatory access control denial — SELinux or AppArmor blocking something that ordinary Unix permissions would otherwise allow.
Step 1: rule out the obvious first
Before suspecting SELinux/AppArmor, confirm the mundane things actually check out:
ls -la /path/to/resource
id serviceuserIf ownership, standard permission bits, and the service’s running user all look correct and it still fails, that’s your cue to check the MAC layer next rather than continuing to stare at ls -la output.
Step 2: check for recent SELinux denials
ausearch -m avc -ts recentAn AVC (Access Vector Cache) denial in this output, timestamped around when the service failed, is close to a smoking gun. audit2why translates the cryptic denial into a human-readable explanation:
ausearch -m avc -ts recent | audit2whyStep 3: check whether SELinux is even the layer in question
getenforceIf this returns Enforcing, SELinux is actively blocking based on policy; Permissive means it’s only logging what would be blocked (useful for testing, but not itself the cause of a real failure); Disabled rules SELinux out as a cause entirely, pointing you toward AppArmor or something else instead.
Step 4: the label mismatch you’re most likely to hit
The single most common SELinux-related failure isn’t a policy that’s genuinely too restrictive — it’s a file that was moved, copied, or restored from backup and now carries the wrong security context relative to what the policy expects for its location:
ls -Z /var/www/html/index.html
restorecon -Rv /var/www/html/restorecon resets a path’s SELinux label back to the policy-defined default for its location — often the actual fix, rather than needing to write a new policy exception at all.
Step 5: for AppArmor systems, check the profile’s mode and log
aa-status
journalctl -k | grep -i apparmoraa-status shows every loaded profile and whether it’s in enforce or complain mode. A denial logged via the kernel log (journalctl -k, filtered for AppArmor entries) names the exact path or capability that was denied, similar in spirit to an SELinux AVC entry.
Step 6: generating a fix rather than guessing
For SELinux, audit2allow can generate a custom policy module permitting exactly what was denied — useful for understanding scope, though it’s worth reviewing rather than blindly applying, since an overly broad generated rule defeats the purpose of having MAC enforcement at all:
ausearch -m avc -ts recent | audit2allow -M mypolicy
semodule -i mypolicy.ppFor AppArmor, put the profile in complain mode, exercise the failing operation, then let aa-logprof review the resulting log and interactively decide which accesses should become permanent allow rules:
aa-complain /usr/sbin/myservice
# ...trigger the failing operation...
aa-logprofThe one habit that prevents most of this confusion
Whenever a service fails with no useful application-level explanation and the standard Unix permissions genuinely look correct, checking ausearch -m avc -ts recent (or the AppArmor kernel log equivalent) should be step two, not step twenty — it turns a mystery failure into a specific, named denial almost immediately, rather than requiring a slow process of elimination across every other possible cause first.