Understanding systemd: Units, Targets, and the Modern Linux Init System

systemd is PID 1 and service manager on many Linux distributions, not a kernel requirement or universal init. It builds transactions from unit dependencies and ordering, supervises cgroups/processes, and integrates activation, logging, resource control, credentials, and sandboxing. The declarative model is powerful only when unit load precedence and lifecycle semantics are understood precisely.

Never test isolation, default-target, or isolate changes on a remote production host without console access and rollback.

Verify the manager and version

ps -p 1 -o pid,comm,args
systemd --version
systemctl --version

Containers can run systemd as their namespace PID 1 while the host differs. User managers (systemctl --user) have separate units, environment, cgroups, and lifecycle. Record whether a command targets system or user scope.

Units are loaded from ordered locations

Unit files may come from vendor directories such as /usr/lib/systemd/system, administrator /etc/systemd/system, and runtime /run/systemd/system, with precedence and aliases/generators. Do not edit vendor units in place; package upgrades replace them.

Inspect the effective unit and fragments:

systemctl cat myapp.service
systemctl show myapp.service -p FragmentPath -p DropInPaths
systemd-delta

Use systemctl edit myapp.service for a drop-in and systemctl revert only after reviewing what local overrides it removes.

Unit types encode activation semantics

Services, sockets, timers, paths, mounts, automounts, devices, targets, swaps, slices, scopes, and other unit types participate in the graph. A .device unit reflects kernel/udev state; it is not started like a daemon. A .scope groups externally created processes; a .slice organizes resource policy.

Generated mount/swap units can originate from /etc/fstab; cryptsetup/generator output can be transient under /run. Trace origin instead of assuming every unit has a static file.

Dependencies and ordering are orthogonal

Wants= and Requires= pull units into a transaction. After=/Before= order jobs if both are present. Requires= alone does not imply start order. If the required unit’s start fails and ordering is configured, the requiring unit is not started; later stop/failure propagation has additional nuances.

BindsTo=, PartOf=, Requisite=, Conflicts=, and Upholds= express different relationships. Choose from the current systemd.unit semantics rather than stacking directives until a test passes.

Targets group state but are not rigid stages

Targets can pull units together and provide synchronization points. multi-user.target, graphical.target, rescue.target, and emergency.target resemble familiar modes, but the graph can run many jobs in parallel.

Read current default and dependencies safely:

systemctl get-default
systemctl list-dependencies --all multi-user.target

systemctl isolate stops units not needed by the target and can terminate networking/session access. set-default changes future boot behavior. Both are state-changing administrative actions, not diagnostic commands.

`[Install]` does not run at service start

WantedBy=, RequiredBy=, Alias=, and Also= describe links created/removed by enablement. They are not runtime dependencies read into every start transaction. systemctl enable can create wants/requires links, aliases, and enable related units depending on [Install]; it does not universally create one .wants symlink.

Use systemctl enable --now only when both persistence and immediate start are intended. Inspect with systemctl is-enabled and list-unit-files, distinguishing static, indirect, masked, generated, enabled-runtime, and alias states.

Service readiness depends on `Type=`

With Type=simple, the service is considered started after the process is forked; Type=exec waits until execve() succeeds; Type=notify requires application readiness notification; Type=forking supports traditional daemons but needs correct PID/tracking behavior; oneshot has its own active-state semantics.

Choose what the daemon implements. Adding arbitrary sleeps or declaring After= does not prove an application is ready. Use sd_notify/documented protocol, health checks, or client retry as appropriate.

`ExecStart=` is not an implicit shell

systemd parses command lines with its own quoting/substitution rules and executes binaries directly. Pipes, redirects, &&, shell variables, and globbing do not work unless an explicit trusted shell is invoked.

Prefer a dedicated executable/script with fixed arguments. Do not place secrets in command lines or Environment= because they can appear in unit introspection, /proc, logs, or crash reports.

Credentials and identity need explicit design

Run as a dedicated User=/Group= with controlled SupplementaryGroups= and file ownership. DynamicUser= can allocate transient identity but has implications for persistent directories. StateDirectory=, CacheDirectory=, LogsDirectory=, and RuntimeDirectory= let the manager create scoped paths.

Use systemd credentials (LoadCredential=, encrypted credentials where supported) or a dedicated secret manager instead of embedding secret values in unit files. Verify version and threat-model guarantees.

Restart policy requires rate limits

Restart=on-failure can improve recovery but also create crash loops, log storms, dependency load, and repeated destructive migrations. RestartSec=, StartLimitIntervalSec=, and StartLimitBurst= bound behavior. Exit-status classification and watchdog/timeouts matter.

Test clean stop, crash, timeout, OOM, dependency loss, and invalid configuration. systemctl reset-failed clears state/counters; it does not fix the cause.

Socket/path/timer activation changes lifecycle

Socket activation requires the service to accept inherited file descriptors or the configured accept model. Binding the socket in both service and socket unit causes conflicts. Path activation is event/coalescing-oriented, not a guaranteed audit log of every filesystem change.

Timers support monotonic and calendar expressions. Persistent=true can catch a missed calendar activation when the timer was inactive, but it does not replay every missed interval. Pair each timer with an idempotent service, randomized delay where fleet stampedes matter, and bounded runtime.

Network targets are commonly misused

network.target does not mean configured connectivity. network-online.target is meaningful only when an enabled provider-specific wait-online service defines readiness; it can delay boot and still cannot guarantee DNS or a remote application remains available.

Long-running network clients should handle loss/retry. Use Wants=/After=network-online.target only for software that truly cannot start before initial configured connectivity, following distribution networking documentation.

Resource control maps to cgroup v2

Properties such as MemoryHigh=, MemoryMax=, CPUWeight=, CPUQuota=, TasksMax=, and IOWeight= configure the unit cgroup when supported/delegated. Limits inherit through slices and can interact with systemd-oomd.

Inspect effective runtime values with systemctl show and cgroup files. A unit-file value is intent; enforcement depends on kernel, hierarchy, controller delegation, and ancestors.

Sandboxing directives compose

NoNewPrivileges=, capability bounds, syscall/address-family filters, private temporary files/devices/network, namespace protections, and filesystem read/write restrictions can reduce exposure. Some options imply mounts/namespaces or fail on unsupported kernels/containers.

Use systemd-analyze security myapp.service as a heuristic checklist, not a certification score. Apply one group of restrictions, run positive and negative tests, and inspect the effective process namespaces/capabilities/LSM/seccomp state.

Journald evidence has retention boundaries

journalctl -u myapp.service -b
journalctl -u myapp.service --since '1 hour ago'
systemctl status myapp.service

The journal may be volatile, rotated, rate-limited, forwarded, or inaccessible by policy. status shows only recent lines. Preserve exact boot ID/timestamps and query the unit invocation ID where useful. Sanitize credentials and tenant data.

Debug the graph rather than `blame`

systemd-analyze verify /etc/systemd/system/myapp.service
systemd-analyze critical-chain myapp.service
systemctl show myapp.service -p Requires -p Wants -p After -p Before

systemd-analyze blame shows activation durations, not necessarily critical-path delay; parallel units can appear slow without delaying the target. Verify syntax, transaction edges, readiness type, actual process exit, cgroup state, and journal.

Use a controlled change workflow

Save systemctl cat/show output, create a drop-in, run daemon-reload, systemd-analyze verify, start/restart in a maintenance window, and test. Roll back the drop-in and reload if acceptance fails. daemon-reload does not restart services; reload and restart have different application impact.

The unit is production-ready when dependency/order/readiness semantics match reality, credentials and privileges are minimal, restart/limits are bounded, boot and failure tests pass, and removing the local drop-in restores the documented vendor behavior.

Related:

Sources: