Skip to content
Shell & TerminalDeep Dive Published Updated 7 min readViews unavailable

systemd ExecStart: Treat Unit Commands as Argument Vectors

Write reliable systemd service commands by separating unit-file parsing from shell syntax, environment expansion, executable lookup, and process supervision.

A systemd unit is not a shell script. In an ExecStart= setting, systemd parses a command line using its own quoting and substitution rules, selects the first item as the executable, and passes the remaining items as arguments. It does not normally interpret >, |, &&, command substitution, for loops, or shell variable assignments. This distinction explains a large class of units that work when pasted into a terminal but fail when started by the service manager.

The reliable default is to configure one program and its arguments directly. If a workflow needs branching, loops, complex environment handling, or multi-step cleanup, put those operations in a separately tested script and make the unit execute that script. An explicit shell can be appropriate for a tiny fixed expression, but it adds another parser and another process boundary that must be understood.

Read the command as a program plus arguments

In this unit, the executable is /usr/local/libexec/report-worker; the remaining words are its argument vector:

[Service]
Type=exec
ExecStart=/usr/local/libexec/report-worker --config /etc/report-worker/worker.conf --foreground

The service manager does not first turn this line into a string and hand it to /bin/sh -c. Quotes group values according to systemd’s unit syntax, not shell grammar. Shell operators therefore have no normal shell meaning. For example, appending >> /var/log/worker.log does not redirect output; those words would be arguments to the selected executable. Likewise, A=one /path/program is not the portable shell idiom for setting one command’s environment. Put environment settings in Environment= or EnvironmentFile=, with the latter’s separate file-format rules, or configure them through the service manager’s environment mechanisms.

Prefer a foreground program that remains in the foreground. A service whose main process daemonizes itself, forks and exits, or backgrounds work changes what systemd can supervise. Type= describes how startup completion is determined; it is not a request to invent a shell wrapper. Use the service type documented for the program’s actual behavior, and avoid Type=forking unless the program really performs the expected forking protocol and provides any required PID file.

Keep systemd quoting separate from shell quoting

Unit files have an escaping and quoting syntax that resembles a shell in places but is not interchangeable with it. A quoted argument can contain spaces, and systemd supports documented escapes, but unquoted shell expansions such as $HOME do not mean “ask the shell to expand this.” Unit parsing happens in the manager before the program is executed. The process receives values resulting from systemd’s own expansion rules and its configured environment, not the interactive user’s startup files.

This difference matters for filenames and arguments with whitespace. Use one quoted unit-file argument when the target program must receive one value containing spaces, and verify the actual argv using the program’s diagnostic mode or a controlled test executable. Do not add extra quote characters just because an example shell command uses them; shell quote delimiters are consumed by the shell, while unit-file quote delimiters are consumed by systemd’s parser.

Environment expansion is another frequent source of surprises. systemd documents its own forms for environment variables, including a distinction between word-splitting $NAME expansion and a single-argument ${NAME} form in command lines. Neither performs shell parameter operators such as ${NAME:-default}, arithmetic expansion, globbing, or command substitution. When correctness depends on an exact argument boundary, explicitly test empty, whitespace-containing, and unset values with the daemon’s documented expansion behavior rather than reasoning from Bash syntax.

Percent-prefixed specifiers are a separate expansion mechanism. A literal percent sign may need escaping as %%; a program argument that contains specifier-looking data should be checked against the relevant unit manual. The complete processing order includes unit syntax, specifier expansion, environment substitution, and eventual program argument handling. Avoid combining every expansion layer in one dense line. Move data transformations into the program or a script where they can be unit-tested.

Invoke a shell only as an explicit executable

If a unit truly needs shell operators, name the interpreter and provide a single script argument. For example, this fixed command uses /bin/sh to apply a controlled conditional and then replaces the shell with the worker:

[Service]
Type=exec
ExecStart=/bin/sh -c 'if test -r /etc/report-worker/enabled; then exec /usr/local/libexec/report-worker --foreground; else exit 0; fi'

The quotes delimit one argument to -c under systemd parsing; the shell then parses the text inside that argument. This still has limits: systemd may process its own variable or specifier syntax before /bin/sh receives the argument. If literal dollar signs or percent signs are involved, consult systemd.service(5) and systemd.syntax(7) and escape for the systemd layer first. A script file is usually easier to review than nested quoting.

When the shell launches one long-running program, use exec so the shell is replaced. That keeps the worker as the service’s main process and makes exit status and signal delivery easier to reason about. Do not use sh -c to concatenate untrusted data into a command. Pass values as arguments, validate them, and avoid turning configuration into source code. Environment variables are not automatically safe merely because systemd supplied them.

For multi-command setup, prefer a dedicated script with a declared interpreter and normal shell tests. A service can execute that script directly if it has a valid shebang and executable permissions, or invoke the interpreter with the script path explicitly. Keep the script’s logging, exit status, and cleanup behavior visible. Multiple ExecStartPre= commands are possible, but systemd’s service-type and failure semantics still apply; they are not a substitute for a complex workflow engine.

Let the service manager own process lifecycle

Do not append & to a long-running service command to make it “background.” The manager already launches the process asynchronously and tracks it according to the unit’s service type and control-group settings. Backgrounding can make the tracked process exit while the real worker continues, which interferes with restart policy, stop handling, resource accounting, and status reporting.

Logging should normally use the service manager’s journal integration or an explicitly configured output target. Redirecting in shell syntax is not available in a normal ExecStart= line. If the program needs a file path as a log destination, pass the documented application option and ensure directory permissions, rotation, and ownership are intentional. For stdout and stderr, use the unit’s StandardOutput= and StandardError= controls where appropriate.

Signals and stop behavior are part of the service contract. systemd sends the configured stop signal to the unit’s processes and applies a timeout before its final action. The exact process set and signal behavior depend on unit settings, including KillMode=. A wrapper that backgrounds children, ignores signals, or fails to wait can outlive its intended shutdown path. Prefer a service that remains in the foreground; if a wrapper must supervise multiple processes, use a real supervisor or implement explicit forwarding and reaping rather than relying on one shell line.

Use systemctl cat to see the effective unit and drop-ins, systemctl show for parsed properties, and systemd-analyze verify to check syntax before activation. Then inspect systemctl status and the journal for runtime failure. A syntax check does not prove that a binary exists, that its arguments are accepted, or that its environment is complete. Test in a staging unit with the same user, working directory, environment files, and sandboxing settings as production.

A practical validation sequence

Start by identifying the exact executable and its documented foreground mode. Construct its argv as a direct ExecStart= list and move environment values into the proper unit directives. If the command appears to need a pipe, redirection, glob, assignment, or semicolon, decide whether a script or a program-native option is clearer. Do not assume an interactive shell’s aliases, PATH, current directory, or startup files exist in the service context.

Next, validate quoting and expansion with representative data: an ordinary value, an empty value, a value with spaces, a literal percent sign, and an unset variable. A tiny diagnostic helper can write each argv element on a separate numbered line during a safe test, but remove or disable it before exposing sensitive values in logs. Test the same parsed unit through systemd, because manually running a shell approximation exercises a different parser.

Finally, verify lifecycle as well as startup. Confirm that the expected process is the main process, that a nonzero exit is reported, that restart policy behaves as intended, and that stop reaches the worker and completes within the unit’s timeout. Inspect the effective drop-ins rather than only the source unit file. This approach treats ExecStart= as a structured process contract: executable, arguments, environment, identity, and lifecycle are explicit instead of hidden inside a terminal command.

Related:

Sources:

Comments