systemd Readiness and Watchdogs with sd_notify

Service startup is not a single instant. A process can exist while it is still loading configuration, recovering a database, binding listeners, or waiting for a dependency. If systemd treats process creation as readiness, dependent units may start too early and monitoring may report a false healthy state. The sd_notify protocol lets a service send lifecycle information to the manager, including a deliberate READY=1 signal and periodic watchdog keepalives. These messages are useful only when they represent real application state.

The design principle is simple: send readiness after the service can satisfy its documented contract, and send watchdog keepalives only while its event loop or health-critical work remains responsive. A keepalive that runs on an independent thread while the main request path is dead can mask a failure. Conversely, a watchdog interval shorter than normal pauses can cause unnecessary restarts. Treat readiness and watchdogs as correctness mechanisms with explicit state transitions, not as decorative unit settings.

Select the correct service type

For a daemon that speaks the notification protocol, configure Type=notify. systemd then waits for a READY=1 message before considering startup complete and proceeding with ordering relationships that depend on the service. This matters for After= consumers: ordering controls sequencing, while the service type determines what systemd considers startup completion. A Type=simple unit generally marks startup much earlier, when the process has been started.

Use Type=notify only when the program or a trusted wrapper sends notifications according to systemd’s protocol. If it never sends READY=1, startup can time out and the unit fails. Do not set Type=notify simply because you want a health check. Health after startup and readiness at startup are related but distinct concerns; a monitor may still need to check the actual API or dependency path.

An example unit is:

# /etc/systemd/system/indexer.service
[Unit]
Description=Search indexer
After=network.target

[Service]
Type=notify
NotifyAccess=main
ExecStart=/usr/local/sbin/indexer --foreground
TimeoutStartSec=90s
WatchdogSec=30s
Restart=on-failure
RestartSec=5s

[Install]
WantedBy=multi-user.target

The executable and timings are examples, not universal recommendations. NotifyAccess=main constrains which process may send notifications; align it with the program’s implementation and systemd version documentation. TimeoutStartSec= bounds startup, but setting an enormous timeout can hide a permanently stuck initialization. Measure startup under expected recovery conditions and set a budget that allows legitimate work without letting dependencies wait indefinitely.

Send readiness at the right boundary

With libsystemd, a daemon can call sd_notify(0, "READY=1") after initialization is complete. The readiness boundary should correspond to the promise made to clients and dependent units. For an HTTP server, it may mean configuration is parsed, required data structures are loaded, the listener is bound, and requests can be served. If a service can start in degraded mode, document whether readiness means “serving a reduced contract” or “all dependencies healthy.” Avoid waiting for optional dependencies unless the application truly cannot function without them.

For a program in another language, use a maintained binding or send to the notification socket according to the documented protocol. systemd exposes the socket through NOTIFY_SOCKET for a notification-enabled unit. Hand-rolling datagram messages requires correct address handling and credential semantics; using the official library reduces protocol mistakes. Never hard-code a socket path because it is supplied by the manager and may use an abstract namespace socket.

Readiness should normally be a one-way transition for a process generation. READY=1 communicates startup completion; it is not a continuous health stream. If the application becomes unhealthy later, use an appropriate health system, fail the process when recovery is impossible, or expose a health endpoint that monitoring can evaluate. Do not repeatedly send READY=1 to paper over crashes or confuse consumers about recovery state.

Configure and implement a watchdog deliberately

When WatchdogSec= is configured, systemd expects watchdog notifications while the service remains active. The manager provides the watchdog interval to the process through the documented environment variable WATCHDOG_USEC; supported libraries can query it. A healthy service should send WATCHDOG=1 often enough to remain inside the interval with margin. Many daemons schedule a heartbeat at a fraction of the interval, but the exact cadence must account for scheduling delays, stop-the-world pauses, and the failure model.

The watchdog should be tied to the part of the program whose failure it is intended to detect. If the event loop is the service’s critical path, the event loop itself should make progress or authorize the heartbeat. A separate timer thread that continues sending notifications while all workers are dead defeats the watchdog. For a multithreaded service, define which component owns the progress signal and how it learns that other essential workers have stopped making progress.

Watchdog expiry causes systemd to mark the service failed and apply the configured failure policy. With Restart=on-failure, a hung process may be terminated and restarted. That can restore service but can also create a crash loop, duplicate external work, or interrupt in-flight operations. Make startup recovery safe, use rate limits thoughtfully, and ensure externally visible operations are idempotent or have explicit deduplication. A process watchdog cannot guarantee that a restarted operation has exactly-once effects.

Observe the state transition

After installing a unit, validate it and inspect systemd’s view rather than trusting only application logs:

sudo systemd-analyze verify /etc/systemd/system/indexer.service
sudo systemctl daemon-reload
sudo systemctl start indexer.service
systemctl show indexer.service -p ActiveState -p SubState -p Result -p ExecMainStatus
systemctl status indexer.service
journalctl -u indexer.service --since '-10 minutes'

For a Type=notify service, compare the reported activation time with the point at which the application emitted readiness. If startup times out, determine whether the notification was not sent, was rejected due to access settings, came from a process systemd did not recognize, or was delayed by genuine initialization. Check the manager journal and the service journal for protocol and timeout details.

Test watchdog behavior in a non-production environment with a controlled fault. One safe test is to add a deliberate test-only switch that stops the progress loop, then confirm the manager detects the missing keepalive and performs the expected restart. Do not induce a CPU spin or kill production dependencies to simulate this. Verify recovery, logs, restart counters, and external side effects after the test. A unit that has WatchdogSec= configured but no watchdog-aware application is not a successful test.

Common implementation errors

Sending readiness immediately after fork() is usually too early if a child still performs initialization. For a wrapper that launches another process, define which process owns the service lifecycle and ensure notification attribution is correct. A shell script that starts a daemon in the background and exits can make the service manager track the wrong process; use direct execution or a deliberately designed supervisor.

Another error is failing startup because an optional remote dependency is temporarily unavailable. Decide whether the service can queue work, serve cached data, or expose a degraded state. If it can, send readiness for that truthful contract and surface degraded health separately. If it cannot, fail startup with actionable diagnostics and a bounded retry policy rather than hanging until an arbitrary timeout.

Watchdog intervals can also conflict with normal maintenance pauses, garbage collection, memory pressure, or slow storage. Measure worst-case pauses under realistic load, use a margin, and alert on repeated watchdog failures. Increasing the timeout may reduce false positives but also lengthens detection time. Choose based on recovery objectives, not convenience.

Acceptance criteria should state observable behavior: dependents do not pass their startup ordering point until readiness is reported; a deliberately withheld readiness signal eventually fails startup; the application sends watchdog updates only while required progress continues; a controlled missed heartbeat triggers the documented failure action; and restart recovery does not corrupt or duplicate external work. Record the tested systemd version and application version because notification behavior and options should be checked against deployed manuals.

sd_notify makes service lifecycle explicit. Use readiness to mark a real startup contract, watchdogs to detect a defined lack of progress, and systemd state plus application-level signals to verify the result. The protocol cannot replace sound recovery design, but it provides a precise boundary between “process exists” and “service is ready.”

Related:

• Understanding systemd: Units, Targets, and the Modern Linux Init System
• Fixing ‘start-limit-hit’ Errors in systemd

Sources:

• systemd.service manual source
• sd_notify(3) manual source