Skip to content
Shell & TerminalDeep Dive Published Updated 6 min readViews unavailable

journalctl Cursor Files: Incremental Journal Ingestion Without Time-Only Checkpoints

Use journalctl --cursor-file for restartable journal reads, then design durable downstream checkpoints that respect retention, filtering, and replay limits.

Polling a journal with –since ‘five minutes ago’ is simple, but a wall-clock window is a weak checkpoint. Clock corrections, delayed runs, repeated timestamps, and arbitrary overlap choices can all produce duplicate or missing records. systemd’s journalctl –cursor-file=FILE provides an entry-position checkpoint for sequential reads: it starts after the cursor saved by a previous invocation and writes the cursor of the final returned entry when it exits. The option was added in systemd 242; check the installed journalctl –version before putting it into automation (systemd v255 journalctl manual).

The cursor solves only the journal-reader side of a pipeline. It does not make a downstream database write and a cursor-file replacement one transaction, guarantee exactly-once delivery, preserve entries after journald vacuums them, or make logs safe to disclose. A reliable design treats those as separate responsibilities.

A cursor is an opaque journal position, not a timestamp

The systemd manual defines –cursor-file for sequential invocations. If the file contains a cursor, the next call starts after that location; otherwise the invocation follows its other query options. At the end, the last entry’s cursor is written back to the file. A cursor is not a date string to parse or sort: the same manual says the printed cursor format is private and subject to change. Preserve it as opaque state and let journalctl interpret it.

Start by choosing the data set intentionally. Without a saved cursor, the first invocation shows entries according to the other given options; an unconstrained system-journal query can therefore read all accessible retained entries. Decide whether enrollment means an initial backfill, a bounded historical window, or “begin from a known point now.” Record that policy, rather than accidentally making the first run an unbounded export.

Keep the query scope stable

Journal matches and options filter which entries the invocation returns. That makes the cursor inseparable from the query policy that produced it. If a collector moves from one unit to several, or changes from the system journal to a user’s journal, do not silently reuse the old checkpoint and assume the expanded query will recover every older matching entry. Keep a distinct cursor per stable source/filter scope or intentionally perform a backfill into an idempotent sink.

For machine processing, –output=json emits newline-separated JSON objects. The reserved cursor field __CURSOR is included in structured output. JSON values are not all guaranteed to be ordinary strings: repeated journal fields can be arrays, non-UTF-8 values can be byte arrays, and fields over 4096 bytes are represented as null unless –all is used. Use a proper JSON parser and decide how to handle oversized fields; do not parse human-formatted output with grep and assume it is a stable record format (Journal JSON format, systemd v255 journalctl output options).

Stage the next cursor until the sink has accepted the batch

If the reader updates its durable cursor before the sink has durably committed the records, a crash can make the next run skip data the sink never stored. The opposite order can replay a batch after a crash between sink commit and cursor replacement. Use a private, serialized state directory, a temporary cursor file on the same filesystem, and a sink that deduplicates by __CURSOR. Provision the example state directory for a dedicated reader account with permission to read the selected journal, then run the service as that account rather than making the script world-readable or dumping records to a general temporary directory. The example uses flock from util-linux and a placeholder ingest-jsonl command; replace the placeholder with a sink that returns success only after its batch is durably committed and that can safely accept a replay.

#!/usr/bin/env bash
set -euo pipefail
umask 077

state=/var/lib/example-journal-reader
if [ ! -d "$state" ]; then
  printf 'State directory must be provisioned first: %s\n' "$state" >&2
  exit 1
fi
exec 9>"$state/reader.lock"
flock -n 9 || exit 75

cursor="$state/system-service.cursor"
next=$(mktemp "$state/.cursor.XXXXXX")
batch=$(mktemp "$state/.batch.XXXXXX")
trap 'rm -f -- "$next" "$batch"' EXIT

if [ -s "$cursor" ]; then
  cp -- "$cursor" "$next"
fi

journalctl --system --unit=example.service \
  --cursor-file="$next" --output=json --no-pager >"$batch"

if [ -s "$batch" ]; then
  # Placeholder: commit the full batch and deduplicate records by __CURSOR.
  ingest-jsonl "$batch" --dedupe-field=__CURSOR
fi

# Promote only after the sink reports success. Both files are in the same
# state directory so replacement does not cross filesystems.
if [ -s "$next" ]; then
  mv -f -- "$next" "$cursor"
fi

The temporary cursor matters. If journalctl fails or the sink returns a failure, the durable cursor remains unchanged and the next run can retry. If the sink committed but the process stopped before the rename, the records can be delivered again; that is why the sink must deduplicate. A successful file rename is an atomic namespace replacement on the same filesystem, but it is not by itself a power-loss durability guarantee. If the checkpoint must survive abrupt storage loss, use the state store’s documented synchronization and transaction behavior.

The first run needs an explicit enrollment decision. With no prior cursor, the example reads the matching entries currently available to the caller, which might be a large backfill. You can bound initialization with a deliberate one-time query policy, or create a reviewed starting cursor before enabling the recurring job. Do not add a moving –since window to every run and then describe it as a reliable cursor stream: changing time windows are a different selection rule, and filters can omit records by design.

Retention and access are part of the ingestion contract

Cursor state cannot reconstruct records that journald no longer retains. systemd’s vacuum controls remove old archived journal files according to time, size, or file-count limits. If the collector is offline longer than the retention window, a cursor may point to history that is no longer available. Set an alertable maximum collector lag, size retention for the expected outage and event volume, and define what operators do when that bound is exceeded. Do not claim that the cursor file alone detects or repairs every retention gap (systemd v255 journalctl vacuum options).

The journal can contain command arguments, user names, paths, network details, and application data. Protect the cursor and temporary batch directory with restrictive permissions, limit which journal is read, and avoid copying whole records into shell traces or world-readable temporary locations. A cursor identifies a position; it is not an access token and does not grant permission to read the journal.

Finally, distinguish continuous collection from a one-time export. journalctl –cursor-file is designed for sequential invocations; it is not a durable queue or remote collector. Monitor the last successful collection time, batch size, sink commit result, cursor replacement result, journal retention headroom, and duplicate count. Alert on a growing lag rather than assuming a scheduled timer ran. On recovery, preserve the old cursor, capture the error, and decide whether to retry, backfill, or declare a bounded data gap.

Related:

Sources:

Comments