Skip to content
LinuxDeep Dive Published Updated 8 min readViews unavailable

Linux Kernel Keyrings: Process Scope, Request Upcalls, and Expiry

Choose Linux keyring scope by process lifetime, trace request_key upcalls, and diagnose permissions, expiry, and per-user quota failures safely.

Linux keyrings are kernel-managed objects used by filesystems and other kernel services to retain keys, authentication tokens, and related data. Their most important operational property is often not cryptography but scope: a key can be present and still be invisible to the process that needs it because that process crossed a fork, exec, login, or service boundary. This guide focuses on the generic key retention service, the userspace request_key() path, and safe diagnosis. It does not describe a hardware security module, an encrypted disk, or a guarantee that privileged administrators cannot inspect or affect key state. Check the kernel configuration, security policy, and key type used by the specific service before relying on any example.

Treat a keyring as a scoped namespace

A kernel key has a type, description, permissions, expiry, state, and possibly a payload. A keyring is itself a special key whose payload is a list of links to other keys. A user key can hold an arbitrary userspace-readable blob when its permissions allow it. A logon key is intended for secrets that kernel code can use but userspace cannot read back through the key payload interface. The type determines behavior; the string logon is not a universal substitute for reviewing who can create, search, link, or use that key.

Each process can be subscribed to three standard rings, searched by request_key() in this order: thread, process, then session. These scopes do not have interchangeable lifetime rules:

Ring Operational lifetime to account for
Thread (@t) A thread keyring is discarded from a child as clone, fork, vfork, or exec creates the child/task context. It is created when needed.
Process (@p) Fork-like creation normally gives the child an empty process keyring; threads in the same thread group share it. execve() replaces the process ring.
Session (@s) Inherited across clone, fork, vfork, and exec, including set-UID or set-GID execution. A process can explicitly join or create a different session ring.

Linux also provides per-UID rings, including a user ring and a default user-session ring. The default session ring is linked to the per-user ring, which is why a key placed under a per-user scope can be visible to a wider set of processes than a service-private session. Do not choose @u simply because it makes a test succeed. Decide which processes should share the credential, how long it should remain usable, and which principal is allowed to search it.

An operational symptom follows directly from the lifetime table: a helper launched through execve() may lose a key held only in the caller’s process ring while still inheriting the session ring. A daemon with a distinct login or service manager session may see neither. When a lookup fails, compare the requesting process’s actual subscriptions and credentials instead of copying the key into increasingly broad rings.

Understand lookup and the userspace upcall

request_key(type, description, callout_info, dest_keyring) first searches the caller’s subscribed rings for a matching key, subject to search permissions. If a match is found, the syscall can link it into the requested destination ring, which must be writable and for which the key must grant link permission. If no key matches and callout_info is non-NULL, the kernel can create an in-construction request and invoke /sbin/request-key. The helper’s configuration selects the userspace action that attempts to obtain or construct the key. A missing-key request with no callout information does not ask a helper to fetch one.

The control flow is roughly:

service asks for (type, description)
        -> search permitted thread/process/session rings
        -> if absent and callout is allowed, invoke request-key policy
        -> helper obtains/constructs the typed key
        -> kernel instantiates it and links it as requested
        -> caller uses the key through the type's allowed operations

This is a callback mechanism, not an implicit network credential fetcher. The key type or consumer must request a key in a way that permits an upcall, and the machine must have an applicable helper configuration and dependencies. A request can fail because no rule matches, the helper cannot reach its source, the key is rejected or expired, or the result cannot be linked into the destination. Inspect the service’s exact type and description before changing request-key policy: broad match rules can route more requests than intended.

For a small userspace client using libkeyutils, the request shape looks like this:

#include <keyutils.h>

key_serial_t id = request_key("user",
                              "demo:fixture-token",
                              "environment=staging",
                              KEY_SPEC_SESSION_KEYRING);
if (id == -1) {
    /* Preserve errno for diagnosis; never log a secret payload. */
}

The strings above are deliberately non-secret fixture labels. In production, treat the description and callout information as routing metadata, not as a place to place passwords or access tokens. The request helper’s precise invocation and accepted configuration are described by the local request-key(8) and request-key.conf(5) documentation; distributions can package and configure those components differently.

Diagnose the process that actually failed

Run keyutils commands as the same user and from the same service or login context as the failing process. A shell’s rings are not automatically the rings of a system service, a container, or another user’s session. In a disposable test account, keyctl show displays the caller’s subscribed rings and their visible contents. keyctl search @s user demo:fixture-token tests a specific type and description in the session tree. A carefully scoped test key can be added with keyctl add user demo:fixture-token fixture-only @s; never substitute a real credential on a shared host merely to prove the command works.

For a request that is supposed to trigger an upcall, reproduce it only after confirming the matching request-key rule and helper are safe for the test identity. Record the requesting executable, real and filesystem UID/GID, key type, description, selected ring, return code, and relevant helper status. Keep payloads, bearer tokens, and unredacted callout secrets out of logs. The U flag in /proc/keys can indicate a key under construction by a userspace callback; /proc/keys otherwise exposes only entries the reader may view and does not reveal key payloads. /proc/key-users reports per-user key and byte usage. These files are diagnostics, not a full inventory for every process or a secret-reading interface.

Use errno to distinguish likely failure classes. ENOKEY means no usable match was found; EACCES points to access or destination-ring permission; EDQUOT means a key or payload operation would exceed a quota; EKEYEXPIRED and EKEYREVOKED identify stale or explicitly revoked material; and EKEYREJECTED can indicate that key construction was rejected. Do not respond to a permission error by granting broad search or write access without first identifying which ring and operation failed.

Each UID is subject to quotas for key count and total description/payload space. A burst of unique descriptions can therefore fail even when individual payloads are small. Reuse stable descriptions when the key type’s update behavior is appropriate, expire or revoke test material, and monitor /proc/key-users before proposing a system-wide quota change. Process and thread rings have specific quota accounting rules, so do not infer total consumption from one ring’s displayed entry count.

Stage expiry and cleanup deliberately

Expiry and revocation are different. Expiry makes a key unusable after its configured lifetime and may allow a replacement request; revocation makes it unavailable for further operations and returns EKEYREVOKED for attempts to use it. Unlinking a key from a ring removes that reference, but does not mean every other link or reference has disappeared. Garbage collection can defer physical reclamation after expiry or revocation, so monitoring should distinguish “unusable” from “already freed.”

For a test fixture, create an isolated session, add only a fake payload, verify lookup in the intended child process, and test the expected exec or service restart boundary. Then revoke or remove the fixture using the narrowest operation supported by the key type. Avoid keyctl clear on a ring whose owner or contents you have not identified. On production hosts, coordinate cleanup with the kernel consumer: revoking a key can interrupt mounts, authentication, or another in-flight service operation.

Use this sequence before deploying a new key consumer:

  1. Confirm that the kernel has key retention enabled and that the required key type is registered by the relevant service.
  2. Identify the process credentials and ring scope used at the moment of lookup, including any fork, exec, login, namespace, or service-manager boundary.
  3. Verify the narrow key description, permissions, destination ring, expiry, and request-key rule with synthetic data.
  4. Test missing, denied, expired, revoked, helper-unavailable, and quota-exceeded cases, and ensure diagnostics never print the payload.
  5. Document who owns rotation, revocation, and cleanup, then revalidate after kernel, distribution, service, or keyutils upgrades.

The useful mental model is a kernel object with explicit lookup, permission, and lifetime rules. Once the process scope and request-key callback are visible, a key that “exists but cannot be found” becomes a boundary diagnosis rather than a reason to copy secrets into a global ring.

Related:

Sources:

Comments