Container Runtime Internals: containerd, CRI-O, and the OCI Spec

“Docker,” “containerd,” “CRI-O,” and “runc” get used almost interchangeably in casual conversation, but they occupy genuinely distinct layers in the actual container execution stack — and the reason they can be swapped in and out of Kubernetes without breaking anything is a deliberate standardization effort: the OCI (Open Container Initiative) specs.

The layers, from top to bottom

A useful mental model, from what you interact with down to what actually creates namespaces and cgroups:

kubectl / docker CLI          — user-facing tools
   │
CRI (Container Runtime Interface)   — Kubernetes' abstraction boundary
   │
containerd / CRI-O              — high-level runtime: image pulling, storage, lifecycle
   │
runc (or another OCI runtime)    — low-level runtime: actually creates the container
   │
Linux kernel (namespaces, cgroups)

Docker itself, under the hood, has used containerd as its own high-level runtime for years — Docker Engine is a layer of UX and API on top of largely the same underlying components Kubernetes uses directly.

The OCI Runtime Spec: what “run a container” actually means

The OCI Runtime Specification defines a precise, tool-agnostic contract: given a root filesystem and a JSON configuration file (config.json) describing namespaces, mounts, capabilities, and the process to execute, a compliant runtime must produce an equivalent running container. runc is the reference implementation, but crun, kata-runtime (running containers inside lightweight VMs for stronger isolation), and others all implement the same spec.

{
  "ociVersion": "1.0.2",
  "process": {
    "args": ["/bin/sh"],
    "cwd": "/"
  },
  "linux": {
    "namespaces": [
      {"type": "pid"},
      {"type": "network"},
      {"type": "mount"}
    ]
  }
}

runc run mycontainer

This standardization is exactly why Kata Containers (running each “container” inside its own lightweight VM for hardware-level isolation, addressing the shared-kernel risk namespaces alone don’t fully solve) can be swapped in as a drop-in alternative to runc — both satisfy the same OCI Runtime Spec contract.

The OCI Image Spec: what an image actually is

Separately, the OCI Image Specification defines the format of a container image itself — a manifest listing content-addressed layers (each a tar archive, typically gzip-compressed) plus a config blob describing environment variables, the entrypoint, and exposed ports:

docker manifest inspect nginx:1.25

{
  "schemaVersion": 2,
  "config": {"digest": "sha256:abc123...", "size": 7023},
  "layers": [
    {"digest": "sha256:def456...", "size": 31411704}
  ]
}

Because every image-building and image-running tool targets this same format, an image built by docker build, buildah, or podman build is fully interoperable with any OCI-compliant runtime, regardless of which tool produced or eventually runs it.

CRI: Kubernetes’ plug-in point for runtimes

The Container Runtime Interface (CRI) is a gRPC API Kubernetes’ kubelet speaks to whatever container runtime is configured on a node — this is the abstraction that let Kubernetes drop direct Docker-specific integration (dockershim) in favor of talking to any CRI-compliant runtime uniformly:

crictl ps
crictl images
crictl inspect <container-id>

containerd and CRI-O both implement CRI directly; crictl is the CRI-native equivalent of docker ps/docker inspect, useful for debugging directly on a Kubernetes node without going through kubectl at all.

Why containerd and CRI-O both exist

containerd began as an extraction of Docker’s own internal runtime component into a standalone, CNCF-governed project — broadly general-purpose, used by Docker itself and directly by Kubernetes. CRI-O was built specifically and only to implement CRI for Kubernetes, with a narrower scope and no independent CLI-driven use case of its own outside that role. Both wrap runc (or another OCI runtime) underneath and both are CNCF graduated projects — the choice between them in a given Kubernetes distribution is largely a matter of ecosystem fit rather than one being definitively superior.

systemctl status containerd
systemctl status crio

Why the layering, in practice

The value of this OCI-standardized stack shows up whenever you need to swap a layer without disrupting everything above it: switching from runc to crun (a faster, lower-memory-footprint OCI runtime) or to kata-runtime (for VM-level isolation) requires no changes to containerd, no changes to Kubernetes, and no changes to how images are built — because every layer only depends on the OCI-specified contract of the layer below it, not on that layer’s specific implementation. That’s the entire point of standardizing the spec rather than just standardizing on one tool.

A shim preserves container lifecycle

The high-level runtime does more than launch runc. containerd uses per-container runtime shims so the long-running container process is not tied directly to the daemon’s process lifetime. The runtime tracks metadata, snapshots, content, tasks, and events, while the OCI runtime performs create, start, kill, and delete operations against a bundle. CRI-O implements the Kubernetes-focused path with OCI runtimes and storage/network integrations.

That layering is why diagnosis must begin at the correct interface. kubectl reports desired and observed Kubernetes state; crictl asks the CRI implementation about pods, containers, images, and logs; containerd’s ctr is primarily a low-level debugging client and does not reproduce every CRI namespace or Kubernetes behavior. Operators should not use a low-level delete to “fix” state without understanding how kubelet will reconcile it.

Images, snapshots, and writable state

OCI images are content-addressed manifests, configuration, and compressed layers. A snapshotter prepares the root filesystem from those layers using the configured storage implementation. The container’s writable layer is runtime state, not a new immutable image and not durable application storage. Logs, volumes, and checkpoint behavior are separate integrations.

Multi-platform image indexes point to platform-specific manifests. A runtime selects a compatible operating system, architecture, and variant; a successful pull of the index does not prove every node architecture has a valid child manifest. Record both index and selected manifest digest during forensic collection. Garbage collection must preserve referenced content, while manual deletion from content stores can corrupt runtime assumptions.

Security boundaries remain layered

Namespaces, cgroups, capabilities, seccomp, LSM policy, user namespaces, mounts, and the kernel combine to form the isolation boundary. OCI compliance does not make two runtimes equally hardened, and a standard container shares the host kernel. RuntimeClass can select alternative handlers for workloads needing a different isolation model, but nodes, admission policy, scheduling, and observability must all support that choice.

Keep runtime, shim, low-level runtime, CNI components, and node kernel within the Kubernetes distribution’s supported matrix. Restrict access to runtime sockets: control over CRI or containerd commonly implies the ability to start privileged workloads or inspect sensitive container state. Rotate nodes through tested images rather than hand-upgrading an untracked set of runtime packages.

Failure evidence

When a pod fails before application startup, correlate kubelet events with CRI status, runtime logs, image pull errors, snapshotter capacity, CNI results, cgroup pressure, and kernel audit messages. Distinguish image resolution, unpack, sandbox creation, network setup, OCI create, process start, and application exit. Preserve versions and configuration for every layer; “containerd failed” is a symptom category, not a root cause.

Related:

Sources: