Go GOMEMLIMIT in Kubernetes: GC Headroom, Cgroups, and OOM Risk

Kubernetes memory limits are enforced by the kernel and can end in an OOMKilled container. Go’s GOMEMLIMIT, introduced in Go 1.19, is a separate soft target that asks the Go runtime to manage garbage collection and memory release so the runtime’s memory use stays near a configured budget. It does not replace or automatically discover the container memory limit, does not reserve memory from the kernel, and does not account for every process allocation.

The safe relationship is usually to set the Go runtime budget below the container limit, leaving measured headroom for memory sources outside the runtime’s accounting. Setting GOMEMLIMIT too close to the cgroup limit can still end in a kernel OOM; setting it too low can push the runtime into excessive garbage-collection work and unacceptable latency. The official Go garbage-collector guide explains both why the setting is soft and how it interacts with GOGC.

Understand which memory the Go limit measures

GOMEMLIMIT and runtime/debug.SetMemoryLimit express the memory budget for a Go process in bytes, optionally using binary suffixes such as MiB or GiB. The runtime’s documented accounting is runtime.MemStats.Sys - runtime.MemStats.HeapReleased, or equivalently /memory/classes/total:bytes - /memory/classes/heap/released:bytes from runtime/metrics.

That accounting includes Go runtime-managed memory that has not been released, but it excludes important sources such as the Go binary’s own mappings, memory allocated by C code, memory mapped with syscall.Mmap, and memory the operating system holds on behalf of the process. It is therefore not the same measurement as container RSS or the cgroup’s total memory use. A service using cgo, native libraries, memory-mapped files, or child processes may need substantially more headroom than a small pure-Go service.

The setting applies to one Go runtime, not to every process or sidecar in a Pod. If the container runs helper processes, they also consume the container’s cgroup budget but are not controlled by that Go process’s GOMEMLIMIT. Likewise, a sidecar has its own container resource accounting; model the Pod’s aggregate requests and limits separately from each container’s runtime budget.

Kubernetes memory limits operate at a different boundary. The container runtime configures a cgroup and the kernel responds to memory pressure, commonly by killing a process in the constrained cgroup. The runtime’s soft memory target cannot prevent an OOM kill if external allocations or a transient live set push the process or container beyond available memory. Treat it as an input to the Go garbage collector, not as a kernel-enforced cap.

Choose the budget with measured headroom

Start with the container memory limit and measure the difference between cgroup-level usage and Go runtime-managed usage under realistic load. Leave headroom for native memory, memory mappings and other process overhead outside the runtime limit’s accounting, transient allocation peaks, and monitoring or helper processes that share the same container. Go-managed goroutine stacks are included in the runtime’s memory accounting; native-library or operating-system allocations are not necessarily included. The Go GC guide offers 5–10% as a rule of thumb for additional memory sources in a controlled containerized application, but that is not a universal production target. Native libraries, cgo, large mappings, and workload spikes can require more. Select the margin from measurements and leave room for uncertainty.

For example, a container with a 1Gi memory limit might begin testing with an 800MiB Go runtime budget. That leaves nominal room for memory the Go runtime does not manage, but it is only a starting experiment: the correct value depends on the process’s real memory profile and the memory requests of other processes that share the same cgroup.

# Relevant container-template excerpt; set this only after measuring the process.
resources:
  requests:
    memory: "768Mi"
  limits:
    memory: "1Gi"
env:
  - name: GOMEMLIMIT
    value: "800MiB"

The request and limit above have different roles. The request contributes to scheduling and memory-pressure decisions; the limit is enforced by the kernel. The 800MiB value is the Go runtime’s soft budget, not a promise that the container will never use more than 800MiB or that the remaining 224MiB will always be sufficient.

Set the environment value in the workload’s source of truth and roll it out through the usual Deployment or StatefulSet update. Changing the environment of an existing process does not alter its environment in place; the process must restart with the new value. If a controller, platform chart, or deployment system owns the Pod template, update that owner rather than relying on an ephemeral manual edit.

For applications that tune the value dynamically, runtime/debug.SetMemoryLimit can change the current runtime budget. This enables adaptation when an application has a well-defined mode transition or a controlled container resize, but it also creates another configuration input to observe and test. A negative argument reads the current limit without changing it. Keep one clear owner for the value; an environment setting at startup and later application code can otherwise make a configuration audit confusing.

Balance `GOMEMLIMIT` with `GOGC`

GOGC controls the garbage collector’s target growth relative to live heap. A higher value usually allows more heap growth and less collection CPU at the cost of memory; a lower value reduces heap growth while increasing collection frequency and CPU work. GOMEMLIMIT adds an absolute runtime memory target. When that target is under pressure, the Go runtime may collect more frequently than the configured GOGC target would otherwise require.

The runtime continues to respect the soft memory limit even if GOGC=off or debug.SetGCPercent(-1) disables the normal growth-based trigger. This does not mean GOGC=off plus an arbitrarily small memory limit is a safe configuration. A process with a large live heap or a limit that leaves no room for runtime and external memory may spend excessive CPU in garbage collection, make little progress, and still exceed the soft target during transient growth.

The GC guide describes a CPU-work limiter that prevents memory-limit enforcement from consuming all application CPU under every condition. That is a safeguard against pathological GC thrashing, not a latency guarantee. If the runtime delays collection to preserve application progress, memory can temporarily rise above the target; if it collects more aggressively, throughput and tail latency can suffer. Monitor both sides of the trade-off rather than assuming the limit turns memory pressure into a harmless event.

Avoid optimizing by setting only GOGC or only GOMEMLIMIT from a rule copied out of context. First establish a stable live-heap baseline, allocation rate, GC CPU share, and external memory use. Then vary one setting at a time under load. A service with bursty traffic may need a budget that accommodates the peak live set; one with a large native allocation may need explicit native memory planning regardless of Go heap tuning.

Observe Go memory and cgroup memory together

Record at least two independent views:

Go runtime-managed memory, including the runtime memory-limit metric or the MemStats.Sys - HeapReleased equivalent.
Container or cgroup memory usage and limit as exposed by the node’s metrics pipeline.
Heap live bytes, allocation rate, GC cycle frequency, GC CPU time, and application latency.
OOMKilled events, exit codes, restart counts, and Pod-level memory pressure or eviction signals.

Do not expect the runtime total to equal container RSS. A rising gap can indicate native allocation, memory maps, other processes, or measurement differences. A small Go heap does not prove the cgroup is safe if a C library or mapped file dominates memory. A Go runtime value near GOMEMLIMIT does not prove an OOM is imminent either; use the cgroup margin and the recent rate of change to understand actual risk.

For a first response to OOMKilled, establish which container was killed and compare its resource limit, Go runtime metrics, cgroup usage, recent heap growth, and deployment revision. A likely memory leak, a brief live-set spike, a too-low soft limit, a native-memory increase, and a container limit that is too small require different responses. Raising GOMEMLIMIT during an incident can make an OOM more likely; lowering it can increase GC work and worsen latency. Change it only with evidence about which memory source is consuming the budget.

Test the limit as an operational change

Use a staging environment with the same Go toolchain, module go directive, cgroup version, runtime, and resource limits as production. Exercise steady-state load, burst traffic, large requests, garbage-collection-heavy paths, and any cgo or memory-mapping behavior. Record a baseline, reduce or increase the runtime budget in a controlled step, and compare:

Peak cgroup memory and the difference between cgroup use and Go runtime-managed use.
GC frequency, GC CPU time, allocation rate, and application throughput.
p95/p99 latency and queue growth while the GC approaches the limit.
Whether memory can temporarily exceed the soft budget without crossing the container limit.
Restart count, OOMKilled reasons, and behavior during a rollout or in-place memory resize.

Use a representative memory limit rather than a tiny artificial container that has no similarity to production. Test a failure case where the live heap itself is larger than the proposed limit; the runtime should not be expected to manufacture free memory. Also test a case where a native allocation grows, because the Go GC cannot reclaim memory it does not own. Define a rollback value and retain enough deployment history to restore the previous known-good configuration.

Production checklist

The deployed Go binary supports the expected memory-limit controls and logs its current value.
GOMEMLIMIT is treated as a Go-runtime soft target, not as a replacement for a Kubernetes memory limit.
The chosen budget is below the cgroup limit by a measured margin for C, mmap, binary, helper-process, and transient memory.
Dashboards show Go runtime-managed memory beside cgroup/container usage, GC work, latency, and OOM/restart events.
Changes to GOGC and GOMEMLIMIT are tested independently against realistic traffic and live-set sizes.
OOM and GC-thrashing runbooks distinguish Go-managed heap pressure from native or other process memory.
The workload’s source of truth owns the environment value, and a rollback path is tested.

GOMEMLIMIT is valuable when the Go process owns a predictable portion of a controlled memory budget. It gives the garbage collector a signal about finite memory instead of relying only on heap growth. Its soft nature is intentional: the runtime may prioritize making progress over staying strictly below the target. Safe operation therefore depends on realistic headroom, paired runtime and cgroup telemetry, and load tests that show the chosen GC trade-off is acceptable.

Related:

Sources: