SRE concepts 3 min

What is OOMKilled Errors?

OOMKilled is a container termination reason indicating that the Linux out-of-memory killer terminated a process because memory could not be allocated. This commonly happens when a container reaches its cgroup memory limit, but it can also occur during node-wide memory exhaustion.

01 Mechanics

What happens during an OOM kill

When memory cannot be allocated, the Linux kernel may terminate a process with SIGKILL. If the container's main process is killed, the container commonly exits with code 137, although exit code 137 by itself is not proof of an OOM kill. Kubernetes memory limits are enforced reactively: a container may temporarily exceed its configured limit and may be killed when the kernel detects memory pressure.

Kubernetes then restarts the container according to its restart policy. If memory pressure recurs quickly the container can enter CrashLoopBackOff, with restarts spaced by an increasing delay. The Pod status shows OOMKilled as the last termination reason, which is where diagnosis normally starts.

02 Value

Common causes

OOM kills tend to come from a small number of recurring situations.

  • Limits set too low: the limit was estimated rather than derived from observed usage.
  • Memory leaks: usage grows steadily until the limit is reached, then repeats after restart.
  • Load-dependent allocation: per-request memory scales with traffic or payload size beyond what was provisioned.
  • Runtime heap misconfiguration: a JVM or similar runtime configured with a heap that leaves too little room for the rest of the process footprint.
  • Node memory exhaustion: the node runs out-of-memory and the kernel selects a process regardless of that container's own limit.
03 Limits

How OOM kills are diagnosed and fixed

Memory-usage patterns provide clues, not definitive diagnoses. Steady growth across comparable load can suggest a leak; sharp increases correlated with traffic can suggest load-dependent allocation; and repeated kills near a configured limit can suggest insufficient headroom. Confirm the cause with runtime metrics, allocation or heap profiles, workload data, cgroup limits, and node-memory conditions.

Raising the limit can be appropriate when measured peak usage and expected workload justify more memory. It is not a substitute for investigating leaks, unbounded allocation, node overcommitment, or workload behavior.

04 Comparison

OOMKilled vs evicted

An OOM kill and a Kubernetes eviction are different mechanisms. A container can be OOM-killed after exceeding its own cgroup limit, or when the node itself reaches OOM before the kubelet can reclaim enough memory. In a node-level OOM, the kernel selects a process using its OOM scoring.

A node-pressure eviction is initiated proactively by the kubelet to reclaim resources. The kubelet ranks Pods using factors including usage relative to requests and Pod priority. Do not infer "container limit problem" from OOMKilled or "well-behaved but unlucky Pod" from eviction without checking node pressure, requests, limits, QoS, priority, and actual usage.

Key takeaways

  • OOMKilled means the kernel terminated a process during an out-of-memory condition; it does not by itself prove that the container exceeded its own configured limit.
  • Exit code 137 commonly accompanies an OOM kill but is not proof of one on its own.
  • Memory-usage patterns suggest a cause; profiles, cgroup limits, and node conditions confirm it.
  • Eviction is a different mechanism: the kubelet reclaims proactively, while an OOM kill is the kernel reacting to allocation failure.

Frequently asked

Sources

Product

  • Agentic Production Engineering

Compliance

All systems normalBuilt in NYC

The autonomous system for production.
SOC 2, GDPR, and HIPAA compliant.