What is OOMKilled Errors?

#What happens during an OOM kill

When memory cannot be allocated, the Linux kernel may terminate a process with SIGKILL. If the container's main process is killed, the container commonly exits with code 137, although exit code 137 by itself is not proof of an OOM kill. Kubernetes memory limits are enforced reactively: a container may temporarily exceed its configured limit and may be killed when the kernel detects memory pressure.

Kubernetes then restarts the container according to its restart policy. If memory pressure recurs quickly the container can enter CrashLoopBackOff, with restarts spaced by an increasing delay. The Pod status shows OOMKilled as the last termination reason, which is where diagnosis normally starts.

#Common causes

OOM kills tend to come from a small number of recurring situations.

• Limits set too low: the limit was estimated rather than derived from observed usage.
• Memory leaks: usage grows steadily until the limit is reached, then repeats after restart.
• Load-dependent allocation: per-request memory scales with traffic or payload size beyond what was provisioned.
• Runtime heap misconfiguration: a JVM or similar runtime configured with a heap that leaves too little room for the rest of the process footprint.
• Node memory exhaustion: the node runs out-of-memory and the kernel selects a process regardless of that container's own limit.

#How OOM kills are diagnosed and fixed

Memory-usage patterns provide clues, not definitive diagnoses. Steady growth across comparable load can suggest a leak; sharp increases correlated with traffic can suggest load-dependent allocation; and repeated kills near a configured limit can suggest insufficient headroom. Confirm the cause with runtime metrics, allocation or heap profiles, workload data, cgroup limits, and node-memory conditions.

Raising the limit can be appropriate when measured peak usage and expected workload justify more memory. It is not a substitute for investigating leaks, unbounded allocation, node overcommitment, or workload behavior.

#OOMKilled vs evicted

An OOM kill and a Kubernetes eviction are different mechanisms. A container can be OOM-killed after exceeding its own cgroup limit, or when the node itself reaches OOM before the kubelet can reclaim enough memory. In a node-level OOM, the kernel selects a process using its OOM scoring.

A node-pressure eviction is initiated proactively by the kubelet to reclaim resources. The kubelet ranks Pods using factors including usage relative to requests and Pod priority. Do not infer "container limit problem" from OOMKilled or "well-behaved but unlucky Pod" from eviction without checking node pressure, requests, limits, QoS, priority, and actual usage.

#Key takeaways

• OOMKilled means the kernel terminated a process during an out-of-memory condition; it does not by itself prove that the container exceeded its own configured limit.
• Exit code 137 commonly accompanies an OOM kill but is not proof of one on its own.
• Memory-usage patterns suggest a cause; profiles, cgroup limits, and node conditions confirm it.
• Eviction is a different mechanism: the kubelet reclaims proactively, while an OOM kill is the kernel reacting to allocation failure.

#Frequently asked

How do you set memory limits correctly?

From observed usage under realistic peak load, with headroom above the peak. Limits estimated without measurement tend to produce either OOM kills or wasted capacity.

Why does a JVM get OOMKilled with a heap smaller than the limit?

Heap is only part of the process footprint. Metaspace, thread stacks, and native allocations add to it, and the total has to fit within the container limit.

What is CrashLoopBackOff?

The state where a container repeatedly fails and restarts, with Kubernetes increasing the delay between attempts. Repeated OOM kills are one common cause.

#Sources

• Kubernetes: managing resources for containers
• Kubernetes: node-pressure eviction