Top Kubernetes Errors Decoded: OOMKilled, CrashLoopBackOff, ImagePullBackOff & Step-by-Step Fixes
· Sourabh G Kulkarni
Master common Kubernetes production errors: OOMKilled 137, CrashLoopBackOff, ImagePullBackOff, and Evicted pods with root causes and exact CLI triage steps.
Every engineer working with Kubernetes has stared at kubectl get pods only to be greeted by a sea of amber and red status messages: CrashLoopBackOff, OOMKilled, ImagePullBackOff, or Evicted.
To a newcomer, these error names look like cryptic verdicts handed down by an unforgiving orchestrator. But Kubernetes is fundamentally a deterministic state machine. When a pod fails, it fails for very specific reasons governed by the Linux kernel, container runtime interfaces, and resource quotas.
Here is an architectural triage guide to the most common Kubernetes production errors, explaining exactly what is happening beneath the abstraction layer and how to resolve them systematically.
1. OOMKilled (Exit Code 137): The Kernel's Executioner
When a pod status displays OOMKilled (or terminates with exit code 137), the application did not crash due to a standard internal software exception. It was forcefully terminated by the Linux Out-Of-Memory Killer (oom-killer).
The Underlying Mechanics
Exit codes greater than 128 indicate that a process was killed by a fatal Unix signal: $$\text{Exit Code} = 128 + \text{Signal Number}$$
For exit code 137: $$137 - 128 = 9 \implies \text{SIGKILL}$$
In Kubernetes, when you set container memory limits:
resources:
limits:
memory: "512Mi"
Kubernetes writes that value directly to the Linux cgroup memory subsystem:
/sys/fs/cgroup/memory/memory.limit_in_bytes.
When your application attempts to allocate physical RAM exceeding that cgroup threshold, the Linux kernel refuses to grant more memory. To protect the host operating system and neighboring pods on the node, the kernel immediately issues an uncatchable SIGKILL (kill -9) to the offending process.
How to Triage and Fix It
-
Verify the OOM Event:
kubectl describe pod <pod-name> -n <namespace>Look for the
Last Statesection:Last State: Terminated Reason: OOMKilled Exit Code: 137 -
Inspect Kernel dmesg / Container Logs:
kubectl logs <pod-name> --previous -n <namespace> -
Remediation Strategies:
- JVM / Node.js Heap Mismatch: If running Java or Node.js, ensure your application heap size (
-Xmxor--max-old-space-size) is configured to ~75% of the container memory limit, leaving 25% for native memory overhead and thread stacks. - Right-Size Memory Limits: Use Prometheus/Grafana or
kubectl top podto observe the container's true memory usage profile over 7 days and raise the limit accordingly.
- JVM / Node.js Heap Mismatch: If running Java or Node.js, ensure your application heap size (
2. CrashLoopBackOff: The Infinite Restart Trap
CrashLoopBackOff does not mean Kubernetes is broken. It means your container started, exited prematurely with a non-zero exit code, and Kubernetes is waiting through an exponential backoff period (10s, 20s, 40s, up to 5m) before trying to restart it again.
Common Root Causes
- Missing Environment Variables or Secrets: The application throws an unhandled configuration exception on startup.
- Failed Database Connectivity: The app requires a PostgreSQL/Redis connection during initialization, but the database is unreachable or secrets are wrong.
- Process Termination: The container's primary foreground process (
CMDorENTRYPOINT) finished executing and exited. (Remember: containers only run as long as their PID 1 process stays alive).
Systematic Debugging Workflow
-
Check the Previous Container's Crash Logs: Standard
kubectl logswill often return empty if the container is currently in its backoff sleep. Always inspect the logs of the container instance that just crashed:kubectl logs <pod-name> -n <namespace> --previous -
Inspect the Exit Code via Describe:
kubectl describe pod <pod-name> -n <namespace>- Exit Code 1: General application runtime error (inspect stack trace).
- Exit Code 137: OOMKilled (SIGKILL).
- Exit Code 143: Graceful shutdown requested by orchestrator (SIGTERM timeout).
- Exit Code 127: Command or binary specified in
command:/args:not found inside the container filesystem.
-
Ephemeral Debug Shell: If the container crashes too quickly to inspect, temporarily override the container's startup command with an infinite sleep loop in your Deployment manifest:
command: ["/bin/sh", "-c", "sleep 3600"]Then
execdirectly into the container to test environment variables and run the application binary manually:kubectl exec -it <pod-name> -n <namespace> -- /bin/sh
3. ImagePullBackOff & ErrImagePull: Registry Failures
ImagePullBackOff indicates that the kubelet on the worker node was unable to pull the container image specified in the pod manifest.
The Underlying Mechanics
When a pod is scheduled to a node, the kubelet communicates with the local container runtime (containerd or CRI-O) via the Container Runtime Interface (CRI) to pull the image layers. If the pull fails, the kubelet backs off exponentially before retrying.
Key Causes & Resolutions
-
Typo in Tag or Image Repository:
kubectl describe pod <pod-name> -n <namespace>Look at the Events log:
Failed to pull image "myregistry.azurecr.io/api:v1.2.9": rpc error: code = NotFoundResolution: Verify the exact tag exists in the registry.
-
Missing
imagePullSecrets: For private repositories (Docker Hub private repos, GitHub Packages, AWS ECR, GCP Artifact Registry), the worker node requires valid credentials.spec: imagePullSecrets: - name: my-registry-secret -
Network or Proxy Restrictions: The node lacks outbound internet access, or private VPC DNS cannot resolve the registry domain name.
4. Evicted: When the Node Runs Out of Air
An Evicted status occurs when the kubelet proactively terminates pods because the worker node itself has crossed a critical resource exhaustion threshold.
Hard Eviction Thresholds
By default, the kubelet enforces hard eviction limits:
memory.available < 100Minodefs.available < 10%(Root filesystem running out of disk space)imagefs.available < 15%(Container image storage exhausted)
When a node crosses a hard eviction threshold, the kubelet immediately terminates pods on that node, prioritizing:
BestEffortpods (pods with zero resource requests or limits).Burstablepods exceeding their requested memory allocations.
Resolution
- Clean up unused container images on nodes:
crictl rmi --prune. - Check node disk space:
kubectl describe node <node-name>. Look for conditionsDiskPressure: TrueorMemoryPressure: True. - Configure log rotation for container logs stored in
/var/log/pods.
Summary Cheat Sheet
| Pod Status | Primary Root Cause | Quickest Diagnostic Command |
|---|---|---|
| OOMKilled (137) | Container memory exceeded cgroup limit | kubectl describe pod <pod> | grep -A 3 "Last State" |
| CrashLoopBackOff | App threw fatal exception on boot | kubectl logs <pod> --previous |
| ImagePullBackOff | Wrong image tag or missing pull secret | kubectl describe pod <pod> | grep Events |
| Evicted | Host node is out of disk space or RAM | kubectl describe node <node> | grep Conditions |
Kubernetes error states are diagnostic signals, not roadblocks. Knowing how to read exit codes and event streams turns hours of frantic guessing into five minutes of calm, structured remediation.