← All writing

Top Kubernetes Errors Decoded: OOMKilled, CrashLoopBackOff, ImagePullBackOff & Step-by-Step Fixes

· Sourabh G Kulkarni

Master common Kubernetes production errors: OOMKilled 137, CrashLoopBackOff, ImagePullBackOff, and Evicted pods with root causes and exact CLI triage steps.

Every engineer working with Kubernetes has stared at kubectl get pods only to be greeted by a sea of amber and red status messages: CrashLoopBackOff, OOMKilled, ImagePullBackOff, or Evicted.

To a newcomer, these error names look like cryptic verdicts handed down by an unforgiving orchestrator. But Kubernetes is fundamentally a deterministic state machine. When a pod fails, it fails for very specific reasons governed by the Linux kernel, container runtime interfaces, and resource quotas.

Here is an architectural triage guide to the most common Kubernetes production errors, explaining exactly what is happening beneath the abstraction layer and how to resolve them systematically.


1. OOMKilled (Exit Code 137): The Kernel's Executioner

When a pod status displays OOMKilled (or terminates with exit code 137), the application did not crash due to a standard internal software exception. It was forcefully terminated by the Linux Out-Of-Memory Killer (oom-killer).

The Underlying Mechanics

Exit codes greater than 128 indicate that a process was killed by a fatal Unix signal: $$\text{Exit Code} = 128 + \text{Signal Number}$$

For exit code 137: $$137 - 128 = 9 \implies \text{SIGKILL}$$

In Kubernetes, when you set container memory limits:

resources:
  limits:
    memory: "512Mi"

Kubernetes writes that value directly to the Linux cgroup memory subsystem: /sys/fs/cgroup/memory/memory.limit_in_bytes.

When your application attempts to allocate physical RAM exceeding that cgroup threshold, the Linux kernel refuses to grant more memory. To protect the host operating system and neighboring pods on the node, the kernel immediately issues an uncatchable SIGKILL (kill -9) to the offending process.

How to Triage and Fix It

  1. Verify the OOM Event:

    kubectl describe pod <pod-name> -n <namespace>
    

    Look for the Last State section:

    Last State:     Terminated
      Reason:       OOMKilled
      Exit Code:    137
    
  2. Inspect Kernel dmesg / Container Logs:

    kubectl logs <pod-name> --previous -n <namespace>
    
  3. Remediation Strategies:

    • JVM / Node.js Heap Mismatch: If running Java or Node.js, ensure your application heap size (-Xmx or --max-old-space-size) is configured to ~75% of the container memory limit, leaving 25% for native memory overhead and thread stacks.
    • Right-Size Memory Limits: Use Prometheus/Grafana or kubectl top pod to observe the container's true memory usage profile over 7 days and raise the limit accordingly.

2. CrashLoopBackOff: The Infinite Restart Trap

CrashLoopBackOff does not mean Kubernetes is broken. It means your container started, exited prematurely with a non-zero exit code, and Kubernetes is waiting through an exponential backoff period (10s, 20s, 40s, up to 5m) before trying to restart it again.

Common Root Causes

  • Missing Environment Variables or Secrets: The application throws an unhandled configuration exception on startup.
  • Failed Database Connectivity: The app requires a PostgreSQL/Redis connection during initialization, but the database is unreachable or secrets are wrong.
  • Process Termination: The container's primary foreground process (CMD or ENTRYPOINT) finished executing and exited. (Remember: containers only run as long as their PID 1 process stays alive).

Systematic Debugging Workflow

  1. Check the Previous Container's Crash Logs: Standard kubectl logs will often return empty if the container is currently in its backoff sleep. Always inspect the logs of the container instance that just crashed:

    kubectl logs <pod-name> -n <namespace> --previous
    
  2. Inspect the Exit Code via Describe:

    kubectl describe pod <pod-name> -n <namespace>
    
    • Exit Code 1: General application runtime error (inspect stack trace).
    • Exit Code 137: OOMKilled (SIGKILL).
    • Exit Code 143: Graceful shutdown requested by orchestrator (SIGTERM timeout).
    • Exit Code 127: Command or binary specified in command: / args: not found inside the container filesystem.
  3. Ephemeral Debug Shell: If the container crashes too quickly to inspect, temporarily override the container's startup command with an infinite sleep loop in your Deployment manifest:

    command: ["/bin/sh", "-c", "sleep 3600"]
    

    Then exec directly into the container to test environment variables and run the application binary manually:

    kubectl exec -it <pod-name> -n <namespace> -- /bin/sh
    

3. ImagePullBackOff & ErrImagePull: Registry Failures

ImagePullBackOff indicates that the kubelet on the worker node was unable to pull the container image specified in the pod manifest.

The Underlying Mechanics

When a pod is scheduled to a node, the kubelet communicates with the local container runtime (containerd or CRI-O) via the Container Runtime Interface (CRI) to pull the image layers. If the pull fails, the kubelet backs off exponentially before retrying.

Key Causes & Resolutions

  1. Typo in Tag or Image Repository:

    kubectl describe pod <pod-name> -n <namespace>
    

    Look at the Events log:

    Failed to pull image "myregistry.azurecr.io/api:v1.2.9": rpc error: code = NotFound
    

    Resolution: Verify the exact tag exists in the registry.

  2. Missing imagePullSecrets: For private repositories (Docker Hub private repos, GitHub Packages, AWS ECR, GCP Artifact Registry), the worker node requires valid credentials.

    spec:
      imagePullSecrets:
        - name: my-registry-secret
    
  3. Network or Proxy Restrictions: The node lacks outbound internet access, or private VPC DNS cannot resolve the registry domain name.


4. Evicted: When the Node Runs Out of Air

An Evicted status occurs when the kubelet proactively terminates pods because the worker node itself has crossed a critical resource exhaustion threshold.

Hard Eviction Thresholds

By default, the kubelet enforces hard eviction limits:

  • memory.available < 100Mi
  • nodefs.available < 10% (Root filesystem running out of disk space)
  • imagefs.available < 15% (Container image storage exhausted)

When a node crosses a hard eviction threshold, the kubelet immediately terminates pods on that node, prioritizing:

  1. BestEffort pods (pods with zero resource requests or limits).
  2. Burstable pods exceeding their requested memory allocations.

Resolution

  • Clean up unused container images on nodes: crictl rmi --prune.
  • Check node disk space: kubectl describe node <node-name>. Look for conditions DiskPressure: True or MemoryPressure: True.
  • Configure log rotation for container logs stored in /var/log/pods.

Summary Cheat Sheet

Pod StatusPrimary Root CauseQuickest Diagnostic Command
OOMKilled (137)Container memory exceeded cgroup limitkubectl describe pod <pod> | grep -A 3 "Last State"
CrashLoopBackOffApp threw fatal exception on bootkubectl logs <pod> --previous
ImagePullBackOffWrong image tag or missing pull secretkubectl describe pod <pod> | grep Events
EvictedHost node is out of disk space or RAMkubectl describe node <node> | grep Conditions

Kubernetes error states are diagnostic signals, not roadblocks. Knowing how to read exit codes and event streams turns hours of frantic guessing into five minutes of calm, structured remediation.