EKS CrashLoopBackOff: How to Diagnose and Fix Restarting Pods

Simple explanation

CrashLoopBackOff is not a crash itself. It is the status a pod shows when a container has crashed more than once and Kubernetes is waiting before trying again.

Think of it like a car that stalls every time you try to start it. After a few failed starts in a row, you stop cranking immediately and start waiting before the next attempt. The car has not fixed itself. The waiting is just Kubernetes not making things worse by hammering a broken container in a tight loop.

The sequence:

1. Container starts and crashes
2. Kubernetes restarts it
3. It crashes again
4. Kubernetes waits progressively longer before restarting (the “BackOff”)
5. Pod shows CrashLoopBackOff during each waiting period

The underlying problem is whatever is causing the crash. CrashLoopBackOff is the messenger, not the cause.

When to use this page

Use this page if kubectl get pods shows your pod in CrashLoopBackOff.

If your pod shows something else, you are likely on the wrong page:

Fastest way to diagnose EKS CrashLoopBackOff

The three commands that matter most#

1. Get the previous container’s logs

kubectl logs <pod-name> --previous -n <namespace>

2. Check the exit code

kubectl describe pod <pod-name> -n <namespace>

Scroll to Containers and find Last State. The exit code tells you the category of problem immediately. See the exit code reference section below.

3. Check cluster events

kubectl get events -n <namespace> --sort-by='.lastTimestamp' | grep <pod-name>

Events surface OOMKill notices, failed volume mounts, and probe failures that never appear in application logs.

Quick diagnostic checklist#

• Did kubectl logs --previous show an error message or stack trace?
• What is the exit code in Last State from kubectl describe pod?
• Is RESTARTS above 5? The container will not recover without a fix.
• Does Last State show Reason: OOMKilled? Memory limit is too low.
• Does kubectl describe pod show Liveness probe failed or Startup probe failed in Events? Check probe config.
• Is AWS_WEB_IDENTITY_TOKEN_FILE missing from the pod’s environment? IRSA or EKS Pod Identity is not configured correctly.

What to check based on what you see#

How CrashLoopBackOff works

Each time a container crashes, Kubernetes restarts it. If it keeps crashing, restarting it instantly would waste CPU, flood logs, and put unstable code under continuous load. So Kubernetes slows down using exponential backoff.

Think of it like calling a friend who keeps not picking up. You try, wait 10 seconds, try again. After a few attempts you start waiting a minute between tries. You have not given up, you are just being realistic about how often to bother.

The pod shows CrashLoopBackOff during each waiting period. The RESTARTS counter in kubectl get pods tells you how many times the container has crashed.

CrashLoopBackOff vs OOMKilled vs CreateContainerConfigError vs ImagePullBackOff

A pod can show CrashLoopBackOff because it was OOMKilled. The Reason field in Last State tells you which one it is. If you see Reason: OOMKilled, jump to the OOMKilled section.

Exit code reference

Common causes

Application startup error#

Signature: Exit code 1 in Last State. Logs show a stack trace, exception, or “failed to initialize” message.

What it usually means: The application threw an uncaught error during startup. This could be a bad config value, a failed database connection, a missing dependency, or invalid input at boot time.

How to confirm: Run kubectl logs --previous. The error message is almost always in the last few lines.

First fix: Read the error and fix the code or configuration. If it says “cannot connect to database,” verify the database is reachable and that the connection string environment variable is correct. See Debugging VPC Connectivity Problems if the app cannot reach an RDS or other AWS endpoint.

Wrong command or entrypoint#

Signature: Exit code 127. Logs may show “command not found” or may be completely empty.

What it usually means: The command or args field in the pod spec references a binary that does not exist inside the container image.

How to confirm: Check the pod spec’s command and args. Verify the binary exists in the image:

docker run --entrypoint sh my-api:latest -c "ls /usr/local/bin/"

First fix: Correct the binary path in the pod spec, or rebuild the image so the binary exists at the expected path. See Kubernetes Pods for how command and args interact with the Dockerfile ENTRYPOINT and CMD.

Missing environment variable#

Signature: Exit code 1. Logs show something like KeyError: 'DATABASE_URL', Missing required env var: API_KEY, or ValueError: invalid literal.

What it usually means: The application requires an environment variable that was not injected into the pod.

How to confirm: Check the pod spec’s env and envFrom sections. Compare what is defined against what the application expects at startup.

First fix: Add the missing variable directly to the pod spec, or reference it from a Secret or ConfigMap. If the value is sensitive, store it in Secrets Manager and inject it using Managing Secrets in Kubernetes.

If the Secret or ConfigMap is referenced in the pod spec but does not exist in the cluster, the pod will show CreateContainerConfigError rather than CrashLoopBackOff.

Failing liveness probe#

Signature: Liveness probe failed in pod events. Container restarts periodically even though the application appears to be running.

What it usually means: The liveness probe is misconfigured. The most common cause is initialDelaySeconds being shorter than the application’s startup time, so the probe starts checking before the app is ready and kills it.

How to confirm:

kubectl describe pod <pod-name> -n <namespace> | grep -A 15 "Liveness"

First fix: Increase initialDelaySeconds to longer than the application’s worst-case startup time. If the app takes 15 seconds to start, set initialDelaySeconds: 30 as a safe starting point.

Failing startup probe#

Signature: Startup probe failed in pod events. Container restarts during the initial startup window before the app finishes initializing.

What it usually means: The startup probe’s failureThreshold or periodSeconds is not giving the application enough time to boot. Once a startup probe exceeds failureThreshold failures, Kubernetes kills and restarts the container.

How to confirm: Check kubectl describe pod for Startup probe failed in the Events section.

First fix: Increase failureThreshold or periodSeconds. A startup probe with failureThreshold: 30 and periodSeconds: 10 gives the app 300 seconds to become ready before Kubernetes kills it. Once the startup probe passes, the liveness probe takes over.

OOMKilled#

Signature: Last State: Terminated, Reason: OOMKilled, exit code 137.

What it usually means: The container exceeded the memory limit defined in the pod spec and the Linux kernel killed it.

How to confirm: Check kubectl describe pod for OOMKilled in Last State. Then check the memory limit in the pod spec:

resources:
  limits:
    memory: "128Mi"

Check actual current usage:

kubectl top pod <pod-name> -n <namespace>

For historical memory metrics and spikes, use CloudWatch Container Insights if it is enabled on your EKS cluster. See Monitoring EKS.

First fix: Either increase the memory limit, or find the memory leak. If usage grows steadily over time, the application has a leak. If it spikes and settles, the limit is simply set too low.

IRSA misconfiguration#

Signature: Exit code 1. Logs show NoCredentialProviders, Unable to load credentials, AccessDenied, or similar AWS SDK errors before the application does any meaningful work.

What it usually means: The pod tried to initialize an AWS SDK client at startup and valid IAM credentials were not available because IRSA is not configured correctly.

Think of IRSA as a keycard system for your pod. The service account annotation is the keycard. The IAM trust policy is the door programmed to recognize that specific card. If the pod arrives without a keycard (missing annotation), or if the door does not recognize the card (wrong trust policy), access is denied and the application crashes before doing any real work.

The sequence that causes the crash:

1. Pod starts
2. Application initializes an AWS SDK client (to read from Secrets Manager, S3, DynamoDB, etc.)
3. SDK cannot find valid credentials
4. SDK throws an exception
5. Application exits — CrashLoopBackOff

How to confirm: Check the service account annotation:

kubectl get serviceaccount <sa-name> -n <namespace> -o yaml

The annotation must be present:

annotations:
  eks.amazonaws.com/role-arn: arn:aws:iam::111122223333:role/my-app-role

Verify the pod spec references the service account:

kubectl get pod <pod-name> -n <namespace> -o yaml | grep serviceAccountName

Check that the credential env vars were injected:

kubectl exec <pod-name> -n <namespace> -- env | grep AWS

If AWS_WEB_IDENTITY_TOKEN_FILE is missing, the service account annotation is wrong or the pod spec does not reference the annotated service account.

The IAM role’s trust policy must also allow the specific service account to assume the role. See IAM Roles for Service Accounts for the full trust policy structure and common mismatches.

EKS Pod Identity misconfiguration or missing agent#

EKS Pod Identity is the modern alternative to IRSA, available on EKS 1.24 and later. Instead of annotating the service account, you create a Pod Identity association using the EKS console or CLI.

Signature: Same as IRSA. Exit code 1, AWS SDK credential errors in logs at startup.

How to confirm: Check whether the EKS Pod Identity Agent add-on is installed:

aws eks list-addons --cluster-name <cluster-name>

Look for eks-pod-identity-agent in the output. If it is missing, credentials are never injected into pods regardless of how associations are configured.

Check existing Pod Identity associations:

aws eks list-pod-identity-associations \
  --cluster-name <cluster-name> \
  --namespace <namespace>

First fix: Install the eks-pod-identity-agent add-on from the EKS console or via eksctl. Then create the association between the service account and the IAM role. See IAM Roles for Service Accounts for a comparison of IRSA vs Pod Identity and when to use each.

When the container crashes before you can debug

Some containers crash in milliseconds — too fast to exec into or capture logs from.

Override the command temporarily:

Edit the deployment to replace the container command with a sleep so the container stays running:

kubectl edit deployment <deployment-name> -n <namespace>

Change the container command:

command: ["/bin/sh", "-c", "sleep 3600"]

Then exec in and run the original start command manually to see what happens:

kubectl exec -it <pod-name> -n <namespace> -- /bin/sh
# Run the original startup command here and observe the output

Use an init container to capture environment state:

Add a debug init container that writes environment variables and filesystem state to a log before the main container starts. Read its output with:

kubectl logs <pod-name> -c init-debug -n <namespace>

Init container logs persist and are readable even after the main container crashes. See Logging in Kubernetes for more on how container logs are collected and forwarded in EKS.