How to Debug Production Issues in AWS: CloudWatch, Logs Insights & X-Ray

What production debugging actually means

Think of debugging a production issue the same way a doctor diagnoses a patient. The patient says “I feel terrible”: that is the alarm. The doctor checks vitals (blood pressure, temperature, heart rate): those are your CloudWatch metrics and dashboards. Next they order a blood test to find the specific abnormality: that is your Logs Insights query. Finally they trace the abnormality back to an underlying condition: that is X-Ray showing which part of the request chain is broken.

The critical distinction in production debugging is between three things:

• Symptom: what the alarm or user reports (“the checkout page is slow”)
• Signal: the metric or log pattern that confirms it (p99 Lambda Duration at 8 seconds)
• Root cause: the actual thing that needs fixing (a downstream payment API timing out)

Most debugging mistakes happen when engineers skip from symptom directly to guessing root causes. The workflow in this guide moves systematically: symptom, to signals, to root cause.

When this workflow applies

Use this guide whenever a production system is behaving differently from normal. Common situations include:

• A CloudWatch alarm fired on error rate, latency, or connection count
• Users are reporting slowness or errors but no alarm has fired yet
• Error rates or p99 latency spiked after a recent deployment
• Lambda functions are timing out or being throttled
• Your application is returning 5xx errors from API Gateway or an ALB
• Your database is refusing connections or response times have degraded
• A downstream service is slow and affecting your application’s response time
• Something broke intermittently and you cannot reproduce it locally

How the debugging workflow works

Every production debugging session follows the same six-step progression, regardless of what broke.

Step 1: Start with the alarm

The alarm notification is your first piece of information. A well-named alarm with a clear description tells you which service is affected, which metric crossed a threshold, what the threshold was, and when the alarm state changed.

The StateReason field in the alarm state tells you the exact metric value that triggered the alarm and the period it covered. This is more specific than just knowing the alarm fired.

# Get current state and reason for a specific alarm
aws cloudwatch describe-alarms \
  --alarm-names "api-handler-error-rate-high" \
  --query 'MetricAlarms[0].{State:StateValue,Reason:StateReason,Time:StateUpdatedTimestamp}'

# Get alarm history to see when it first fired
aws cloudwatch describe-alarm-history \
  --alarm-name "api-handler-error-rate-high" \
  --history-item-type StateUpdate \
  --max-records 10 \
  --query 'AlarmHistoryItems[*].{Timestamp:Timestamp,NewState:NewStateValue,Reason:NewStateReason}' \
  --output table

Step 2: Check dashboards to understand scope and timing

Open your CloudWatch dashboard and look at every metric on the board. The question you are answering here is: which metrics are abnormal and which are not?

• If Lambda error rate is up but Lambda duration is normal, the issue is in your code logic, not a slow dependency.
• If duration is also high, you likely have a slow downstream call. X-Ray will confirm which one.
• A gradual increase in error rate suggests resource exhaustion (filling up, running out of connections). A sudden spike suggests a bad deploy or a downstream service failure.

Note the exact timestamp when the degradation started. You will use it repeatedly when querying logs and traces. This timestamp is the anchor for your entire investigation.

Step 3: Query logs with CloudWatch Logs Insights

CloudWatch Logs Insights lets you query all streams in a log group simultaneously. The console is at CloudWatch → Logs → Logs Insights. Select the relevant log group and set the time range to the incident window.

Start broad to find the error distribution:

fields @timestamp, @message
| filter @message like /ERROR/ or @message like /Exception/ or @message like /error/
| sort @timestamp desc
| limit 100

If you have structured JSON logs, extract the fields that matter and group by error type:

fields @timestamp, level, error_code, message, request_id
| filter level = "ERROR"
| stats count(*) as occurrences by error_code, message
| sort occurrences desc
| limit 20

If one error code accounts for 90% of the errors, that is your root cause candidate. If errors are spread evenly across many codes, the problem is more likely infrastructure (connection limits, memory pressure) than application logic.

Step 4: Inspect traces with X-Ray

Once Logs Insights shows you the error pattern, AWS X-Ray traces show you exactly where in the request chain the problem lives. Open CloudWatch → X-Ray traces and filter for the same incident window.

In the trace timeline, look for:

• The widest bar: that is where the most time is being spent
• Red segments: unhandled exceptions with stack traces attached
• Subsegments labeled with service names (DynamoDB, RDS, external HTTP): is a dependency slow?
• A large Initialization block at the top of a Lambda trace: that is a cold start

# Get trace summaries for the incident window with errors
aws xray get-trace-summaries \
  --start-time 1742300400 \
  --end-time 1742304000 \
  --filter-expression 'fault = true OR error = true' \
  --query 'TraceSummaries[*].{Id:Id,Duration:Duration,HasFault:HasFault,HasError:HasError}' \
  --output table

Step 5: Correlate with changes

Most production incidents are caused by changes: a new code deploy, a configuration update, a traffic spike, or an upstream dependency changing behavior. Once you know when the incident started, ask what changed around that time.

• AWS CloudTrail: every API call in your account is logged here. Filter for the affected service and resource around the incident start time to find configuration changes.
• Deployment history: your CI/CD pipeline logs or AWS CodeDeploy records show what was deployed and when.
• Auto Scaling events: a scale-in that removed too many instances can cause latency spikes without triggering obvious error alarms.
• AWS Health Dashboard: check whether an AWS service itself reported degradation during your incident window.

# Check CloudTrail for Lambda configuration changes in the last 2 hours
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=ResourceName,AttributeValue=my-function \
  --start-time $(date -u -d '2 hours ago' --iso-8601=seconds) \
  --end-time $(date -u --iso-8601=seconds) \
  --query 'Events[*].{Time:EventTime,Event:EventName,User:Username}' \
  --output table

Step 6: Confirm the fix is actually working

After taking a resolution action, do not close the investigation until monitoring confirms recovery. The specific signs to look for:

• Error rate returns to its baseline value (usually near zero)
• Latency metrics (p50, p99) return to their pre-incident levels
• The CloudWatch alarm transitions from ALARM state to OK state
• Logs Insights queries for error patterns return no results or return to a normal low frequency

Keep watching for at least 10 to 15 minutes after metrics appear to recover. Some problems manifest intermittently and appear fixed before returning. Sustained normal metrics are the confirmation.

Which AWS tool should I use first?

CloudWatch metrics, CloudWatch Logs Insights, and AWS X-Ray each answer a different question. Knowing which one to open first saves significant time.

Useful Logs Insights queries for common problems

These queries address the patterns that come up most often in production debugging. Set the time range to your incident window before running them.

Lambda cold starts: identify cold start frequency and duration

filter @type = "REPORT"
| fields @requestId, @duration, @billedDuration, @memorySize, @maxMemoryUsed, @initDuration
| filter ispresent(@initDuration)
| sort @initDuration desc
| limit 50

Lambda timeouts: find invocations that hit the timeout limit

filter @message like /Task timed out/
| fields @timestamp, @logStream, @message
| sort @timestamp desc
| limit 50

Database connection errors: count connection failures over time

fields @timestamp, @message
| filter @message like /connection/ and (@message like /timeout/ or @message like /refused/ or @message like /exhausted/)
| stats count(*) as connection_errors by bin(5m)
| sort @timestamp asc

HTTP 5xx errors from API access logs: breakdown by endpoint

fields @timestamp, httpMethod, resourcePath, status, responseLatency
| filter status >= 500
| stats count(*) as error_count, avg(responseLatency) as avg_latency by resourcePath, httpMethod
| sort error_count desc

Find the slowest requests in a time window

fields @timestamp, @duration, @requestId
| filter @type = "REPORT"
| sort @duration desc
| limit 20

Common production scenarios

These are the failure patterns you will debug most often in AWS, with the signals that point to each one and the tools to check first.

Lambda cold starts or timeouts

Symptom: p99 latency spikes, but p50 looks normal. Some requests succeed while others take much longer.

Signals to check: Init Duration in CloudWatch Logs REPORT lines; Duration metric p99 in CloudWatch; Throttles metric spike.

Best tools first: CloudWatch Duration metric, then Logs Insights filtering for ispresent(@initDuration), then X-Ray to see the cold start block in a trace.

Likely root causes: cold starts due to low traffic keeping the function unloaded; timeout set too low for a downstream call; memory set too low causing the runtime to slow down.

For detailed steps, see the guide on monitoring Lambda functions and the Lambda failure debugging page.

API Gateway or ALB 5xx errors

Symptom: users see error pages; 5xx count rises on the CloudWatch dashboard.

Signals to check: 5xxError metric in CloudWatch for API Gateway or HTTPCode_Target_5XX_Count for ALB; latency percentiles.

Best tools first: CloudWatch dashboard to confirm scope, then Logs Insights against the API Gateway access log group querying for status >= 500, then X-Ray traces filtered for fault = true.

Likely root causes: backend Lambda or ECS task returning 5xx; integration timeout between API Gateway and the backend; unhealthy backend registered with the ALB. See the load balancer backend unhealthy guide if the ALB is involved.

RDS or database connection exhaustion

Symptom: application errors referencing “too many connections”, “connection refused”, or “connection pool exhausted”.

Signals to check: RDS DatabaseConnections metric compared to the instance’s max_connections limit; FreeableMemory metric; application error logs.

Best tools first: CloudWatch DatabaseConnections metric, then Logs Insights querying for connection error messages grouped by time bin to find the onset.

Likely root causes: connection pool in the application is too large for the database instance; a code change removed connection pool limits; an upstream traffic spike opened too many parallel connections. See the full RDS connection refused guide for remediation steps.

Slow downstream dependency or external API

Symptom: your service latency is high but CPU and error rate are normal. Users report slowness but nothing looks obviously broken.

Signals to check: Lambda Duration p99 rising while error rate stays low; X-Ray subsegment for an external HTTP call is the widest bar in traces.

Best tools first: X-Ray trace timeline. The wide subsegment labeled with the dependency name (DynamoDB, external-api.example.com, S3) immediately shows where time is spent.

Likely root causes: external API degradation; database query without an index running slowly at scale; network latency to a cross-region dependency. For deeper investigation, see distributed tracing with X-Ray.

Bad deploy or config change

Symptom: error rate or latency spikes immediately after a deployment. Everything was fine before the release.

Signals to check: error rate rise correlates with the deployment timestamp in CloudWatch; new error codes appear in Logs Insights that did not exist before the deploy.

Best tools first: CloudWatch error rate metric overlaid against the deployment event time, then Logs Insights for new error types, then CloudTrail for any configuration changes made around that time.

Likely root causes: bug in the new code; environment variable removed or renamed; IAM permission missing for a new API call. Rolling back the deploy is often the fastest resolution while you investigate.