AWS Incident Response Runbook: CloudWatch, X-Ray, Logs Insights & SNS

How incident response works in AWS

Think of your monitoring setup as a hospital triage system. The alarm is the paramedic that detects something is wrong and calls it in. The dashboard is the intake desk that establishes how serious it is and which department is affected. Logs Insights is the diagnostic lab — detailed test results for a specific patient. X-Ray is the imaging system that shows how organs (services) connect and where the blockage is. SNS is the paging system that routes the right alert to the right team.

Without all five pieces in place, you end up either missing incidents entirely (no alarm), wasting time on the wrong service (no dashboard), or diagnosing blind (no logs or traces). The Amazon CloudWatch overview covers how these components fit into the broader CloudWatch architecture.

Each tool answers a different question, and reaching for the right one first is half the battle:

• Alarms: Is something wrong right now?
• Dashboards: How bad is it and when did it start?
• Logs Insights: What specific error or event is happening inside this service?
• X-Ray: Which part of the request path is broken or slow?
• SNS: Who needs to know and how do we reach them?

When this workflow applies

This runbook fits any AWS incident where a service is producing errors, running slowly, or failing to serve traffic. Specific scenarios it handles well:

• Latency spike: API p99 response time jumps from 200ms to 4s
• 5xx surge: ALB HTTPCode_Target_5XX_Count spikes from near zero to hundreds per minute
• Lambda throttling: Throttles metric rises as a concurrency limit is hit during a traffic burst
• RDS load spike: Performance Insights surfaces a new expensive query degrading the database
• Downstream dependency failure: X-Ray shows faults on an external service segment while your internal logic is healthy
• Bad deploy: metrics degrade within minutes of a code or config change

When this workflow is the wrong fit:

• Security incidents (unauthorized access, IAM key compromise): investigate in CloudTrail and GuardDuty, not CloudWatch dashboards
• Data corruption or compliance incidents: these require a separate escalation path and may involve legal review
• Infrastructure-level AWS failures (availability zone outage, service degradation): check the AWS Health Dashboard first before assuming the cause is in your code

Before an incident happens

If you set these up only after something breaks, you lose the most valuable window. Get these in place while the system is running normally:

CloudWatch alarms on key metrics. At minimum: error rate, latency (p99), and healthy host count. For Lambda, add Throttles and Duration. For RDS, add DatabaseConnections and FreeStorageSpace. See creating alarms in CloudWatch for threshold guidance and CLI examples.

An SNS topic with confirmed subscriptions. Create a topic for production incidents and subscribe your on-call email, a PagerDuty endpoint, or a Slack-forwarding Lambda. AWS sends a confirmation link and does not deliver notifications until you click it. Test every subscription before you need it. More on how SNS topics and subscriptions work in the Amazon SNS overview and the SNS messaging model.

A pre-built incident dashboard. Build your CloudWatch dashboard with critical metrics pinned and accessible before an incident. The dashboard should show: request count, error rate, latency percentiles, and service-specific metrics for each tier of your stack. Building it during an incident wastes the first 15 minutes.

Structured JSON logs flowing to CloudWatch. Plain text logs make Logs Insights queries fragile. Structured logging gives every log event named fields — level, error_code, request_id, message — that you can filter and aggregate without text pattern matching. This is the difference between “search for errors” and “count errors by type in the last 30 minutes.”

X-Ray enabled on entry-point services. X-Ray only records traces from the moment it is enabled. Enable it during initial service setup on API Gateway, ALB targets, and public-facing Lambda functions. The AWS X-Ray overview covers instrumentation options. If you wait until an incident to enable it, the traces from that incident are gone.

A known runbook location. Every team member should know where this runbook lives. During a high-pressure incident, people revert to the familiar. Make the runbook the familiar thing.

How the incident lifecycle works

Every incident moves through six stages. The order matters: skipping stages creates false resolutions and repeat incidents.

1. Detect: an alarm fires or a user report surfaces; the team is notified
2. Assess impact: how many users are affected, which services, and since when
3. Isolate: narrow from “something is wrong” to “this specific component is failing for this specific reason”
4. Mitigate: take the action that restores service (rollback, scaling, config fix)
5. Communicate: keep stakeholders informed without pulling the responder off the investigation
6. Review: post-mortem covering timeline, root cause, and action items

Most teams handle steps 1, 4, and 5 well. The teams that improve fastest are disciplined about steps 2, 3, and 6. Proper impact assessment prevents over-escalation. Proper isolation prevents fixing the wrong thing. Post-mortems prevent recurrence.

Which AWS tool to use when

Each tool in the incident workflow answers a different question. Reaching for the right one first saves the time lost from diving into logs before you know which service to look at.

Step-by-step runbook

Step 1: Detect

A production incident starts with one of two signals: a CloudWatch alarm fires, or a user reports a problem. Alarms should fire first. If users report problems before your alarms do, your detection setup is too slow or too sparse.

The alarm notification tells you: which service is affected, which metric crossed which threshold, and when the state changed. If your alarm names and descriptions are vague, fix that now. A clear alarm description is worth five minutes during a real incident.

# Create the SNS topic that receives all incident notifications
aws sns create-topic --name production-incidents \
  --query 'TopicArn' --output text

# Subscribe on-call email
aws sns subscribe \
  --topic-arn <TOPIC_ARN> \
  --protocol email \
  --notification-endpoint <EMAIL>

# Subscribe a PagerDuty HTTPS endpoint
aws sns subscribe \
  --topic-arn <TOPIC_ARN> \
  --protocol https \
  --notification-endpoint <PAGERDUTY_ENDPOINT>

Wire an alarm to the topic. Set both —alarm-actions (fires when the alarm triggers) and —ok-actions (fires on recovery) so your team knows when the incident starts and when it ends:

aws cloudwatch put-metric-alarm \
  --alarm-name "api-5xx-rate-high" \
  --namespace "AWS/ApplicationELB" \
  --metric-name "HTTPCode_Target_5XX_Count" \
  --dimensions Name=LoadBalancer,Value=app/my-alb/<ALB_SUFFIX> \
  --statistic Sum \
  --period 60 \
  --threshold 10 \
  --comparison-operator GreaterThanThreshold \
  --evaluation-periods 2 \
  --alarm-actions <TOPIC_ARN> \
  --ok-actions <TOPIC_ARN>

For the full alarm setup including composite alarms and recommended thresholds, see creating CloudWatch alarms.

Step 2: Triage

The goal of triage is to answer three questions in under two minutes: how bad is it, when did it start, and which service is affected. Open the CloudWatch dashboard for the affected service before looking at anything else.

• Set the time range to start 30 minutes before the alarm fired.
• Identify which metrics deviated first. That metric points you toward the affected component.
• Note the exact timestamp of the first deviation. You will use this in every query you run.
• Check whether the pattern is sudden (bad deploy, downstream failure) or gradual (resource exhaustion, memory leak).

Step 3: Isolate

Isolation is detective work. You start with all the suspects (every service in your stack) and narrow down to one. X-Ray is the map that shows where each suspect was when the incident started. Logs Insights is the interview that gets them to explain what went wrong.

The goal is to go from “the API is returning errors” to something specific: “the Lambda function process-order is failing when it calls DynamoDB because a new GSI query is exceeding the provisioned read capacity.”

Start with Logs Insights when you already know which service is affected. Open CloudWatch → Logs → Logs Insights, select the log group, and set the time range to the incident window. Start broad:

fields @timestamp, @message
| filter @message like /ERROR/ or @message like /Exception/
| sort @timestamp desc
| limit 50

For services using structured JSON logs, extract fields for a more useful view:

fields @timestamp, level, error_code, message, request_id
| filter level = "ERROR"
| stats count(*) as occurrences by error_code
| sort occurrences desc
| limit 20

A sudden spike in a new error code that did not appear before the incident is a reliable root cause candidate. Compare the error distribution against a normal window earlier in the day to confirm what changed. For teams that want to alert on these patterns automatically, log-based metrics let you build CloudWatch alarms directly from log patterns.

Switch to X-Ray when the source of errors spans multiple services or when logs alone cannot tell you which component in the chain is slow. Open CloudWatch → X-Ray traces → Service map and look for nodes with red rings (faults) or orange rings (errors). The map shows at a glance whether the problem is in your service or in a downstream dependency it calls.

# Pull trace summaries with faults from a specific time window
# Replace start-time and end-time with Unix epoch values for your incident window
aws xray get-trace-summaries \
  --start-time 1700000000 \
  --end-time 1700003600 \
  --filter-expression 'fault = true' \
  --query 'TraceSummaries[*].{Id:Id,Duration:Duration,HasFault:HasFault}' \
  --output table

Sort traces by duration to find the slowest requests. Click into one representative trace and look at each segment: which is widest (most time spent), which has a red bar (unhandled error), and whether a subsegment for an external call (DynamoDB, RDS, HTTPS) is disproportionately large. The distributed tracing guide covers reading trace timelines in depth.

Step 4: Mitigate / Resolve

Resolution depends on what isolation found. Match the action to the signal:

Step 5: Communicate

During an active incident, the engineer investigating needs to stay focused on the problem. Assign a separate person to handle communication, sometimes called the incident commander or communicator.

That person posts a short factual update to a Slack incident channel or status page every 10–15 minutes:

• What is affected and what is confirmed unaffected
• Current status: investigating / cause identified / fix in progress / monitoring recovery
• Expected resolution time if known

Engineers focus on fixing. One person focuses on communicating. This prevents the responder from context-switching between debugging and writing status updates, which slows down both.

Step 6: Post-incident review

A post-mortem is a written record of what happened, why, and what specific changes will prevent it from happening again. It is not about blame. Teams that skip post-mortems fix the same incidents three times a year instead of once.

A useful post-mortem includes:

• Incident summary: one paragraph covering what failed, for how long, and what the user impact was
• Timeline: timestamped list of events from first detection to resolution, reconstructed from CloudWatch alarm history and log timestamps
• Root cause: specific and technical. “High load” is not a root cause. “A missing index caused full table scans on the orders table under peak traffic” is a root cause.
• Detection gap: how long between the incident starting and your team being paged? If longer than 5 minutes, tighten or add alarms.
• Action items: concrete tasks with owners and due dates: add the alarm, fix the missing index, add the circuit breaker

# Reconstruct the incident timeline from alarm state history
aws cloudwatch describe-alarm-history \
  --alarm-name "api-5xx-rate-high" \
  --history-item-type StateUpdate \
  --max-records 20 \
  --query 'AlarmHistoryItems[*].{Timestamp:Timestamp,NewState:NewStateValue,Reason:NewStateReason}' \
  --output table