Distributed Tracing in AWS: X-Ray, Sampling, and Troubleshooting

Simple explanation

This is especially valuable in microservices architectures, where a single user action can trigger five or ten separate service calls. Without tracing, you are left manually correlating timestamps across multiple log groups, hoping the clocks align.

Why distributed tracing matters

A typical request might touch an API Gateway, a Lambda function, a DynamoDB table, and an external payment API. If the user sees a five-second response time, which hop caused it?

CloudWatch metrics show each service in isolation. CloudWatch Logs show each service’s text output. Neither one shows you the chain. You would need to manually correlate timestamps across multiple log groups and hope they tell a coherent story.

Distributed tracing solves this directly. Open one trace and see: API Gateway took 5ms, Lambda initialization took 300ms, the DynamoDB query took 4,600ms, the external API took 80ms. The culprit is obvious without any manual correlation.

AWS X-Ray also generates a service map: a live diagram of all services and the connections between them, with latency and error rates on each edge. This is useful not just for debugging but for understanding what your system actually looks like after months of changes.

How distributed tracing works in AWS

Here is a concrete end-to-end example using a common AWS pattern:

1. API Gateway receives the user request and generates a unique trace ID. It adds the ID to an HTTP header called X-Amzn-Trace-Id, then forwards the request to Lambda.
2. Lambda reads the trace header, records a segment (its own start time, end time, and result), and makes a call to DynamoDB through the AWS SDK. The SDK automatically creates a subsegment for the DynamoDB call and passes the trace ID along.
3. DynamoDB is a managed service, so X-Ray records the call as a subsegment on Lambda’s segment. You see the query duration directly on the Lambda timeline, not as a separate entry.
4. Lambda also calls an external payment API over HTTPS. The X-Ray SDK creates a subsegment for the outbound HTTP call and records its duration and response status.
5. X-Ray collects all segments and subsegments, links them by trace ID, and assembles the full trace. You see a Gantt-style timeline: the full request took 680ms, of which 550ms was the DynamoDB query.

Core concepts: traces, segments, and subsegments

X-Ray uses a three-level hierarchy to organize trace data:

Trace

A trace is the complete record of one request from start to finish, identified by a unique 96-bit trace ID that is propagated as an HTTP header across every service the request passes through. By collecting all segments that share the same trace ID, X-Ray assembles the full picture.

Segment

A segment is one service’s contribution to the trace. Each Lambda function, ECS task, or EC2 instance creates a segment when it handles the request. The segment records: service name, start and end time, whether it succeeded or faulted, and any annotations or metadata you attach.

Subsegment

A subsegment is a unit of work inside a segment. A DynamoDB query, an outbound HTTP call, or a custom code block wrapped with the SDK each appear as subsegments nested inside the parent segment. They let you see not just “Lambda took 800ms” but “Lambda took 800ms, of which 650ms was a DynamoDB GetItem call.”

Annotations and metadata

X-Ray lets you attach extra data to any segment. Annotations are indexed key-value pairs you can filter on in the console, useful for customer ID, order ID, or feature flag names. Metadata is arbitrary JSON stored but not indexed, useful for larger debugging details you want to inspect manually but do not need to search on.

from aws_xray_sdk.core import xray_recorder

# Searchable annotation — filterable in the X-Ray console
xray_recorder.current_segment().put_annotation('customer_id', '12345')

# Non-indexed metadata — stored but not searchable
xray_recorder.current_segment().put_metadata('order_details', {
    'items': 3,
    'total': 49.99
})

How trace propagation works

Propagation is what keeps all segments linked to the same trace across service boundaries. X-Ray uses an HTTP header called X-Amzn-Trace-Id:

X-Amzn-Trace-Id: Root=1-5e272119-2c3c0ade7f1a9f1234567890;Parent=53995c3f42cd8ad8;Sampled=1

When a request enters through API Gateway, X-Ray generates the root trace ID. API Gateway adds the header before forwarding to Lambda. Lambda reads the header, creates its segment linked to that trace ID, then passes the same header when calling downstream services. Each downstream service does the same.

Sampling rules

X-Ray does not trace every request. The default rule traces the first request each second, then 5% of additional requests. You can define custom sampling rules that match on service name, URL path, HTTP method, or other attributes to trace more (or fewer) requests for specific paths.

# List current sampling rules
aws xray get-sampling-rules

# Create a rule that traces 20% of POST requests to /checkout
aws xray create-sampling-rule --cli-input-json '{
  "SamplingRule": {
    "RuleName": "checkout-tracing",
    "ResourceARN": "*",
    "Priority": 1,
    "FixedRate": 0.20,
    "ReservoirSize": 5,
    "ServiceName": "payment-service",
    "ServiceType": "*",
    "Host": "*",
    "HTTPMethod": "POST",
    "URLPath": "/checkout",
    "Version": 1
  }
}'

For a deeper look at X-Ray features beyond the basics, see the AWS X-Ray Overview, which covers groups, insights, and pricing.

Enabling tracing in AWS services

How you enable X-Ray depends on where your code runs. This also varies depending on which compute option you are using.

Lambda

Set TracingConfig.Mode to Active on the function. No code changes are needed for basic tracing: Lambda automatically creates segments and passes the trace header to any downstream calls made through the AWS SDK. For a complete picture of Lambda observability, see monitoring Lambda with CloudWatch.

# Enable active tracing on an existing Lambda function
aws lambda update-function-configuration \
  --function-name my-api-handler \
  --tracing-config Mode=Active

API Gateway

Enable X-Ray tracing on the stage. API Gateway records its own segment and injects the X-Amzn-Trace-Id header into the request before forwarding to your integration. This is the entry point that starts the trace chain.

# Enable X-Ray on an API Gateway stage
aws apigateway update-stage \
  --rest-api-id abc123 \
  --stage-name prod \
  --patch-operations op=replace,path=/tracingEnabled,value=true

ECS and EC2

Run the X-Ray daemon as a sidecar container (ECS) or a background process (EC2). Your application SDK sends trace segments to the daemon over UDP port 2000. The daemon batches them and forwards them to X-Ray.

EKS

Deploy the X-Ray daemon as a DaemonSet so it runs on every node in the cluster. Your application pods send UDP packets to the daemon on port 2000. For a full guide to Kubernetes observability on AWS, see monitoring EKS.

When to use distributed tracing

Tracing is most valuable when a request touches more than one service and you need to understand the full latency breakdown across the chain.

Good fit for tracing

• Your system has multiple Lambda functions, ECS services, or microservices that call each other
• You are debugging intermittent slow requests and cannot tell which service is responsible
• You need to understand cold start impact on end-to-end latency, not just per-function latency
• You want to identify unnecessary sequential service calls that could be parallelized
• You are building SLAs and need p99/p95 latency data per service, not just per endpoint
• You want a visual dependency map as your system grows and changes over time

When logs or metrics are enough

Tracing has overhead and cost. If your system is simple, you may not need it yet:

• A single Lambda function with no downstream calls: structured logs are sufficient
• Batch jobs where success or failure counts matter but per-request latency does not
• Very early-stage apps with one or two services: add tracing before your first unexplained slowdown, not necessarily on day one

Distributed tracing vs logs vs metrics

These three tools answer different questions. Understanding the difference helps you reach for the right one quickly during an incident.

In practice you use all three together. An alert on a CloudWatch metric fires. You open the X-Ray service map to find the failing service (traces). You then read that service’s structured logs to see the exact error message and context (logs).

AWS X-Ray vs OpenTelemetry

When you instrument your application for tracing, you have two main choices: the X-Ray SDK or the AWS Distro for OpenTelemetry (ADOT). Both can send data to X-Ray.

Current AWS direction

AWS is actively investing in OpenTelemetry as the standard instrumentation path. The AWS Distro for OpenTelemetry (ADOT) is AWS’s own distribution of the OpenTelemetry Collector and SDKs, with native support for sending traces to X-Ray. For new projects, starting with ADOT means you are not locked to a single backend and your instrumentation will work if you add a second observability tool later.

If you are already using the X-Ray SDK, there is no urgency to migrate. Both are actively supported and both send data to the same X-Ray backend. The service map, trace viewer, and sampling rules all work the same way regardless of which SDK produced the segments.

Finding bottlenecks from a real trace

Once tracing is active, here is a repeatable step-by-step workflow for diagnosing a slow or failing request:

Step 1: Detect the issue

An alert fires: latency is up, error rate increased, or a user reported a slow page. Note the time window. Go to CloudWatch > X-Ray traces > Service map and set the time range to the incident window.

Step 2: Open the service map

Look for nodes with yellow or red rings. These indicate elevated error rates or latency on that service. Click the node to see its request volume, p50/p99 latency, and error breakdown (2xx, 4xx, 5xx). This narrows the problem to one or two services within seconds.

Step 3: Inspect individual traces

Switch to the Traces view and sort by duration descending. Click a slow trace to open the timeline. Look for:

• Long subsegments: a subsegment consuming most of the trace duration is the bottleneck. If it is a database call, investigate the query. If it is an outbound HTTP call, check the downstream service.
• Sequential calls that could be parallel: if your service calls three downstream APIs one after another, consider calling them concurrently to cut total latency.
• Initialization time on Lambda: the Initialization block appears on cold starts. A large block means runtime startup is slow. Consider provisioned concurrency for latency-sensitive functions.
• Red segments: these indicate unhandled exceptions. The segment detail shows the error message and stack trace captured by the X-Ray SDK.

Step 4: Correlate with logs

Once you have identified the problematic service and approximate time, open its log group in CloudWatch. Using structured logging with a trace ID field lets you filter log lines directly to that specific request. This gives you the error details that the trace timeline cannot show. For a complete incident workflow, see incident response with monitoring.

Step 5: Decide next action

With the specific service, subsegment, and error message identified, you can make a targeted fix: optimize the slow query, add a cache, parallelize calls, or fix the error in the identified function. No more guessing.

# Retrieve recent traces with errors from the last hour
aws xray get-trace-summaries \
  --start-time $(date -u -d '1 hour ago' +%s) \
  --end-time $(date -u +%s) \
  --filter-expression 'fault = true' \
  --query 'TraceSummaries[*].{Id:Id,Duration:Duration,HasError:HasError}'

For a complete guide to production debugging that goes beyond tracing alone, see debugging production systems in AWS.