Distributed Tracing in Azure

How distributed tracing works

Distributed tracing builds on a simple idea: every request gets a globally unique ID (the trace ID), and every unit of work within that request gets a locally unique ID (the span ID). When service A calls service B, it passes both IDs in the request headers. Service B creates a new span with its own span ID, but records service A’s span ID as its parent. The result is a tree structure — the trace — that shows exactly how the request flowed through the system.

The standard headers used for this propagation are defined in the W3C Trace Context specification:

• traceparent — Contains the version, trace ID, parent span ID, and trace flags. Example: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
• tracestate — Vendor-specific data that travels alongside the trace. Application Insights uses this to pass the operation name.

Application Insights reads these headers on inbound requests and injects them on outbound calls automatically when you use the SDK, so in many cases trace propagation works without any manual code.

Spans, requests, and dependencies

In Application Insights terminology:

• Request — An inbound operation received by a service (HTTP, gRPC, queue message consumption). This becomes the root span for that service’s contribution to the trace.
• Dependency — An outbound call made by a service (HTTP call, SQL query, Redis command, Service Bus message send). Each dependency call becomes a child span.
• Exception — An error recorded within a span.
• Trace / Log — Custom messages recorded within the context of a span.

All four types are linked by the operation ID (the trace ID). This means you can query all of them together using the operation ID as a join key.

End-to-end transaction diagnostics: a real example

Scenario: A user reports that checkout is slow. The request ID from the browser’s network tab is 4bf92f3577b34da6a3ce929d0e0e4736. Here is how you investigate using KQL.

// Step 1: Find the root request and its duration
AppRequests
| where OperationId == "4bf92f3577b34da6a3ce929d0e0e4736"
| project TimeGenerated, Name, DurationMs, ResultCode, Success,
    RoleName = cloud_RoleName

// Step 2: See all spans (requests + dependencies) in the trace, ordered by time
union AppRequests, AppDependencies
| where OperationId == "4bf92f3577b34da6a3ce929d0e0e4736"
| project TimeGenerated,
    SpanType = itemType,
    Name,
    Target,
    DurationMs,
    Success,
    ResultCode,
    RoleName = cloud_RoleName,
    ParentId = operation_ParentId
| order by TimeGenerated asc

// Step 3: Find the slowest spans in this trace
union AppRequests, AppDependencies
| where OperationId == "4bf92f3577b34da6a3ce929d0e0e4736"
| project Name, DurationMs, SpanType = itemType, RoleName = cloud_RoleName
| order by DurationMs desc
| limit 10

// Step 4: Check for exceptions in this trace
AppExceptions
| where OperationId == "4bf92f3577b34da6a3ce929d0e0e4736"
| project TimeGenerated, ExceptionType, OuterMessage, InnermostMessage, RoleName = cloud_RoleName

Sampling: keeping costs under control

On a high-traffic application, storing every trace for every request is prohibitively expensive. Application Insights uses sampling to retain a statistically representative subset of traces. There are three sampling modes:

Importantly, sampling is trace-aware: all spans belonging to the same trace are either kept or dropped together. You will never see half a trace in your data — if any span from a trace is sampled in, all related spans are sampled in.

Enabling cross-service tracing

For tracing to work across service boundaries, each service must use a compatible SDK that reads and propagates the W3C traceparent header. Application Insights SDKs do this automatically for HTTP calls. For messaging systems like Azure Service Bus and Event Hubs, the SDK auto-instruments message send and receive operations.

# Verify that an Application Insights connection string is configured
az webapp config appsettings list \
  --resource-group myRG \
  --name myapp \
  --query "[?name=='APPLICATIONINSIGHTS_CONNECTION_STRING']" \
  --output table

# Set the connection string if missing
az webapp config appsettings set \
  --resource-group myRG \
  --name myapp \
  --settings APPLICATIONINSIGHTS_CONNECTION_STRING="InstrumentationKey=<key>;IngestionEndpoint=https://eastus-8.in.applicationinsights.azure.com/"

# For .NET apps, also set the role name so traces show which service they came from
az webapp config appsettings set \
  --resource-group myRG \
  --name myapp \
  --settings APPLICATIONINSIGHTS_ROLE_NAME="OrderService"

OpenTelemetry integration

For new applications, Microsoft recommends using the Azure Monitor OpenTelemetry distro instead of the legacy Application Insights SDK. OpenTelemetry is a vendor-neutral standard, which means your instrumentation code is not tied to Azure. If you ever need to send traces to a different backend, you change the exporter configuration — not the instrumentation code.

# Install the Azure Monitor OpenTelemetry distro for .NET
dotnet add package Azure.Monitor.OpenTelemetry.AspNetCore

# For Node.js
npm install @azure/monitor-opentelemetry

# For Python
pip install azure-monitor-opentelemetry

The Azure Monitor distro automatically instruments HTTP clients, SQL, Redis, Azure SDK calls, and more — the same auto-instrumentation as the legacy SDK, but through the standardized OpenTelemetry pipeline.