Debugging Production Systems in Azure

The production debugging mindset

Production debugging is hypothesis-driven. You start with the alert or complaint that triggered the investigation, form a hypothesis about the likely cause, find data that confirms or refutes it, and iterate. The key discipline is not to start remediation until you understand the root cause — a fix based on a wrong hypothesis often masks the real problem or creates new ones.

The data sources available to you, in roughly decreasing order of precision:

1. Distributed traces — The exact path and timing of the failing request across all services
2. Structured application logs — What the application was doing and what it recorded
3. Exception details — The specific error, stack trace, and state at the point of failure
4. Dependency calls — Which downstream services were called and whether they succeeded
5. Infrastructure metrics — CPU, memory, disk, network at the time of the failure
6. Resource events and changes — Deployments, configuration changes, platform events

Step 1: Establish the timeline and scope

Before investigating the cause, establish when the problem started and how widespread it is. This narrows your search space dramatically.

// Establish the timeline: when did errors increase?
AppRequests
| where TimeGenerated > ago(6h)
| summarize
    TotalRequests = count(),
    Errors = countif(Success == false),
    AvgLatency = avg(DurationMs)
    by bin(TimeGenerated, 2m)
| extend ErrorRate = round(100.0 * Errors / TotalRequests, 2)
| render timechart

// Determine scope: which services are affected?
AppRequests
| where TimeGenerated > ago(1h)
| where Success == false
| summarize ErrorCount = count() by ServiceName = cloud_RoleName
| order by ErrorCount desc

// Determine user impact: how many users are experiencing errors?
AppRequests
| where TimeGenerated between (datetime(2026-03-19 14:20:00) .. datetime(2026-03-19 15:00:00))
| summarize
    TotalUsers = dcount(UserId),
    AffectedUsers = dcountif(UserId, Success == false)
    by bin(TimeGenerated, 10m)
| extend ImpactPercent = round(100.0 * AffectedUsers / TotalUsers, 1)

Step 2: Trace a specific failure end-to-end

Once you know the time range and affected endpoints, pick a specific failing request and trace it completely. Find its operation ID and use it to pull the full execution context.

// Find a specific failing request and get its operation ID
AppRequests
| where TimeGenerated between (datetime(2026-03-19 14:25:00) .. datetime(2026-03-19 14:35:00))
| where Success == false
| where Name == "POST /api/orders"
| project TimeGenerated, OperationId = operation_Id, DurationMs, ResultCode, UserId
| order by TimeGenerated desc
| limit 5

// Once you have an operation ID, trace the full journey
// Replace 'abc123' with the actual operation ID
let targetOp = "abc123def456789";
union AppRequests, AppDependencies, AppExceptions, AppTraces
| where OperationId == targetOp
| project
    TimeGenerated,
    ItemType = itemType,
    ServiceName = cloud_RoleName,
    Name,
    DurationMs,
    Success,
    ResultCode,
    ExceptionType,
    Message,
    OuterMessage
| order by TimeGenerated asc

Step 3: Diagnose dependency failures

Many production failures trace back to a downstream dependency — a database timing out, a third-party API returning errors, a message queue becoming unavailable. Application Insights dependency tracking makes this easy to detect.

// Find failing dependencies in the incident window
AppDependencies
| where TimeGenerated between (datetime(2026-03-19 14:20:00) .. datetime(2026-03-19 15:00:00))
| where Success == false
| summarize
    FailureCount = count(),
    AvgDuration = avg(DurationMs),
    MaxDuration = max(DurationMs),
    SampleMessage = any(Data)
    by Target, Type, Name
| order by FailureCount desc

// Compare dependency performance before and during the incident
let before = AppDependencies
| where TimeGenerated between (datetime(2026-03-19 13:00:00) .. datetime(2026-03-19 14:20:00))
| where Success == true
| summarize BaselineP95 = percentile(DurationMs, 95) by Target, Type;
let during = AppDependencies
| where TimeGenerated between (datetime(2026-03-19 14:20:00) .. datetime(2026-03-19 15:00:00))
| summarize
    IncidentP95 = percentile(DurationMs, 95),
    FailureRate = round(100.0 * countif(Success == false) / count(), 1)
    by Target, Type;
before
| join during on Target, Type
| extend LatencyIncrease = round((IncidentP95 - BaselineP95) / BaselineP95 * 100, 0)
| order by LatencyIncrease desc

Azure App Service diagnostics tools

For App Service applications, Azure provides a suite of built-in diagnostics beyond Log Analytics. These tools are especially useful when the application is behaving unexpectedly at the host level — memory pressure, process crashes, file system issues.

# View recent application crash logs
az webapp log download \
  --resource-group myRG \
  --name myapp \
  --log-file application-logs.zip

# Enable detailed error logging temporarily
az webapp config set \
  --resource-group myRG \
  --name myapp \
  --detailed-error-messages true \
  --failed-request-tracing true

# Get current application settings for configuration verification
az webapp config appsettings list \
  --resource-group myRG \
  --name myapp \
  --output table

# Check if the application is restarting unexpectedly
az monitor activity-log list \
  --resource-group myRG \
  --resource-provider Microsoft.Web \
  --start-time "2026-03-19T14:00:00Z" \
  --end-time "2026-03-19T16:00:00Z" \
  --query "[?contains(operationName.value, 'restart')]" \
  --output table

# Stream live logs during debugging
az webapp log tail \
  --resource-group myRG \
  --name myapp \
  --provider http

Snapshot Debugger: capturing state at the point of failure

When an exception occurs in production, the Snapshot Debugger can capture a “mini dump” of the application state — including local variable values, the call stack, and parameter values — at the exact moment the exception was thrown. This is invaluable for NullReferenceException, unexpected null values, or any bug where the “what were the values?” question is critical.

# Enable Snapshot Debugger on App Service
az webapp config appsettings set \
  --resource-group myRG \
  --name myapp \
  --settings \
    APPLICATIONINSIGHTS_CONNECTION_STRING="<connection-string>" \
    SNAPSHOTDEBUGGER_EXTENSION_VERSION="disabled"

# Install the Snapshot Debugger extension
az webapp extension add \
  --resource-group myRG \
  --name myapp \
  --extension Microsoft.ApplicationInsights.SnapshotDebugger

After enabling, navigate to Application Insights → Failures → select an exception → “Open Debug Snapshot” to view the captured state.

Correlating across multiple data sources

The most powerful debugging sessions combine data from multiple sources — application logs, infrastructure metrics, database logs, and platform events — in the same query window.

// Correlate App Service resource metrics with application errors
// This shows whether errors correlate with infrastructure stress
let errors = AppRequests
| where TimeGenerated > ago(2h)
| where Success == false
| summarize ErrorCount = count() by bin(TimeGenerated, 2m);
let cpuMetrics = AzureMetrics
| where TimeGenerated > ago(2h)
| where ResourceProvider == "MICROSOFT.WEB"
| where MetricName == "CpuPercentage"
| summarize MaxCPU = max(Maximum) by bin(TimeGenerated, 2m);
errors
| join cpuMetrics on TimeGenerated
| project TimeGenerated, ErrorCount, MaxCPU
| render timechart

// Correlate errors with recent deployments (Activity Log)
let incidentStart = datetime(2026-03-19 14:20:00);
AzureActivity
| where TimeGenerated between ((incidentStart - 2h) .. incidentStart)
| where OperationNameValue has "write"
| where ActivityStatusValue == "Success"
| where ResourceProviderValue == "MICROSOFT.WEB"
| project TimeGenerated,
    OperationName = OperationNameValue,
    Resource = _ResourceId,
    InitiatedBy = Caller
| order by TimeGenerated desc

Step 4: Verifying the fix is working

After deploying a fix, do not close the incident immediately. Verify the fix is actually working using the same metrics and logs that identified the problem.

// Verify error rate is returning to baseline after fix deployment
AppRequests
| where TimeGenerated > ago(2h)
| summarize
    Total = count(),
    Errors = countif(Success == false),
    AvgLatency = avg(DurationMs)
    by bin(TimeGenerated, 2m)
| extend ErrorRate = round(100.0 * Errors / Total, 2)
| extend Phase = iff(TimeGenerated < datetime(2026-03-19 15:00:00), "incident", "post-fix")
| render timechart

// Confirm the specific error pattern is gone
AppExceptions
| where TimeGenerated > ago(30m)
| where ExceptionType == "System.Data.SqlClient.SqlException"
| count
// Expected: 0 (or dramatically reduced from the incident count)