Incident Response with Azure Monitor

An incident is a race against time. The goal is to detect the problem fast, understand it accurately, and restore service before the impact grows. Azure Monitor does not just observe your system — it can be the backbone of your incident response process: alerting on the right conditions, routing notifications to the right people, and providing the data you need to understand and fix the problem quickly.

The five phases of incident response

Effective incident response follows a consistent structure. Azure Monitor tools support each phase:

  1. Detection — An alert fires. Azure Monitor metric alerts, log search alerts, and availability tests detect abnormal conditions and notify the on-call team via action groups.
  2. Triage — The on-call engineer assesses severity. Operations dashboards provide immediate situational awareness. Is this affecting all users or a subset? One region or all regions? Which service is the source?
  3. Investigation — The engineer digs into the cause. KQL queries, distributed traces, Application Map, and profiler data help identify the root cause.
  4. Resolution — The engineer takes action — deploys a fix, scales a resource, restarts a service, rolls back a deployment. Azure Monitor Action Groups can trigger automated remediation runbooks.
  5. Post-incident review — The team analyzes what happened, why it was not caught earlier, and how to prevent recurrence. Log Analytics provides historical data for the entire incident timeline.

From alert to action: automated remediation

For well-understood failure modes, you can automate the remediation entirely. When an alert fires, the action group can trigger an Azure Automation runbook that takes corrective action before a human engineer even sees the notification.

# Create an Automation account for remediation runbooks
az automation account create \
  --resource-group myRG \
  --name myAutomationAccount \
  --location eastus \
  --sku Free

# Create an action group that triggers an automation runbook
az monitor action-group create \
  --resource-group myRG \
  --name "AutoRemediation" \
  --short-name "autorem" \
  --automation-runbook-receiver \
    name="RestartAppService" \
    automation-account-id=/subscriptions/<sub-id>/resourceGroups/myRG/providers/Microsoft.Automation/automationAccounts/myAutomationAccount \
    runbook-name="Restart-AppService" \
    webhook-resource-id=/subscriptions/<sub-id>/resourceGroups/myRG/providers/Microsoft.Automation/automationAccounts/myAutomationAccount/webhooks/RestartWebhook \
    is-global-runbook=false \
    use-common-alert-schema=true

# Example: auto-scale out when CPU is sustained high
az monitor metrics alert create \
  --name "HighCPUAutoScale" \
  --resource-group myRG \
  --scopes /subscriptions/<sub-id>/resourceGroups/myRG/providers/Microsoft.Web/serverFarms/myAppServicePlan \
  --condition "avg CpuPercentage > 85" \
  --window-size 10m \
  --evaluation-frequency 5m \
  --severity 2 \
  --action /subscriptions/<sub-id>/resourceGroups/myRG/providers/microsoft.insights/actionGroups/AutoRemediation

Structured incident investigation with KQL

Here is a complete KQL-based investigation sequence for a production incident. Start broad, then narrow to the specific failure.

// Phase 1: Get the full timeline of the incident window
// Understand when things changed
AppRequests
| where TimeGenerated between (datetime(2026-03-19 14:00:00) .. datetime(2026-03-19 15:30:00))
| summarize
    RequestRate = count(),
    ErrorRate = countif(Success == false),
    AvgLatency = avg(DurationMs)
    by bin(TimeGenerated, 2m)
| extend ErrorPercent = round(100.0 * ErrorRate / RequestRate, 1)
| render timechart
// Phase 2: Find which endpoints are affected
AppRequests
| where TimeGenerated between (datetime(2026-03-19 14:20:00) .. datetime(2026-03-19 15:00:00))
| where Success == false
| summarize
    ErrorCount = count(),
    DistinctUsers = dcount(UserId),
    AvgDuration = avg(DurationMs)
    by Name, ResultCode
| order by ErrorCount desc
// Phase 3: Correlate with deployments in the Activity Log
AzureActivity
| where TimeGenerated between (datetime(2026-03-19 13:00:00) .. datetime(2026-03-19 14:30:00))
| where OperationNameValue contains "write"
| where ActivityStatusValue == "Success"
| where ResourceProviderValue in ("MICROSOFT.WEB", "MICROSOFT.CONTAINERSERVICE")
| project TimeGenerated, OperationNameValue, ResourceGroup, _ResourceId, Caller
| order by TimeGenerated desc
// Phase 4: Find the specific exception causing failures
AppExceptions
| where TimeGenerated between (datetime(2026-03-19 14:20:00) .. datetime(2026-03-19 15:00:00))
| summarize ExceptionCount = count() by ExceptionType, OuterMessage
| order by ExceptionCount desc
| limit 10
// Phase 5: Find the first occurrence of the error pattern (incident start time)
AppExceptions
| where TimeGenerated > ago(6h)
| where ExceptionType == "System.Data.SqlClient.SqlException"
| summarize FirstOccurrence = min(TimeGenerated), LastOccurrence = max(TimeGenerated), Count = count()
| project FirstOccurrence, LastOccurrence, Count, Duration = LastOccurrence - FirstOccurrence

Azure Change Analysis: what changed before the incident?

”Did anything change before the incident?” is the first question in almost every post-incident discussion. Azure Change Analysis answers this by tracking configuration changes across all Azure resources, code deployments, and application settings — all without additional configuration.

# View changes in a resource group over the last 24 hours
az change-analysis list \
  --resource-group myRG \
  --start-time "2026-03-19T00:00:00Z" \
  --end-time "2026-03-19T23:59:59Z" \
  --output table

# View changes for a specific resource
az change-analysis list \
  --resource-id /subscriptions/<sub-id>/resourceGroups/myRG/providers/Microsoft.Web/sites/myapp \
  --start-time "2026-03-19T12:00:00Z" \
  --end-time "2026-03-19T16:00:00Z" \
  --output json
Note

Change Analysis detects changes to resource configuration (app settings, connection strings, SKU changes), ARM resource properties, and code deployments. It does not detect changes to application business logic that do not modify Azure resource properties.

Post-incident review: preventing recurrence

A well-documented post-incident review (PIR) is the compounding interest of reliability work. Each PIR should answer:

  • Impact — How many users were affected? Which features were unavailable? What was the business impact?
  • Detection — How was the incident detected? By a monitor alert or by a user report? If by user report, what monitoring gap needs to be closed?
  • Timeline — When did the incident start? When was it detected? When was the on-call notified? When was service restored? (KQL queries provide all of this.)
  • Root cause — What was the technical cause? Was it a code change, a configuration change, an infrastructure event, or external dependency failure?
  • Corrective actions — What specific changes will be made to prevent this from happening again? Who owns each action? By when?
// PIR support: build the user impact timeline
AppRequests
| where TimeGenerated between (datetime(2026-03-19 14:15:00) .. datetime(2026-03-19 15:05:00))
| summarize
    RequestCount = count(),
    ErrorCount = countif(Success == false),
    AffectedUsers = dcountif(UserId, Success == false)
    by bin(TimeGenerated, 5m)
| extend ErrorRate = round(100.0 * ErrorCount / RequestCount, 1)
| project TimeGenerated, RequestCount, ErrorCount, ErrorRate, AffectedUsers

Tuning alerts based on incident history

After each incident, review your alert configuration:

  • If the alert fired too late — Reduce the evaluation window or lower the threshold. Consider a precursor metric that changes earlier in the failure sequence.
  • If the alert fired on a false positive — Increase the threshold, increase the number of required evaluation periods, or switch to a dynamic threshold.
  • If there was no alert at all — This is a monitoring gap. Write a new alert rule that would have caught this incident, and add it before the next sprint ends.
  • If the alert was too noisy — Use alert processing rules to suppress known patterns, or refine the KQL query to exclude benign conditions.

Common mistakes

  1. No runbook for common alert types. An alert that fires to a blank Slack message gives the on-call engineer nothing to work with. Every alert rule should have an associated runbook (even just a wiki link) that describes: what this alert means, the most common causes, and the first 3 investigation steps. Link the runbook in the alert description field.
  2. Treating every alert as equally urgent. Severity 1 alerts at 3 AM that turn out to be informational noise train on-call engineers to ignore pages. Use the 5-level severity scale correctly, and reserve out-of-hours pages for true user-impacting severity 0 and 1 alerts.
  3. Not capturing the incident timeline during the incident. During an incident, engineers are under pressure and save their query history in browser tabs that they close afterward. Establish a practice of copying key query results to the incident ticket or a shared document during the incident, not just afterward from memory.
  4. Skipping the post-incident review for “small” incidents. Small incidents often have the same root cause as large ones, just caught earlier. Skipping the PIR for minor incidents means you miss the opportunity to catch systematic issues before they cause a major outage.

Related topics to read next

These pages support the full incident response lifecycle.

Frequently asked questions

What is MTTR?

MTTR stands for Mean Time to Resolution — the average time from when an incident is detected to when the system is fully restored. Reducing MTTR is the primary goal of incident response tooling and processes.

What is an Azure Monitor runbook?

In Azure Monitor context, a runbook is an Azure Automation runbook that can be triggered automatically by an alert rule via an action group. It can execute remediation actions like restarting a service, scaling out resources, or clearing a queue.

How do I capture a snapshot of system state during an incident?

Save the specific time range of the incident in your Log Analytics queries and export the results. Azure Monitor workbooks can also be "snapshotted" by sharing the link with the current parameters, preserving the view state for post-incident review.

Last verified: 19 March 2026 Cloud services change frequently. Verify details against official documentation before making infrastructure decisions.