Monitoring AKS with Azure Monitor and Container Insights

Enabling Container Insights on AKS

Container Insights is an add-on to your AKS cluster. It deploys a monitoring agent DaemonSet that collects metrics and log data from every node.

# Enable Container Insights on a new AKS cluster
az aks create \
  --resource-group myRG \
  --name myAKSCluster \
  --node-count 3 \
  --node-vm-size Standard_D4s_v3 \
  --enable-addons monitoring \
  --workspace-resource-id /subscriptions/<sub-id>/resourceGroups/myRG/providers/Microsoft.OperationalInsights/workspaces/myWorkspace \
  --generate-ssh-keys

# Enable Container Insights on an existing AKS cluster
az aks enable-addons \
  --resource-group myRG \
  --name myAKSCluster \
  --addons monitoring \
  --workspace-resource-id /subscriptions/<sub-id>/resourceGroups/myRG/providers/Microsoft.OperationalInsights/workspaces/myWorkspace

# Verify the monitoring add-on is enabled
az aks show \
  --resource-group myRG \
  --name myAKSCluster \
  --query addonProfiles.omsagent.enabled \
  --output tsv

What Container Insights collects

Container Insights populates a set of dedicated tables in your Log Analytics workspace:

Essential KQL queries for AKS monitoring

These queries cover the most critical Kubernetes monitoring scenarios.

// Node CPU and memory utilization in the last hour
Perf
| where TimeGenerated > ago(1h)
| where ObjectName == "K8SNode"
| where CounterName in ("cpuUsageNanoCores", "memoryRssBytes")
| summarize AvgValue = avg(CounterValue) by Computer, CounterName, bin(TimeGenerated, 5m)
| render timechart

// Pods with high restart counts (crash looping)
KubePodInventory
| where TimeGenerated > ago(1h)
| where RestartCount > 2
| summarize MaxRestarts = max(RestartCount) by PodName = Name, Namespace, ContainerName
| order by MaxRestarts desc
| limit 20

// Container CPU usage as percentage of limit
Perf
| where TimeGenerated > ago(30m)
| where ObjectName == "K8SContainer"
| where CounterName == "cpuLimitNanoCores" or CounterName == "cpuUsageNanoCores"
| summarize AvgValue = avg(CounterValue) by CounterName, InstanceName
| evaluate pivot(CounterName, sum(AvgValue))
| extend CPUPercent = round(100.0 * cpuUsageNanoCores / cpuLimitNanoCores, 1)
| where CPUPercent > 80
| order by CPUPercent desc

// Kubernetes events in the last hour (errors and warnings)
KubeEvents
| where TimeGenerated > ago(1h)
| where Type in ("Warning", "Error")
| project TimeGenerated, Namespace, Name, Reason, Message
| order by TimeGenerated desc
| limit 50

// Container log errors in the last 15 minutes
ContainerLog
| where TimeGenerated > ago(15m)
| where LogEntry has_any ("ERROR", "FATAL", "Exception", "panic")
| project TimeGenerated, ContainerName, PodName = split(ContainerID, "/")[1], LogEntry
| limit 100

Recommended alerts for AKS clusters

Microsoft provides a curated set of recommended alert rules for AKS that you can enable with one click in the portal, or deploy via CLI:

# Enable recommended alert rules for an AKS cluster
# (Available through Azure Monitor → Alerts → Alert rules → + Create → Recommended alerts)
# These include:
# - Node CPU utilization > 80%
# - Node memory utilization > 80%
# - Pods in failed state
# - Pods not ready
# - Container restarts > 3 in last 5 minutes
# - OOM (out of memory) kills

# Create a custom alert for pod restarts using log search
az monitor scheduled-query create \
  --name "PodCrashLoop" \
  --resource-group myRG \
  --scopes /subscriptions/<sub-id>/resourceGroups/myRG/providers/Microsoft.OperationalInsights/workspaces/myWorkspace \
  --condition-query "KubePodInventory | where TimeGenerated > ago(5m) | where RestartCount > 5 | summarize MaxRestarts = max(RestartCount) by Name, Namespace | where MaxRestarts > 5" \
  --condition-column "MaxRestarts" \
  --condition-operator "GreaterThan" \
  --condition-threshold 0 \
  --condition-time-aggregation "Count" \
  --evaluation-frequency PT5M \
  --window-duration PT5M \
  --severity 2 \
  --action-groups /subscriptions/<sub-id>/resourceGroups/myRG/providers/microsoft.insights/actionGroups/OnCallTeam \
  --description "Pod restart count exceeds 5 in last 5 minutes — likely crash looping"

Live container log streaming

Container Insights collects container logs into the ContainerLog table, but for real-time debugging you often want to stream logs directly without waiting for Log Analytics ingestion delay. Use kubectl for immediate log access:

# Stream logs from a specific pod
kubectl logs -f <pod-name> -n <namespace>

# Stream logs from all pods in a deployment
kubectl logs -f -l app=myapp -n production --prefix=true

# Get recent logs from a crashed container (previous run)
kubectl logs <pod-name> -n <namespace> --previous

# Stream logs from all containers in a pod
kubectl logs <pod-name> -n <namespace> --all-containers=true -f

# Check events for a pod that is not starting
kubectl describe pod <pod-name> -n <namespace>
# Look at the Events section at the bottom for scheduling, image pull, and probe failures

Azure Monitor managed service for Prometheus

Container Insights covers Kubernetes-native metrics well, but many workloads expose custom application metrics via the Prometheus format. Azure Monitor Managed Service for Prometheus (preview/GA depending on your region) collects Prometheus metrics from your AKS cluster and stores them in an Azure Monitor workspace — separate from Log Analytics.

# Enable Azure Monitor managed Prometheus on an AKS cluster
az aks update \
  --resource-group myRG \
  --name myAKSCluster \
  --enable-azure-monitor-metrics \
  --azure-monitor-workspace-resource-id /subscriptions/<sub-id>/resourceGroups/myRG/providers/microsoft.monitor/accounts/myMonitorWorkspace \
  --grafana-resource-id /subscriptions/<sub-id>/resourceGroups/myRG/providers/Microsoft.Dashboard/grafana/myGrafana

# Prometheus metrics are then queryable in Grafana (linked above)
# or via the Azure Monitor workspace's PromQL endpoint