Logging in Kubernetes and AKS
Kubernetes defines a simple logging contract: write to stdout and stderr, and the platform handles the rest. This page explains what “the rest” actually means, how to read logs with kubectl, how Container Insights collects and retains them in Log Analytics, and what structured logging looks like in practice.
The Kubernetes logging contract
Kubernetes does not prescribe a logging framework. It requires exactly one thing: applications write their log output to stdout and stderr. The container runtime (containerd on AKS) captures these streams and writes them to log files on the node at /var/log/containers/.
This is a deliberate design decision. The application stays simple — no log agents, no file handles, no configuration for log paths. The platform decides how to collect and route logs. On AKS, that means the omsagent DaemonSet (part of Container Insights) tails these files and ships them to Log Analytics.
Log files on the node are subject to a rotation policy. By default, each container log file is rotated when it reaches 10 MB, with a maximum of 5 rotated files kept. Logs older than the rotation window — or from deleted pods — are gone from the node. This is why shipping to a central store like Log Analytics is essential for any workload that needs more than a few minutes of log history.
Do not write logs to files inside the container filesystem. Files written inside a container are lost when the container restarts or is replaced. Some teams mix file-based logging with stdout logging, creating a situation where some logs ship to Log Analytics and others disappear silently. Write everything to stdout/stderr.
Reading logs with kubectl
kubectl reads logs directly from the node’s log files via the Kubernetes API server. It does not query Log Analytics — it reads the live, local copy of the log.
# Basic log retrieval for a pod
kubectl logs myapp-7d4b8c9f6-xkj2p -n production
# Follow live output (streams new lines as they arrive)
kubectl logs -f myapp-7d4b8c9f6-xkj2p -n production
# Last 200 lines only
kubectl logs myapp-7d4b8c9f6-xkj2p -n production --tail=200
# Logs since a specific time
kubectl logs myapp-7d4b8c9f6-xkj2p -n production --since=1h
# Logs from a crashed/restarted container (previous instance)
kubectl logs myapp-7d4b8c9f6-xkj2p -n production --previous
# Logs from all pods matching a label (useful during rolling updates)
kubectl logs -l app=myapp -n production --tail=50
# If a pod has multiple containers, specify which container
kubectl logs myapp-7d4b8c9f6-xkj2p \
-c sidecar \
-n productionThe -l selector flag is particularly useful during deployments — it aggregates logs across all replicas of a Deployment without needing to know the individual pod names.
kubectl logs has limits. It only reaches logs that still exist on the node. If the pod was deleted minutes ago, or if log rotation has already removed older entries, kubectl returns nothing or truncated output. For any investigation spanning more than a few minutes, query Log Analytics instead.
Multi-container pod log selection
Pods can have multiple containers: the main application container plus init containers, sidecar containers, and the injected service mesh proxy if you are using something like Istio. Each container has its own separate log stream.
# List the containers in a pod
kubectl get pod myapp-7d4b8c9f6-xkj2p \
-n production \
-o jsonpath='{.spec.containers[*].name}'
# Get logs from a specific container by name
kubectl logs myapp-7d4b8c9f6-xkj2p \
-n production \
-c myapp
# Get logs from an init container
kubectl logs myapp-7d4b8c9f6-xkj2p \
-n production \
-c init-db-migration
# Get logs from all containers in the pod
kubectl logs myapp-7d4b8c9f6-xkj2p \
-n production \
--all-containers=trueIf you do not specify -c and the pod has only one container, kubectl defaults to that container. If there are multiple containers and no -c flag, kubectl returns an error listing the available container names.
Querying logs in Log Analytics
Container Insights ships logs to the ContainerLogV2 table (or ContainerLog on older agent versions). ContainerLogV2 includes the pod name, namespace, and container name directly in each record, making queries straightforward.
Logs from a specific pod over the last hour:
ContainerLogV2
| where TimeGenerated > ago(1h)
| where PodName == "myapp-7d4b8c9f6-xkj2p"
| project TimeGenerated, LogMessage
| order by TimeGenerated ascError logs from all pods in a namespace:
ContainerLogV2
| where TimeGenerated > ago(1h)
| where Namespace == "production"
| where LogLevel == "Error" or LogMessage contains "ERROR"
| project TimeGenerated, PodName, ContainerName, LogMessage
| order by TimeGenerated descLogs from a deleted pod — this works even after the pod is gone:
ContainerLogV2
| where TimeGenerated between (datetime(2026-03-19 14:00) .. datetime(2026-03-19 15:00))
| where PodName startswith "myapp-"
| project TimeGenerated, PodName, LogMessage
| order by TimeGenerated ascParsing JSON log lines to extract a specific field:
ContainerLogV2
| where TimeGenerated > ago(1h)
| where Namespace == "production"
| extend LogJson = parse_json(LogMessage)
| where LogJson.level == "error"
| project TimeGenerated, PodName, RequestId = tostring(LogJson.requestId), Message = tostring(LogJson.message)The parse_json() function is what makes structured JSON logging genuinely useful at scale. Without it, you are reduced to string matching against the full log line. With it, you can filter on any field in your log schema.
Structured JSON logging best practices
Plain text logs like 2026-03-19 14:23:11 ERROR Failed to connect to database: timeout are readable by humans but hard to query at scale. JSON logs with consistent field names are queryable by any field, enabling powerful filtering in Log Analytics.
A well-structured JSON log line looks like:
{
"timestamp": "2026-03-19T14:23:11.452Z",
"level": "error",
"message": "Failed to connect to database",
"error": "context deadline exceeded",
"requestId": "req_a4f2b91c",
"userId": "usr_7e3a1d",
"durationMs": 5012,
"service": "api-server",
"version": "2.1.0"
}Include a consistent set of fields in every log line:
- timestamp — ISO 8601 format in UTC. Do not rely on Log Analytics ingestion time; network delays mean ingestion time can differ from event time by seconds to minutes.
- level — info, warn, error, debug. Use a consistent enum so KQL filters on
LogJson.level == “error”work reliably. - message — a human-readable description of the event.
- requestId / traceId — a unique identifier that links all log lines for a single request or transaction. Essential for tracing a request across multiple services.
- service / version — the service name and version, even though Container Insights captures the pod name separately. This helps when logs from multiple services are queried together.
Avoid logging sensitive data (passwords, tokens, PII) to stdout even in debug builds. Container Insights ships all stdout/stderr to Log Analytics, where it may be retained for months and accessible to anyone with Log Analytics read access. Sanitize log output before writing it.
kubectl logs vs Log Analytics for production debugging
These two methods serve different purposes and are not interchangeable.
| Situation | Use kubectl logs | Use Log Analytics |
|---|---|---|
| Pod is currently running and crashing | Yes — —previous flag | Also works |
| Pod was deleted 30 minutes ago | No — logs gone from node | Yes — shipped to Log Analytics |
| Need last 5 lines to see immediate error | Fast — kubectl logs —tail=5 | Possible but slower to write |
| Search across 50 pods for a specific error | Impractical — one pod at a time | Single KQL query |
| Correlate logs from 6 hours ago with an alert | No | Yes — retained up to 2 years |
| Filter logs by JSON field value | Grep manually or use jq | parse_json() in KQL |
In practice, start with kubectl for quick live debugging of running pods, then switch to Log Analytics for any investigation that involves historical data, multiple pods, or structured field filtering.
Common mistakes
- Writing logs to files inside the container instead of stdout. Container filesystems are ephemeral. Files written inside a running container disappear when the container restarts. Log files written this way are never shipped to Log Analytics, never accessible via kubectl logs, and lost permanently on pod restart. Always write to stdout/stderr.
- Using plain text logging in production. Unstructured log lines cannot be filtered by field in KQL without fragile regex parsing. When you are investigating an incident at 2 AM, searching for all error logs with a specific requestId across 30 pod replicas requires structured fields. Switching log format after an incident is much harder than doing it from the start.
- Logging at DEBUG level in production. Debug logs often include sensitive data and generate very high volumes. At scale, a few hundred pods all logging at DEBUG can ingest gigabytes into Log Analytics per day, generating significant cost. Use INFO as the default production log level, with a mechanism (environment variable or ConfigMap) to enable DEBUG temporarily on a specific pod without redeploying everything.
Summary
- Kubernetes logging requires applications to write to stdout/stderr; the container runtime captures these streams to node-local log files that Container Insights ships to Log Analytics.
- kubectl logs reads node-local files and is useful for live debugging of running pods; Log Analytics retains historical logs and supports querying across all pods simultaneously.
- Use —previous with kubectl logs to read the output of a crashed container before it was restarted.
- Structured JSON logging with consistent fields like level, requestId, and timestamp makes KQL queries on ContainerLogV2 practical and powerful.
Frequently asked questions
What happens to logs when a pod is deleted?
When a pod is deleted from Kubernetes, its local log files on the node are also deleted. If you only use kubectl logs, those logs are gone permanently. Container Insights ships logs to Log Analytics in near real-time, so logs from deleted pods remain queryable in Log Analytics even after the pod no longer exists.
How do I view logs from a crashed container?
Use kubectl logs <pod-name> --previous. This returns the logs from the most recent terminated container instance, which is useful when a container is in CrashLoopBackOff and the current instance has already been replaced. Without --previous, you only see logs from the currently running instance.
Should I write logs as plain text or JSON?
JSON is strongly preferred for any production workload. Plain text logs are hard to query and filter in Log Analytics. JSON logs allow KQL queries to extract specific fields using the parse_json() function or the dynamic column type. This makes it practical to query by error code, request ID, or user ID across millions of log entries.