Logging in Kubernetes on GKE: kubectl logs, Cloud Logging, and Structured Logs

How GKE logging works in plain terms

Your application writes messages to stdout or stderr, the same output streams that any process has. GKE picks those messages up automatically, labels them with metadata about the pod and cluster, and stores them in Cloud Logging where you can search them at any time.

There are two main ways to read those logs:

• kubectl logs — go directly to the running container and ask what it said recently. Fast, but limited to what is still available on that node.

• Cloud Logging — search the full historical archive. Works even after the pod is gone, and lets you search across all pods at once.

How the logging pipeline works

The logging pipeline in GKE operates at the node level in three stages:

1. Application writes to stdout or stderr. The container runtime (containerd on GKE) captures these streams and writes them to log files on the node’s filesystem under /var/log/pods/. The container itself does not need to know it is running in Kubernetes.

2. Node logging agent forwards logs. GKE deploys Fluent Bit as a DaemonSet (one pod per node). Fluent Bit tails the log files written by the container runtime, enriches each entry with Kubernetes metadata (pod name, namespace, container name, cluster name, node name), and forwards the entries to Cloud Logging.

3. Cloud Logging stores and indexes logs. Cloud Logging receives the enriched entries and stores them in the _Default log bucket for the project. Entries are indexed and queryable immediately via Logs Explorer, the gcloud CLI, or the Cloud Logging API.

For metrics-based observability alongside log data, see Monitoring GKE Clusters.

kubectl logs vs Cloud Logging

These two tools serve different purposes. Knowing when to use each saves time during debugging.

Reading logs with kubectl

For immediate log inspection during live debugging, use the kubectl logs subcommand:

# Stream live logs from a pod (Ctrl+C to stop)
kubectl logs -f my-pod -n production

# Read the last 200 lines only
kubectl logs my-pod --tail=200 -n production

# Read logs from a specific container in a multi-container pod
kubectl logs my-pod -c sidecar -n production

# Read logs from all pods matching a label selector
kubectl logs -l app=my-app -n production

# Read logs from the PREVIOUS (crashed) container instance
kubectl logs my-pod --previous -n production

For more on pod lifecycle and restarts, see Kubernetes Pods Explained.

Querying logs in Cloud Logging

Logs Explorer is the browser-based search tool in the Google Cloud console. It uses the Logging Query Language (LQL): a filter syntax that combines resource labels, field values, and text searches.

Basic container log filter:

resource.type="k8s_container"
resource.labels.namespace_name="production"
resource.labels.pod_name=~"my-app-.*"

The =~ operator matches using regular expressions, so my-app-.* matches every pod in a deployment without needing to know the exact generated suffix.

Filter by severity:

resource.type="k8s_container"
resource.labels.namespace_name="production"
severity>=ERROR

Filter by a specific container name (useful in multi-container pods):

resource.type="k8s_container"
resource.labels.container_name="my-app"
resource.labels.namespace_name="production"

Reading logs with gcloud:

# Read recent logs for a specific pod
gcloud logging read \
  'resource.type="k8s_container" AND resource.labels.pod_name="my-pod-7d9f8c-xkzpl"' \
  --limit=50 \
  --format="table(timestamp,severity,textPayload)"

# Read ERROR and above from a namespace in the last hour
gcloud logging read \
  'resource.type="k8s_container" AND resource.labels.namespace_name="production" AND severity>=ERROR' \
  --freshness=1h \
  --project=my-project

Log entry structure in Cloud Logging

Every log entry from a GKE container arrives with a consistent set of metadata labels. The monitored resource type is k8s_container. Each entry carries:

For plain text log lines, Cloud Logging stores the content in textPayload. For JSON log lines, it parses the object and stores it in jsonPayload, making every field individually searchable.

For example, if your app logs {“severity”:“ERROR”,“message”:“timeout”,“requestId”:“abc123”}, you can filter directly on jsonPayload.requestId=“abc123” in Logs Explorer. With textPayload, you can only do a full-text search across the entire raw line.

Structured logging

Plain text logs are hard to query at scale. Searching for the word “error” returns every line containing it: function names, variable names, error messages, and debug statements all mixed together. Structured logging writes log entries as JSON objects to stdout, so each piece of information is a named, individually queryable field.

When Cloud Logging receives a JSON-formatted log line from a container, it automatically parses the object and stores it in jsonPayload. For a deeper look at patterns and field conventions, see Structured Logging.

Example structured log entry (what your app writes to stdout):

{
  "severity": "ERROR",
  "message": "Database connection failed",
  "requestId": "req-abc123",
  "userId": "user-456",
  "durationMs": 3042
}

Minimal Python implementation:

import json
import logging
import sys

class JsonFormatter(logging.Formatter):
    def format(self, record):
        log_entry = {
            "severity": record.levelname,
            "message": record.getMessage(),
            "logger": record.name,
        }
        return json.dumps(log_entry)

handler = logging.StreamHandler(sys.stdout)
handler.setFormatter(JsonFormatter())
logging.getLogger().addHandler(handler)
logging.getLogger().setLevel(logging.INFO)

Cloud Logging recognises these special fields in JSON log entries:

Including a logging.googleapis.com/trace field with the trace ID from your incoming request automatically correlates log entries with the corresponding trace in Cloud Trace, letting you jump from a log line directly to the full distributed trace for that request.

Logging in multi-container pods

Each container in a pod produces its own log stream, stored separately in Cloud Logging under the container_name label. To query only the application container’s logs and exclude a sidecar:

resource.type="k8s_container"
resource.labels.pod_name=~"my-app-.*"
resource.labels.container_name="my-app"

When reading via kubectl, specify the container with -c:

kubectl logs my-pod -c my-app -n production
kubectl logs my-pod -c envoy-proxy -n production

Init containers (which run to completion before the main containers start) also produce logs. Their output is available in Cloud Logging for the full retention period even after the init container has finished.

Log retention, costs, and log-based metrics

The _Default log bucket retains logs for 30 days by default. You can increase this up to 3,650 days in the bucket settings, or export logs long-term using a log sink:

gcloud logging sinks create gke-logs-archive \
  storage.googleapis.com/my-log-archive-bucket \
  --log-filter='resource.type="k8s_container"'

Sinks export matching entries to Cloud Storage, BigQuery, or Pub/Sub in near-real time for cost-effective long-term archival, without changing retention inside Cloud Logging itself.

You can also convert any log query into a Cloud Monitoring metric. This lets you alert on application-level events that infrastructure metrics do not capture — for example, alerting when the ERROR rate in the production namespace exceeds 10 per minute.

Application logging best practices for GKE

• Use structured JSON logging. Even a minimal structure (severity, message, and a request ID) dramatically improves your ability to filter and query logs at scale. See Structured Logging for patterns and examples.

• Include a request or trace ID in every log entry. Without a shared ID, you cannot find every log line produced during a specific request. Without one, you are reduced to guessing based on timestamps.

• Use appropriate severity levels. In production, use INFO for normal operations, WARNING for recoverable issues, and ERROR for failures that need attention. Avoid excessive DEBUG logging at production scale — it drives up ingestion costs and makes real errors harder to find.

• Control workload access to Cloud Logging. Use Workload Identity for GKE to control which workloads can write to Cloud Logging, and review Securing GKE Clusters for the broader cluster access model.