GCP Structured Logging Explained: JSON Logs vs Plain Text

Simple explanation

When your app writes {“severity”:“ERROR”,“user_id”:“u-123”} to stdout, Cloud Logging stores user_id as its own indexed field. You can then query jsonPayload.user_id=“u-123” and get back only that user’s entries, not every log that happens to contain that string somewhere in the body.

This is the core difference between jsonPayload and textPayload. With jsonPayload, every field is queryable individually. With textPayload, you have one big string and substring matching only.

Why structured logging matters in GCP

Structured logs are not just a style preference. They unlock capabilities in Cloud Monitoring that plain text cannot provide:

• Precise filtering. Find every request from a specific user, to a specific endpoint, with a specific status code. In Logs Explorer, a query like jsonPayload.endpoint=“/api/checkout” AND jsonPayload.status_code=500 returns exactly the entries you care about.
• Numeric comparisons. Filter by jsonPayload.retry_count>2 or jsonPayload.duration_ms>1000. Impossible with text logs unless you parse the string yourself.
• Log-based metrics. Log-based metrics extract values from jsonPayload fields and turn them into Cloud Monitoring metrics. For example, a latency histogram built directly from jsonPayload.duration_ms.
• Alerting. Build alert policies that fire when error rates or latency values cross thresholds, derived from your structured log fields.
• Trace correlation. Include the logging.googleapis.com/trace field and your log entries link directly to spans in Cloud Trace. During an incident you can jump from a slow trace to every log entry emitted during that specific request.

How it works in Cloud Logging

The path from your application to a queryable log entry is straightforward:

1. Your app writes a line to stdout (or stderr).
2. The GCP logging agent (or Cloud Run’s built-in collector) captures that line.
3. If the line is a plain string, Cloud Logging stores it in textPayload.
4. If the line is a valid JSON object, Cloud Logging parses it and stores it in jsonPayload.
5. Any field named severity becomes the entry’s log level. Any field named message appears as the summary line in the console.
6. Every other field is indexed under jsonPayload and available for filtering, metrics, alerts, and trace linking.

Structured logging vs plain text logs

Key fields to include in your log entries

Cloud Logging recognizes certain field names and treats them specially. These are worth including in every log entry:

• severity: the log level. Valid values: DEBUG, INFO, NOTICE, WARNING, ERROR, CRITICAL, ALERT, EMERGENCY.
• message: the human-readable summary shown in Logs Explorer. Keep it short and descriptive.
• labels: a map of string key-value pairs for metadata you want to filter on but not extract as a metric.
• httpRequest: a structured object for HTTP request details. Cloud Logging renders it with special formatting and makes fields like httpRequest.status and httpRequest.latency directly queryable.
• sourceLocation: the file, line number, and function that emitted the log. Useful for linking entries back to specific code.
• logging.googleapis.com/trace: links the entry to a trace in Cloud Trace. Value format: projects/PROJECT_ID/traces/TRACE_ID.
• logging.googleapis.com/spanId: links to a specific span within the trace.

Beyond these special fields, include application-specific fields you will actually use when debugging or building metrics:

• user_id: find all logs for a specific user
• endpoint: filter by API route
• status_code: filter by HTTP response status
• duration_ms: measure latency and build histograms
• retry_count: surface flaky infrastructure issues
• order_id or session_id: trace a single transaction across services

Example Logs Explorer queries

Once your logs are structured, these Logs Explorer queries work immediately with no additional setup:

# Find all logs for a specific user
jsonPayload.user_id="u-8821"

# Filter by API endpoint
jsonPayload.endpoint="/api/checkout"

# Errors only
severity="ERROR"

# HTTP 500 responses
jsonPayload.status_code=500

# Requests with more than 2 retries
jsonPayload.retry_count>2

# Scope to a specific GKE namespace
resource.type="k8s_container"
resource.labels.namespace_name="production"

# Combine filters: checkout errors for one user
jsonPayload.user_id="u-8821"
jsonPayload.endpoint="/api/checkout"
severity="ERROR"

# Find logs linked to a specific trace
logging.googleapis.com/trace="projects/my-project/traces/abc123"

All of these work only because the fields are individually indexed. With textPayload, none of these queries are possible.

Structured logging on Cloud Run

Cloud Run captures everything your container writes to stdout and forwards it to Cloud Logging. You do not need a logging agent, an SDK, or any configuration. If what you write is valid JSON, it becomes jsonPayload. See Monitoring Cloud Run for how structured logs feed into Cloud Run observability dashboards.

Here is a minimal Python helper that produces structured log entries:

import json
import sys

def log(severity, message, **kwargs):
    entry = {
        "severity": severity,
        "message": message,
        **kwargs
    }
    print(json.dumps(entry), file=sys.stdout, flush=True)

# Every keyword argument becomes a queryable jsonPayload field
log("INFO", "Request received", user_id="u-123", endpoint="/api/orders")
log("ERROR", "Database connection failed", error="connection timeout", retry_count=3)
log("WARNING", "Cache miss", cache_key="product:42", duration_ms=12)

To correlate logs with traces, include the trace field in your entries:

def log_with_trace(severity, message, project_id, trace_id, span_id=None, **kwargs):
    entry = {
        "severity": severity,
        "message": message,
        "logging.googleapis.com/trace": f"projects/{project_id}/traces/{trace_id}",
        **kwargs
    }
    if span_id:
        entry["logging.googleapis.com/spanId"] = span_id
    print(json.dumps(entry), file=sys.stdout, flush=True)

If you use OpenTelemetry for tracing, the trace ID and span ID are available from the current span context. Most OpenTelemetry logging integrations inject these fields automatically.

Structured logging on GKE

GKE works the same way. Any container writing valid JSON to stdout has its logs parsed into jsonPayload automatically. The Cloud Logging agent runs as a DaemonSet on each node and collects stdout and stderr from every container via the Kubernetes logging infrastructure.

In addition to your own JSON fields, every GKE log entry includes these resource labels automatically:

• resource.labels.cluster_name: the GKE cluster
• resource.labels.namespace_name: the Kubernetes namespace
• resource.labels.pod_name: the specific pod
• resource.labels.container_name: the container within the pod

You can combine these with your structured fields to scope queries tightly. To find all errors in the checkout container in the production namespace:

resource.type="k8s_container"
resource.labels.namespace_name="production"
resource.labels.container_name="checkout"
severity="ERROR"

See Monitoring GKE for how structured logs integrate with GKE dashboards and alerting, and Logging in Kubernetes for how the Kubernetes logging pipeline works end-to-end.

When to use structured logging

Structured logging pays off whenever you need to debug, monitor, or alert on production systems. Use it for:

• Production APIs. Any service handling real user traffic. When something breaks, you want to filter by user, endpoint, and status code right away, not grep through text.
• Checkout and payment flows. High-stakes paths where you need to trace exactly what happened for a specific order ID or session.
• Error-rate monitoring. Use log-based metrics to count status_code=500 entries and alert when the rate exceeds a threshold.
• Tracing a single request end-to-end. Include a request_id or trace ID in every log entry for a request. You can then find every log entry across every service that handled it with one query.
• Multi-service incident response. During an incident, structured logs let you correlate failures across services without parsing text from multiple systems.

Structured logging vs Cloud Audit Logs

• Structured application logs are generated by your code. You decide what to log, when, and what fields to include. They appear in Cloud Logging under whatever resource your service runs on.
• Cloud Audit Logs are generated by GCP itself. They record who did what to which GCP resource: who created a Cloud Storage bucket, who changed a firewall rule, who read a secret from Secret Manager. You do not write these; GCP writes them automatically.

Use structured application logs for operational debugging and monitoring. Use Audit Logs for security, compliance, and access auditing. For a broader overview of every log type available in GCP, see Log Types in GCP.