AWS Structured Logging: JSON Logs for Lambda & CloudWatch

Structured logging, simply explained

A log is a record of something that happened in your application. Most developers start by writing log lines like sentences: a timestamp, a severity level, and a message in plain English. That works on your laptop.

Structured logging is the same idea, but the record is a JSON object instead of a sentence. Every piece of data has a name. Every value is in a predictable place. The log is still readable by humans but is also instantly queryable by machines.

Here is the same event written both ways:

Plain-text log:

2026-05-03 14:32:01 ERROR Failed to process order 789 for user 456: payment declined (code: CARD_DECLINED)

JSON log:

{
  "timestamp": "2026-05-03T14:32:01Z",
  "level": "ERROR",
  "message": "Order processing failed",
  "orderId": "789",
  "userId": "456",
  "errorCode": "CARD_DECLINED",
  "durationMs": 243
}

Both records contain the same information. But in the JSON version, every piece of data is in a named field. CloudWatch Logs Insights discovers those fields automatically. Instead of writing filter @message like /CARD_DECLINED/, you write filter errorCode = “CARD_DECLINED”. If the message wording ever changes, the query still works because the field name stays the same.

Why plain-text logs break in production

Plain-text logs have four failure modes at scale:

• Searchability. Finding all errors of a specific type means writing a regex. Regex queries are brittle; one wording change in a log message silently breaks the query.
• Grouping and aggregation. To count errors by error code, you first have to extract the error code from a string. That requires a parse step before every query.
• Cross-service correlation. To trace one request across Lambda, SQS, and EventBridge, all services need to include a shared ID in a predictable position. In plain text, that requires strict formatting discipline enforced across every team.
• Query cost. CloudWatch Logs Insights charges per GB scanned. A regex or keyword search scans every character of every log line. A field-level filter on JSON only examines the fields you reference.

These problems are manageable with one service and light traffic. They become painful when you have multiple Lambda functions, an API Gateway, SQS queues, and EventBridge rules all writing logs at the same time.

How structured logging works in AWS

The flow from application to query is straightforward:

1. Your application emits one JSON object per log event to stdout or via the AWS SDK.
2. Logs land in a CloudWatch Logs log group (for example, /aws/lambda/order-service).
3. CloudWatch Logs Insights scans incoming JSON events and discovers fields automatically. You can see discovered fields in the query console without writing any extra configuration.
4. Teams write queries using those field names to filter errors, aggregate latency, and correlate requests across services.
5. Metric filters watch for specific field values, like level = “ERROR”, and publish counts to CloudWatch Metrics.
6. Alarms fire when those metric counts exceed thresholds.

CloudWatch Logs Insights also supports dot notation for nested JSON fields. If your log contains {“request”: {“method”: “POST”, “path”: “/orders”}}, you can query request.method and request.path directly. Keep nesting shallow; deeply nested structures are harder to query and can make field discovery less reliable.

When to use structured logging

Structured logging pays off whenever you need to search, group, or alert on log data. Concrete use cases:

• Debugging failed requests. Filter by requestId or correlationId to see every event tied to one failed call. This is the foundation of debugging production systems.
• Cross-service tracing. Pass a correlation ID from API Gateway through Lambda, SQS, and EventBridge. Query all services by that ID in a single Logs Insights query.
• Error breakdowns. Group errors by errorCode to see which failure modes dominate.
• Latency analysis. Aggregate durationMs by endpoint or function version to detect regressions.
• Log-based metrics and alarms. Create metric filters on fields like errorCode = “PAYMENT_FAILED”, then trigger alarms when they spike.
• Incident response. During an incident, structured logs let you filter to the relevant service, time window, and error category in seconds. See the guide on incident response with monitoring.

For a truly one-off script that runs once, writes a few lines, and never needs to be queried again, plain text is fine. But if the script runs on a schedule, gets debugged by others, or might scale, structuring from the start is easier than retrofitting later.

Structured logging vs unstructured logs

Recommended JSON log schema

A shared schema across all your services is more valuable than a perfect schema for one service. Agree on field names before you start. You can add fields later, but renaming userId to user_id mid-project means updating every Logs Insights query and every metric filter that references it.

Implementation options in AWS

You have three realistic paths depending on how much control and automation you need.

Option 1: Native Lambda JSON logging

Lambda supports a built-in JSON log format you can enable in the function’s logging configuration (in the console or via the CLI with —logging-config LogFormat=JSON). When enabled, Lambda emits platform events (START, END, REPORT) as structured JSON and wraps each application log line in a JSON envelope with timestamp, requestId, level, and message fields.

In Python, the message field contains whatever string your print() or logger.info() call emits. To get queryable custom fields, print a JSON string explicitly:

import json

print(json.dumps({
    "message": "Order processing failed",
    "orderId": order_id,
    "errorCode": "PAYMENT_FAILED",
    "durationMs": duration_ms,
}))

This is the right choice for simple Lambda functions where you want structured output without adding a dependency. The tradeoff: you get no automatic Lambda context fields (function name, cold start, memory size) unless you add them manually to each event.

Option 2: AWS Lambda Powertools (recommended for Lambda)

For Lambda functions that need richer context, team-wide conventions, and automatic correlation ID propagation, Lambda Powertools is the recommended path. The Logger utility automatically adds Lambda execution context to every log event: function name, version, ARN, memory size, request ID, and a cold_start boolean. See the monitoring Lambda guide for more Lambda-specific observability patterns.

Option 3: Containers, EC2, and other services

The core idea is language-agnostic: emit one JSON object per log event to stdout. The CloudWatch agent or a container log driver (awslogs or Fluent Bit) picks it up and ships it to CloudWatch Logs, where Logs Insights discovers the fields automatically. Node.js, Go, Java, and Ruby all have JSON logging libraries that follow this pattern. The schema table above applies equally to all of them.

Python example with Lambda Powertools

pip install aws-lambda-powertools

from aws_lambda_powertools import Logger
from aws_lambda_powertools.utilities.typing import LambdaContext

logger = Logger(service="order-service")


@logger.inject_lambda_context(log_event=True)
def handler(event: dict, context: LambdaContext) -> dict:
    order_id = event["orderId"]
    user_id = event["userId"]

    logger.info("Processing order", extra={"orderId": order_id, "userId": user_id})

    try:
        result = process_payment(order_id)
        logger.info(
            "Payment succeeded",
            extra={"orderId": order_id, "amount": result["amount"]}
        )
        return {"statusCode": 200, "body": "ok"}
    except PaymentError as e:
        logger.error(
            "Payment failed",
            extra={"orderId": order_id, "errorCode": e.code, "errorMessage": str(e)}
        )
        return {"statusCode": 500, "body": "payment failed"}

Every event Powertools emits includes Lambda context automatically:

{
  "level": "INFO",
  "message": "Payment succeeded",
  "timestamp": "2026-05-03T14:32:01.456Z",
  "service": "order-service",
  "cold_start": false,
  "function_name": "process-orders",
  "function_memory_size": 256,
  "function_request_id": "abc-123-def-456",
  "orderId": "order-789",
  "amount": 49.99
}

The cold_start field is especially useful for Lambda performance monitoring. You can query cold start frequency directly in Logs Insights without any extra instrumentation.

CloudWatch Logs Insights queries that become easier

Once your logs are structured, these queries are straightforward. Each would require fragile regex or manual parsing with plain-text logs.

Error breakdown by error code:

fields @timestamp, errorCode, orderId
| filter level = "ERROR"
| stats count(*) as errorCount by errorCode
| sort errorCount desc

P95 and P99 latency over time:

fields @timestamp, durationMs
| filter ispresent(durationMs)
| stats pct(durationMs, 95) as p95, pct(durationMs, 99) as p99, avg(durationMs) as avg
  by bin(5m)

Full event history for one request:

fields @timestamp, level, message, service
| filter correlationId = "req-789-xyz"
| sort @timestamp asc

Cold start rate over time (Lambda Powertools):

fields @timestamp, cold_start
| filter ispresent(cold_start)
| stats sum(cold_start) as coldStarts, count(*) as totalInvocations
  by bin(1h)

Top users by error count:

fields @timestamp, userId, errorCode
| filter level = "ERROR"
| stats count(*) as errors by userId
| sort errors desc
| limit 20

All of these use exact field names rather than text patterns: faster to write, more reliable when message wording changes, and cheaper to run because Logs Insights only scans the fields you reference.

Correlation IDs: tracing requests across services

A correlation ID is a unique string you generate at the entry point of a request (at API Gateway, for example) and pass to every downstream service: Lambda, SQS, EventBridge, and any Lambda those services invoke. Every service logs it on every event. When something goes wrong, search for that one ID in Logs Insights and get the complete timeline of the request across all services.

Without a correlation ID, reassembling what happened means matching timestamps across services and hoping no other requests were in-flight at the same time.

import uuid
from aws_lambda_powertools import Logger

logger = Logger(service="order-service")


def handler(event: dict, context) -> dict:
    correlation_id = (
        event.get("headers", {}).get("X-Correlation-ID")
        or event.get("correlationId")
        or str(uuid.uuid4())
    )

    # All subsequent log calls in this invocation include correlationId
    logger.append_keys(correlationId=correlation_id)
    logger.info("Handler invoked")

    return {"correlationId": correlation_id, "status": "ok"}

Security and data-handling rules

Structured logs are expressive, which also makes it easy to accidentally log things you should not. A few rules worth enforcing from the start:

• Never log secrets. API keys, passwords, session tokens, and OAuth credentials should never appear in log events. Structured logging makes it tempting to log full event payloads; always check what is in the object before logging it.
• Avoid PII unless required. Email addresses, phone numbers, and full names should be omitted unless your use case explicitly requires them and you have the appropriate retention policies, access controls, and compliance posture in place.
• Sanitize before logging. If you log request payloads or event data, strip or hash sensitive fields before the log call, not after.
• Watch high-cardinality fields. Full URLs with query strings, raw user agents, or full IP addresses can increase query complexity. Be deliberate about what you include.
• Keep field names consistent. Inconsistent naming across services (userId vs user_id vs userID) breaks cross-service queries as reliably as missing fields do, just less visibly.

How to roll this out safely

Switching an existing service to structured logging is low-risk when done incrementally:

1. Define your shared field names as a team document before writing any code.
2. Instrument one low-traffic Lambda or service first.
3. Open CloudWatch Logs Insights, run a test query, and confirm your fields are discovered.
4. Add requestId and correlationId to every log event in that service.
5. Set a log retention policy on the log group. Ninety days is a reasonable default; indefinite retention is a common and expensive mistake.
6. Create a metric filter on level = “ERROR” and connect it to a CloudWatch Alarm.
7. Document your field naming conventions so the next service that gets instrumented follows the same schema.
8. Roll out to remaining services one at a time.