EKS Logging with Fluent Bit: How Kubernetes Logs Reach CloudWatch

Simple explanation

Here is the full journey of a log line, in plain terms:

1. Your application writes a message to stdout or stderr, the standard output streams of the process.
2. The kubelet on the node captures that output and writes it to a file on the node’s local filesystem, under /var/log/containers/.
3. When you run kubectl logs pod-name, Kubernetes reads that file from the node and returns it to you.
4. When the pod is deleted or the node terminates, that file is deleted too. The log is gone.
5. Fluent Bit runs on every node as a DaemonSet. It continuously reads from those log files and copies each line to an external system (CloudWatch, S3, OpenSearch) before anything can delete them.

That last step is the one that matters for production. Without it, you are debugging from memory.

How logging works in Kubernetes and EKS

Every container in Kubernetes is expected to write output to stdout and stderr. The kubelet on each EKS node captures this output and writes it to log files on the node’s local filesystem. Files follow a naming convention: <pod-name>_<namespace>_<container-name>-<container-id>.log.

The end-to-end flow looks like this:

• Container writes to stdout/stderr
• Kubelet writes to /var/log/containers/ on the node
• kubectl logs reads from that file while it exists
• Fluent Bit DaemonSet reads from the same files and forwards to CloudWatch, OpenSearch, S3, or Firehose in near real-time
• External destination stores the logs durably, independent of pod and node lifecycle

This is the node-level logging model. Kubernetes does not natively ship logs anywhere. It leaves that entirely to an external agent. Fluent Bit is that agent on EKS.

Two limitations of the native model you must account for in production:

1. Logs are local to the node. A Spot interruption, a scale-down event, or a hardware failure takes the log files with it.
2. Logs are per-pod. Once a pod is deleted, its log files are cleaned up. The pre-crash window is exactly when you need them most.

Why Fluent Bit on EKS

Fluent Bit is a lightweight log processor and forwarder written in C. It runs as a DaemonSet, one pod per node, so it is always co-located with the log files it needs to read. Because it runs on the node itself, it reads directly from /var/log/containers/ without any changes to your application.

Why a DaemonSet specifically? Log files live on the node where the container ran. The only process that can reliably read them is one running on the same node. A DaemonSet guarantees coverage: as EKS node groups scale out and new nodes join, a Fluent Bit pod is placed on each new node automatically.

Why Fluent Bit rather than something else? AWS maintains an EKS-optimised Fluent Bit container image and a Helm chart that pre-configures it for common EKS destinations. The amazon-cloudwatch-observability EKS add-on includes Fluent Bit out of the box. Compared to Fluentd, Fluent Bit has a smaller memory footprint per node, which matters when running one forwarder instance on every node in a large cluster.

Kubernetes metadata enrichment. Fluent Bit’s Kubernetes filter annotates every log record with pod_name, namespace_name, container_name, and the pod’s labels, automatically, without any code changes to your services. This makes it possible to filter CloudWatch Logs queries by namespace or service name even when all logs flow into a single log group.

Deploying Fluent Bit on EKS

Option 1: EKS add-on

The amazon-cloudwatch-observability add-on installs both the CloudWatch metrics agent and Fluent Bit together. This is the lowest-effort path for teams using Amazon EKS and sending logs to CloudWatch.

aws eks create-addon \
  --cluster-name my-cluster \
  --addon-name amazon-cloudwatch-observability \
  --region eu-west-1

Logs are sent to CloudWatch Logs automatically once the add-on is installed and the required IAM permissions are in place.

Option 2: Helm chart

Use the Helm chart when you need to customise destinations, log filtering, or output format.

helm repo add aws-observability https://aws-observability.github.io/helm-charts
helm repo update

helm install aws-for-fluent-bit aws-observability/aws-for-fluent-bit \
  --namespace amazon-cloudwatch \
  --create-namespace \
  --values fluent-bit-values.yaml

A basic fluent-bit-values.yaml to send logs to CloudWatch:

cloudWatch:
  enabled: true
  region: eu-west-1
  logGroupName: /eks/my-cluster/containers
  logStreamPrefix: from-eks-

firehose:
  enabled: false

kinesis:
  enabled: false

elasticsearch:
  enabled: false

IAM permissions via IRSA

Fluent Bit needs IAM permissions to write to CloudWatch Logs. Grant these via IRSA (IAM Roles for Service Accounts). Do not rely on the node instance profile, which grants the same permissions to every process running on the node.

The Fluent Bit service account needs these permissions:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "logs:CreateLogGroup",
        "logs:CreateLogStream",
        "logs:PutLogEvents",
        "logs:DescribeLogStreams"
      ],
      "Resource": "arn:aws:logs:*:*:*"
    }
  ]
}

Annotate the Fluent Bit service account with the IAM role ARN:

kubectl annotate serviceaccount fluent-bit \
  -n amazon-cloudwatch \
  eks.amazonaws.com/role-arn=arn:aws:iam::123456789012:role/FluentBitRole

Where EKS logs can go

Fluent Bit can forward to multiple destinations simultaneously. The main options on AWS:

CloudWatch Logs is the default for teams already using CloudWatch. It supports log queries with CloudWatch Insights, integrates with alarms and metric filters, and requires no additional infrastructure. Good starting point for most EKS clusters.

Amazon OpenSearch Service is better for full-text search, complex log queries, and dashboards. Consider it when CloudWatch Insights becomes slow or expensive at your log volume, or when you need rich visualisations alongside your metrics.

Amazon S3 is the cheapest long-term storage. Use S3 as a secondary destination for compliance retention when logs must be kept for years but are rarely queried. Not suitable for real-time debugging on its own.

Amazon Kinesis Data Firehose is a streaming intermediary that delivers to S3, OpenSearch, or Splunk with batching and optional transformation. Useful when your destination requires a specific format or you need to fan out to multiple systems.

When you need centralised log forwarding

You need Fluent Bit or an equivalent log forwarder in any of these situations:

• Production EKS workloads. Pods restart, nodes terminate, and Spot instances disappear. You cannot reconstruct incidents from logs that no longer exist.
• Multiple pods running the same service. kubectl logs shows one pod at a time. A centralised system lets you search across all replicas simultaneously.
• Compliance or audit requirements. Financial services, healthcare, and other regulated industries often require log retention for months or years. CloudWatch and S3 both support configurable retention policies.
• Alerting on log content. CloudWatch metric filters can count error-level log events and trigger alarms, but only if logs reach CloudWatch in the first place.
• Cross-service debugging. When a request touches several microservices, searching by a trace ID across all logs at once is far faster than switching between kubectl logs commands.

For local development or short-lived test clusters, kubectl logs is usually sufficient. For anything that handles real traffic, set up log forwarding before your first deployment.

kubectl logs vs Fluent Bit vs centralised logging

The practical rule: use kubectl logs for development and quick checks on running pods. Use Fluent Bit with a centralised destination for any cluster that handles real traffic.

Structured logging and why it matters here

The most queryable log format is JSON. When your application writes structured JSON to stdout, Fluent Bit can parse individual fields and you can filter on them in CloudWatch Insights or OpenSearch. The full detail is in the structured logging guide.

Unstructured (hard to query):

2026-03-16 14:23:51 INFO User 12345 logged in from 203.0.113.50

Structured JSON (queryable by user_id, action, ip_address):

{"timestamp":"2026-03-16T14:23:51Z","level":"INFO","action":"login","user_id":"12345","ip_address":"203.0.113.50","duration_ms":42}

A CloudWatch Insights query to find all login events for a specific user:

fields @timestamp, action, user_id, ip_address, duration_ms
| filter action = "login" and user_id = "12345"
| sort @timestamp desc

Without structured logging, this query is not possible without manual text parsing. Switch to JSON logs early. Retrofitting it later requires application changes across every service.

Filtering and enriching logs with Fluent Bit

Fluent Bit processes logs through an input, filter, and output pipeline. The Kubernetes filter is the most important stage for EKS. It adds pod metadata to every log record:

[FILTER]
    Name                kubernetes
    Match               kube.*
    Kube_URL            https://kubernetes.default.svc:443
    Kube_CA_File        /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    Kube_Token_File     /var/run/secrets/kubernetes.io/serviceaccount/token
    Merge_Log           On
    K8S-Logging.Parser  On
    K8S-Logging.Exclude Off

After this filter, every log record automatically includes kubernetes.pod_name, kubernetes.namespace_name, kubernetes.container_name, and the pod’s labels, without any changes to your application code.

You can also drop noisy health check logs before they consume log storage:

[FILTER]
    Name    grep
    Match   kube.*
    Exclude path /healthz

Filtering at the Fluent Bit level is cheaper than ingesting everything and filtering at query time, especially at high log volumes.