EKS Monitoring Guide: CloudWatch, Prometheus & Grafana

Your monitoring options, and when to use each

Four main options exist for EKS monitoring:

• CloudWatch Container Insights: the AWS-native path. Install the Amazon CloudWatch Observability EKS add-on and you get infrastructure metrics, Kubernetes state metrics, and log forwarding out of the box. Good fit if your team already works in the AWS Console and wants minimal operational overhead.
• Prometheus and Grafana (self-managed): the most widely used open-source monitoring stack for Kubernetes, installed via the kube-prometheus-stack Helm chart. More powerful query language (PromQL), richer dashboards, and better support for application-level metrics. Requires you to run and maintain the stack yourself.
• Amazon Managed Service for Prometheus (AMP) with Grafana: Prometheus-compatible managed backend. Your Prometheus or the AWS managed collector scrapes your cluster and sends metrics to AMP; Grafana queries AMP for dashboards. Best when you want Prometheus without managing its storage and availability.
• Hybrid: CloudWatch for infrastructure and audit logs, Prometheus or AMP for application metrics and dashboards. Common in production environments where teams want both deep Kubernetes metrics and AWS-native compliance tooling.

For a deeper look at the CloudWatch side of this, see the How to Monitor Amazon EKS with CloudWatch guide. This page focuses on comparing your options and implementing them.

What EKS monitoring actually means

EKS monitoring has four distinct concerns that are easy to blur together:

• Infrastructure metrics: CPU, memory, disk, and network at the node level. This tells you whether your EC2 instances are under pressure. The CloudWatch agent or Prometheus node exporter collects these.
• Kubernetes state metrics: pod phases (Running, Pending, Failed), deployment replica counts, job completions. These come from the Kubernetes API via kube-state-metrics, not from the nodes themselves.
• Logs: the stdout/stderr output from containers, plus EKS control plane logs (API server, scheduler, controller manager, audit). Fluent Bit forwards container logs; control plane logs are enabled separately per cluster.
• Alerts and dashboards: rules that fire when something crosses a threshold, visualized in CloudWatch or Grafana.

These four concerns are separate. You can have excellent infrastructure metrics but no visibility into whether your deployments are degraded. You can have all your logs and no alerts. A useful monitoring setup covers all four.

How EKS monitoring works

Understanding the data flow makes the architecture less confusing:

1. Sources: pods, nodes, and the EKS control plane generate metrics and logs continuously.
2. Agents and collectors: the CloudWatch agent (or Prometheus node exporter + kube-state-metrics) runs on each node and scrapes or receives these signals. Fluent Bit collects logs.
3. Storage: metrics flow to CloudWatch (custom metrics namespace) or to Prometheus / AMP. Logs flow to CloudWatch Logs log groups.
4. Query and dashboards: CloudWatch console, CloudWatch Dashboards, or Grafana query the stored data.
5. Alerts: CloudWatch Alarms or Prometheus Alertmanager evaluate rules and send notifications to SNS, PagerDuty, Slack, or email.

Pods / Nodes / Control Plane
       |
       | (metrics)              (logs)
       v                          v
CloudWatch Agent           Fluent Bit DaemonSet
or Prometheus scraper             |
       |                          v
       v                    CloudWatch Logs
CloudWatch Metrics               |
or AMP (Prometheus storage)      v
       |                   CloudWatch Insights
       v                   (query and search)
Dashboards + Alarms
(CloudWatch Console or Grafana)

Which setup should you choose?

For a comparison of broader EKS cluster management models, see Managed vs Self-Managed Kubernetes in AWS.

Quick checks with kubectl top

The fastest way to check current resource usage is kubectl top. This is useful for ad-hoc spot checks, not for monitoring.

# Check node resource usage
kubectl top nodes

# NAME                           CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
# ip-10-0-1-100.ec2.internal    245m         12%    1824Mi          47%
# ip-10-0-1-101.ec2.internal    412m         20%    2134Mi          55%

# Check pod resource usage in a namespace
kubectl top pods -n production --sort-by=memory

kubectl top depends on Metrics Server being installed (see the Horizontal Pod Autoscaling guide for that setup).

What it is good for: seeing which pods or nodes are consuming the most resources right now. Useful when debugging an active issue.

CloudWatch Container Insights on EKS

Container Insights collects metrics at the cluster, node, namespace, pod, and container level and makes them visible in the CloudWatch console under Insights > Container Insights.

What it collects:

• Node CPU and memory utilization
• Pod CPU, memory, network, and disk usage
• Number of running pods per namespace
• Container restart counts
• EKS control plane metrics via CloudWatch Logs

Installing via the EKS add-on (recommended)

The Amazon CloudWatch Observability EKS add-on installs the CloudWatch agent and Fluent Bit as DaemonSets in a single step. It also enables CloudWatch Application Signals for automatic application performance monitoring:

aws eks create-addon \
  --cluster-name my-cluster \
  --addon-name amazon-cloudwatch-observability \
  --region eu-west-1

IAM permissions

The add-on needs an IAM role with CloudWatchAgentServerPolicy. The preferred approach is EKS Pod Identity, which is the current AWS-recommended credential path for pod-level IAM access. IAM Roles for Service Accounts (IRSA) also works and is still widely used on existing clusters.

# Create the role and attach the policy
aws iam create-role \
  --role-name EKSCloudWatchObservabilityRole \
  --assume-role-policy-document file://eks-pod-identity-trust-policy.json

aws iam attach-role-policy \
  --role-name EKSCloudWatchObservabilityRole \
  --policy-arn arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy

# Associate the role with the add-on service account
aws eks create-pod-identity-association \
  --cluster-name my-cluster \
  --namespace amazon-cloudwatch \
  --service-account cloudwatch-agent \
  --role-arn arn:aws:iam::123456789012:role/EKSCloudWatchObservabilityRole

Metrics appear in the ContainerInsights CloudWatch namespace within a few minutes.

Prometheus and Grafana on EKS

When self-managed Prometheus makes sense:

• Your team already knows PromQL
• You need application-level metrics exposed by your services (request rate, error rate, latency percentiles)
• You want community dashboards — Grafana has thousands of pre-built Kubernetes dashboards
• Cost at scale is a concern and you can carry the operational overhead

Install via the kube-prometheus-stack Helm chart, which bundles Prometheus, Grafana, Alertmanager, and pre-built Kubernetes dashboards. See the Helm Package Manager guide if you are new to Helm.

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

helm install prometheus-stack prometheus-community/kube-prometheus-stack \
  --namespace monitoring \
  --create-namespace \
  --set grafana.adminPassword=change-this-password

Access Grafana locally via port forwarding during initial setup:

kubectl port-forward service/prometheus-stack-grafana 3000:80 -n monitoring

Open http://localhost:3000, log in with admin and the password you set. The default dashboards show cluster-wide CPU and memory usage, pod restart rates, and network traffic.

Amazon Managed Service for Prometheus (AMP)

AMP is a fully managed Prometheus-compatible backend. You send metrics to it the same way you would send them to a self-managed Prometheus server, but AWS handles storage, scaling, and high availability.

When AMP is better than self-managed Prometheus:

• You need long-term metric retention beyond what a local PersistentVolume can hold
• You have multiple clusters and want to aggregate metrics in one place
• You do not want to manage Prometheus availability and storage yourself
• You are already using Amazon Managed Grafana and want a tightly integrated stack

Two ways to get metrics into AMP

Option 1: Prometheus remote_write. If you already have Prometheus running, configure it to write metrics to AMP:

# prometheus-values.yaml
prometheus:
  prometheusSpec:
    remoteWrite:
    - url: https://aps-workspaces.eu-west-1.amazonaws.com/workspaces/ws-xxxxx/api/v1/remote_write
      sigv4:
        region: eu-west-1
        roleArn: arn:aws:iam::123456789012:role/prometheus-remote-write-role

Option 2: AWS managed scraper. AMP can deploy a managed scraper into your cluster via the AWS API. You do not need to install Prometheus at all. AWS provisions the scraper, discovers your pods, and sends metrics directly to your AMP workspace.

aws amp create-scraper \
  --source eksConfiguration="{clusterArn='arn:aws:eks:eu-west-1:123456789012:cluster/my-cluster',subnetIds=['subnet-abc123']}" \
  --destination ampConfiguration="{workspaceArn='arn:aws:aps:eu-west-1:123456789012:workspace/ws-xxxxx'}" \
  --scrape-configuration configurationBlob=$(cat scrape-config.yaml | base64)

Grafana with AMP: use either Amazon Managed Grafana or a self-hosted Grafana instance. Both connect to AMP using the Prometheus data source plugin with SigV4 authentication, configured via IRSA or EKS Pod Identity.

Logs and control plane visibility

Metrics tell you something is wrong. Logs tell you why. The two are not interchangeable.

Pod and container logs are written to stdout/stderr by containers and stored as files on each node. Fluent Bit, running as a DaemonSet, reads those files and forwards them to CloudWatch Logs. The CloudWatch Observability add-on installs Fluent Bit automatically. For more on how log forwarding works, see Logging in Kubernetes.

CloudWatch Logs organizes container logs by:

• /aws/containerinsights/CLUSTER_NAME/application: container stdout/stderr
• /aws/containerinsights/CLUSTER_NAME/host: node system logs
• /aws/containerinsights/CLUSTER_NAME/dataplane: kubelet and containerd logs

EKS control plane logs are separate from container logs. They cover the API server, scheduler, controller manager, authenticator, and audit trail. Enable them per cluster:

aws eks update-cluster-config \
  --name my-cluster \
  --region eu-west-1 \
  --logging '{"clusterLogging":[{"types":["api","audit","authenticator","controllerManager","scheduler"],"enabled":true}]}'

Audit logs record every API call made to the cluster: who made it, which resource they accessed, and whether it was allowed. They are essential for security investigations. Query them with CloudWatch Logs Insights:

fields @timestamp, @message
| filter @logStream like /kube-apiserver-audit/
| filter requestURI like /secrets/
| sort @timestamp desc
| limit 100

For guidance on querying and searching logs, see CloudWatch Logs.

What to alert on first

Start with a small set of high-signal alerts. The goal is to catch real problems quickly, not to alert on everything that could theoretically go wrong.

These are starter thresholds for a typical cluster. Tune them once you have baseline data. A batch-processing cluster running near capacity will need looser CPU thresholds than a lightly loaded API cluster.

For step-by-step instructions on creating these in CloudWatch, see Creating CloudWatch Alarms.

Which setup fits your situation