GKE Monitoring Explained: Cloud Monitoring, kubectl top, and Managed Prometheus

What GKE monitoring gives you

Think of monitoring a GKE cluster as watching three layers at once: are the machines (nodes) healthy, are the workloads (pods) running, and are the applications behaving correctly. GKE handles the first two layers automatically. The third layer requires your application to expose metrics.

The tools GKE gives you out of the box:

• Cloud Monitoring — the main observability platform. GKE automatically sends CPU, memory, network, and restart metrics here. You build dashboards and alerting policies on top of those metrics.

• Cloud Logging — collects logs from every container running in your cluster. Logs complement metrics when you need to understand why something went wrong, not just that it went wrong. See Logging in Kubernetes on GKE for how this works.

• kubectl top — shows current CPU and memory usage for nodes and pods directly from the metrics-server. Good for quick spot-checks, not a substitute for Cloud Monitoring.

• Managed Service for Prometheus — a fully managed Prometheus-compatible backend. Use it when your applications expose custom metrics that Cloud Monitoring does not collect by default.

None of these require manual agent installation. The cluster handles the integration automatically.

How monitoring flows from cluster to alert

1. GKE emits metrics continuously. Every node runs a monitoring agent that collects CPU utilisation, memory usage, disk I/O, network throughput, and pod restart counts. These are sent to Cloud Monitoring in near real time.

2. Cloud Monitoring stores the metrics as time-series data. All GKE metrics are stored under the kubernetes.io monitored resource, keyed by cluster, namespace, pod, and container. They are retained for 24 months by default.

3. Dashboards visualise trends over time. You can use the pre-built GKE dashboards under Kubernetes Engine › Observability, or build custom dashboards that combine GKE metrics with metrics from other services.

4. Alerting policies notify you when thresholds are crossed. You define conditions like “node CPU above 80% for 5 minutes” and Cloud Monitoring fires a notification to email, Slack, PagerDuty, or Pub/Sub when the condition is met.

5. Managed Service for Prometheus adds application-level metrics. If your application exposes a /metrics endpoint, a PodMonitoring resource tells GMP to scrape it and send those metrics into Cloud Monitoring alongside the built-in GKE metrics.

6. Logs and traces fill in the gaps. When metrics show something is wrong, Cloud Trace shows request latency through your services and Cloud Profiler shows which functions are consuming CPU or memory.

Built-in monitoring in GKE

When you create a GKE cluster, Cloud Monitoring is enabled by default. A monitoring agent runs on each node and forwards metrics automatically, with no installation or configuration required.

Metrics cover three levels:

• Node metrics — CPU utilisation, memory utilisation, disk I/O, and network throughput per node. See Node Pools Explained for how nodes are organised.

• Pod and container metrics — CPU and memory requests versus actual usage for every running container. Kubernetes Pods Explained covers the pod model this builds on.

• Cluster-level metrics — node count, node conditions, API server request latency, and etcd metrics.

All GKE metrics in Cloud Monitoring live under the kubernetes.io monitored resource:

kubernetes.io/container/cpu/core_usage_time
kubernetes.io/container/memory/used_bytes
kubernetes.io/node/cpu/allocatable_utilization
kubernetes.io/node/memory/allocatable_utilization
kubernetes.io/pod/network/received_bytes_count
kubernetes.io/pod/network/sent_bytes_count

Explore these in Metrics Explorer by searching for the kubernetes.io namespace. For a broader overview of how GCP metrics work, see Metrics in GCP.

Monitoring from the Cloud Console

The Cloud Console provides a live Kubernetes view under Kubernetes Engine › Workloads. This is often the fastest place to start when something looks wrong.

For each workload (Deployments, StatefulSets, DaemonSets, Jobs), you can see:

• Pod status — Running, Pending, Failed, or CrashLoopBackOff for each pod.
• Restart count — a rising restart count is one of the clearest signals that a container is crashing repeatedly.
• Unavailable pods — any gap between desired and running pods means a deployment is degraded.
• CPU and memory usage — displayed relative to the requests set on each container, so you can see at a glance which workloads are near their limits.
• Node placement — which pods are running on which nodes.

Clicking into a specific workload shows historical CPU and memory graphs, the associated pod list, and a direct link into Cloud Logging for that workload’s logs. For most incidents, you can go from “something is wrong” to a working hypothesis without leaving this view.

Real-time checks with kubectl top

For instant command-line visibility, kubectl top queries the metrics-server pre-installed on GKE clusters and returns current CPU and memory usage.

# Current CPU and memory for all nodes
kubectl top nodes

# Current usage for all pods in a namespace
kubectl top pods -n my-namespace

# Usage across all namespaces
kubectl top pods -A

# Sort by memory usage
kubectl top pods -n my-namespace --sort-by=memory

# Show per-container usage within pods
kubectl top pods -n my-namespace --containers

The output shows actual current usage, not configured requests or limits. Comparing kubectl top output against the requests and limits in your manifests quickly reveals pods near their limits (throttling or OOM risk) and pods that are massively over-allocated (wasted cost).

Dashboards in Cloud Monitoring

Dashboards give you a persistent, visual view of cluster health over time. Unlike the Workloads console view, dashboards let you compare metrics across time windows, correlate node pressure with pod restarts, and share a single URL with your team during an incident.

GKE provides pre-built dashboards under Kubernetes Engine › Observability that display cluster health, node status, and workload performance without any setup. For a custom dashboard, open Cloud Monitoring › Dashboards › Create Dashboard and add charts for:

• Node CPU utilisation — kubernetes.io/node/cpu/allocatable_utilization grouped by node name
• Node memory utilisation — kubernetes.io/node/memory/allocatable_utilization grouped by node name
• Pod CPU usage — kubernetes.io/container/cpu/core_usage_time grouped by namespace or pod name
• Pod memory usage — kubernetes.io/container/memory/used_bytes grouped by pod name
• Container restart count — kubernetes.io/container/restart_count; rising values are an early warning signal

For a deeper look at how to build and structure dashboards, see Dashboards in Cloud Monitoring.

Alerting for GKE clusters

Dashboards are passive: you have to be looking at them. Alerting policies are active. Configure them in Cloud Monitoring › Alerting › Create Policy so you get notified when something crosses a threshold, even at 3am.

Four policies to set up first:

Container restart rate — signals CrashLoopBackOff or repeated crashes:

Metric:    kubernetes.io/container/restart_count
Filter:    namespace_name = "production"
Condition: rate of change > 5 restarts in 10 minutes

Node CPU pressure:

Metric:    kubernetes.io/node/cpu/allocatable_utilization
Condition: value > 0.80 (80%) for 5 minutes

Sustained high CPU on a node means workloads are competing for resources. New pods may fail to schedule and existing pods may be throttled.

Node memory pressure:

Metric:    kubernetes.io/node/memory/allocatable_utilization
Condition: value > 0.85 (85%) for 5 minutes

Deployment unavailable replicas — signals a failed rollout or crashing pods:

Metric:    kubernetes.io/deployment/desired_pods minus kubernetes.io/deployment/available_pods
Condition: value > 0 for 5 minutes

Any gap between desired and available replicas means your service is running at reduced capacity.

Alerts can notify via email, PagerDuty, Slack webhooks, or Pub/Sub topics. See Creating Alerts in Cloud Monitoring for full policy configuration, and Incident Response with Monitoring for how to structure your response process.

Managed Service for Prometheus

The built-in GKE metrics cover infrastructure: nodes, pods, containers. They do not know anything about your application’s internal state — how many items are in a queue, how fast requests are completing, or how many errors a specific endpoint is returning.

For application-level metrics, use Google Cloud Managed Service for Prometheus (GMP): a fully managed, Prometheus-compatible backend integrated with Cloud Monitoring. You get Prometheus scraping without running or managing a Prometheus server.

A PodMonitoring resource scrapes metrics from pods matching a label selector within a single namespace:

apiVersion: monitoring.googleapis.com/v1
kind: PodMonitoring
metadata:
  name: my-app-metrics
  namespace: production
spec:
  selector:
    matchLabels:
      app: my-app
  endpoints:
    - port: metrics
      interval: 30s

ClusterPodMonitoring works the same way but is cluster-scoped, scraping matching pods across all namespaces. Use PodMonitoring when you want namespace-level control; use ClusterPodMonitoring for platform-wide metrics collection.

Once configured, your custom metrics appear in Cloud Monitoring under the prometheus.googleapis.com namespace. Query them using PromQL in Metrics Explorer, add them to dashboards, or connect Grafana to GMP as a data source. GMP is also what powers custom-metric-based autoscaling with Horizontal Pod Autoscaling.

Cloud Monitoring vs kubectl top vs Managed Prometheus

Use all three. kubectl top answers “what is consuming resources right now.” Cloud Monitoring answers “what happened over the last hour, day, or month and what should alert me.” Managed Prometheus answers “what is my application doing internally.”

When to set up GKE monitoring

The short answer is: before your first production deployment. Specifically, set up dashboards and alerting when:

• You are running any workload in production. Alerts should be in place before go-live, not after the first incident.
• You need historical metrics to understand trends, compare before and after a change, or run a post-mortem.
• You are diagnosing performance issues: comparing actual usage against requests and limits, or identifying throttled containers.
• You need to catch node pressure before pods get evicted and the cluster runs out of scheduling capacity.
• Your applications expose custom metrics and you want to alert on them or use them to drive autoscaling.
• You are doing cluster upgrades. Monitoring before and after confirms the upgrade did not degrade workload performance. See Upgrading GKE Clusters Safely.