GKE Cluster Upgrades: Safe Upgrade Checklist, Strategies, and Downtime Prevention

Simple explanation

A GKE cluster upgrade means moving from one Kubernetes version to a newer one. The cluster has two layers:

• The control plane is the brain: it runs the API server, scheduler, and controller manager. GKE manages it entirely. You cannot SSH into it or schedule pods on it.
• The node pools are the worker machines where your pods actually run. These are upgraded separately, after the control plane.

Upgrades always happen in order: control plane first, then node pools. You cannot upgrade a node pool past the control plane version.

The control plane upgrade is low risk. GKE replaces it in-place, your pods keep running, and the API server is briefly unreachable (usually under ten minutes on a regional cluster). The node pool upgrade is where the real risk lives. GKE drains each node, evicts your pods, and replaces the machine. If your workloads only have one replica, or if no PodDisruptionBudget is in place, that eviction can briefly take an application offline.

The goal of a safe upgrade is to make sure your workloads tolerate that disruption gracefully.

How GKE upgrades work

Kubernetes version support

Kubernetes follows an N-2 support policy: the project actively supports the current release and the two previous minor versions. If the current release is 1.32, versions 1.31 and 1.30 are supported but 1.29 is not. GKE adds its own extended support window, but the principle holds. Running an unsupported version means no security patches and eventual forced upgrades.

Each minor version goes through a GKE lifecycle:

1. Available — the version can be selected for new clusters or upgrades
2. Default — the version used when no specific version is requested
3. Deprecated — an end-of-support date has been announced; clusters should be upgraded
4. End of life — GKE will automatically upgrade clusters on this version, potentially disruptively

Control plane vs node pool upgrades

The control plane is always upgraded first and runs on Google-managed infrastructure, separate from your node VMs. A control plane upgrade typically takes 5–10 minutes on a regional cluster (which has multiple control plane replicas spread across zones). During that window, kubectl commands may fail, but your running pods and application traffic are unaffected.

Node pool upgrades come after. GKE drains each node, evicting pods so they reschedule elsewhere, then replaces the machine with one running the target Kubernetes version. This is the stage that can affect your workloads if they are not resilient.

Automatic vs manual upgrades

You can let GKE handle upgrades automatically by enrolling in a release channel, or manage upgrades manually for tighter control. Most teams benefit from release channels. See the comparison section below for when manual makes sense.

# List currently available GKE versions
gcloud container get-server-config --region=europe-west2

# Check what version your cluster is running
gcloud container clusters describe my-cluster \
  --region=europe-west2 \
  --format='value(currentMasterVersion,currentNodeVersion)'

# Check the release channel your cluster is enrolled in
gcloud container clusters describe my-cluster \
  --region=europe-west2 \
  --format='value(releaseChannel.channel)'

When this guidance applies

This page is most relevant if you are:

• Running production clusters on GKE Standard mode and need to plan a controlled upgrade
• Managing clusters manually and want to understand release channel trade-offs
• Upgrading a cluster that has fallen behind by one or more minor versions
• Running workloads that cannot tolerate unexpected downtime during a node drain
• Preparing a staging environment to validate an upgrade path before touching production
• Evaluating surge vs blue-green upgrades for a critical node pool

If you are running GKE Autopilot, node upgrades are handled entirely by Google. You still benefit from understanding release channels and maintenance windows, but you do not manage node pool upgrade strategies directly.

Before you upgrade a GKE cluster

Run through this checklist before every upgrade. Most post-upgrade incidents trace back to one of these being skipped.

• Confirm current versions. Check both the control plane and node pool versions, and identify the exact target version you intend to upgrade to.
• Check release channel enrolment. Know whether your cluster is in a channel or unmanaged. If it is in a channel, confirm the available versions for that channel.
• Review deprecated APIs. Use Pluto or the GKE Console upgrade warnings to audit your manifests and Helm charts for API versions removed in the target release.
• Read the Kubernetes changelog. Review release notes for each minor version you are crossing. Look for breaking changes to admission controllers, security defaults, RBAC behaviour, and networking.
• Check PodDisruptionBudgets. Confirm PDBs are configured for every production Deployment that cannot tolerate downtime.
• Verify replica counts. Any workload with a single replica has no redundancy during a node drain. Increase replicas before upgrading.
• Review node pool upgrade strategy. Confirm max-unavailable is 0 for production pools. Decide whether surge or blue-green is more appropriate for critical workloads.
• Review maintenance windows and exclusions. If you are triggering a manual upgrade, confirm no active exclusion is blocking it.
• Test in staging first. Upgrade a non-production cluster to the target version and run your full test suite before touching production.
• Notify stakeholders. Even upgrades that complete without incident carry some risk. Make sure the right people know an upgrade is happening and have monitoring dashboards open.
• Confirm your rollback plan. Know what you would do if something breaks after the upgrade. Blue-green upgrades give you a soak period for rollback; surge upgrades do not.

GKE release channels explained

The simplest way to keep a GKE cluster up to date is to enrol it in a release channel: a subscription that automatically upgrades the cluster’s control plane and, optionally, its node pools to newer Kubernetes versions as they are validated by Google.

GKE offers three channels:

• Rapid — newest Kubernetes versions as soon as they are available in GKE, before reaching Regular or Stable. Suitable for development and testing environments where you want early access to new features or need to test future API compatibility.
• Regular — versions reach this channel two to three months after Rapid. This is the recommended channel for most production workloads. It balances access to recent features with additional validation time from Google and the wider GKE user base.
• Stable — versions reach this channel two to three months after Regular and have typically been running in production across many GKE clusters for months. Use Stable for business-critical workloads where any change carries organisational risk.

# Create a cluster enrolled in the Regular release channel
gcloud container clusters create my-cluster \
  --region=europe-west2 \
  --release-channel=regular \
  --num-nodes=3

# Enrol an existing cluster in a release channel
gcloud container clusters update my-cluster \
  --region=europe-west2 \
  --release-channel=regular

# Enable automatic node pool upgrades (recommended with release channels)
gcloud container node-pools update default-pool \
  --cluster=my-cluster \
  --region=europe-west2 \
  --enable-autoupgrade

Release channels vs manual upgrades

Choosing between a release channel and manual version management is one of the first decisions you make when setting up a GKE cluster.

When release channels are better: You want GKE to stay current without manual intervention. Configure maintenance windows to restrict timing and let GKE do the rest. This is the right choice for the majority of production clusters.

When manual upgrades are better: Your organisation requires change control approval before any version bump, or your workloads have hard dependencies on specific Kubernetes API behaviour that must be validated before committing to an upgrade. Manual management is more work and more error-prone. You take on the responsibility of tracking version EOL dates and triggering upgrades before GKE forces one.

Maintenance windows and exclusions

Even when enrolled in a release channel, you can control when GKE applies upgrades using maintenance windows and maintenance exclusions. This lets you avoid upgrades during peak traffic periods, major deployments, or planned deployment freezes.

A maintenance window defines recurring time slots during which GKE is allowed to perform upgrades. Outside of those windows, GKE will not start an upgrade automatically.

A maintenance exclusion blocks upgrades during a specific period, even if a maintenance window would otherwise allow it. Use exclusions for product launches, major sales periods, or team holidays.

# Set a recurring maintenance window: every Saturday 02:00–06:00 UTC
gcloud container clusters update my-cluster \
  --region=europe-west2 \
  --maintenance-window-start=2026-03-07T02:00:00Z \
  --maintenance-window-end=2026-03-07T06:00:00Z \
  --maintenance-window-recurrence="FREQ=WEEKLY;BYDAY=SA"

# Add a maintenance exclusion to block upgrades during a busy period
gcloud container clusters update my-cluster \
  --region=europe-west2 \
  --add-maintenance-exclusion-name=black-friday \
  --add-maintenance-exclusion-start=2026-11-27T00:00:00Z \
  --add-maintenance-exclusion-end=2026-11-30T23:59:59Z \
  --add-maintenance-exclusion-scope=NO_UPGRADES

# List current maintenance policy
gcloud container clusters describe my-cluster \
  --region=europe-west2 \
  --format='yaml(maintenancePolicy)'

Node pool upgrade strategies

The control plane is always upgraded first and is fully managed by GKE. The node pools — where your pods run — have configurable upgrade strategies that control how individual nodes are replaced.

Surge upgrades

Surge upgrades are the default strategy. Think of it like a relay race: a fresh runner (the surge node) steps in before the existing runner steps off. GKE temporarily adds extra nodes to the pool, then drains and replaces existing nodes one at a time. No node goes offline until a replacement is ready.

Two parameters control surge behaviour:

• max-surge — how many extra nodes to add during the upgrade (default: 1)
• max-unavailable — how many nodes can be simultaneously unavailable during the upgrade (default: 0)

# Configure surge upgrade settings for a node pool
gcloud container node-pools update default-pool \
  --cluster=my-cluster \
  --region=europe-west2 \
  --max-surge-upgrade=1 \
  --max-unavailable-upgrade=0

Setting max-unavailable to 0 means no node is taken offline until a surge replacement is ready. This avoids reduced capacity at any point but slows the upgrade. Increasing max-surge speeds the upgrade at the cost of temporarily higher compute bills.

Blue-green upgrades

Blue-green upgrades take a different approach. Think of it like keeping your old apartment while you move into a new one — you do not hand back the keys until you are completely settled.

GKE provisions a completely new node pool (the green pool) running the target Kubernetes version alongside the existing pool (blue). GKE cordons the blue nodes, migrates workloads to the green pool, waits for a configurable soak period, then deletes the blue pool.

The main advantage is rollback. During the soak period, if you discover a problem — application behaviour changes, unexpected scheduling issues, API compatibility errors — you can redirect workloads back to the blue pool before it is deleted. The trade-off is cost: you run double the nodes for the duration of the soak.

# Enable blue-green upgrade strategy for a node pool
gcloud container node-pools update default-pool \
  --cluster=my-cluster \
  --region=europe-west2 \
  --node-pool-soak-duration=3600s \
  --strategy=BLUE_GREEN

Surge vs blue-green upgrades

For most production node pools, surge upgrades with max-unavailable=0 are sufficient and much cheaper. Use blue-green when you are crossing a major version boundary, when workloads have complex state, or when you need the ability to abort the upgrade after workloads have migrated.

PodDisruptionBudgets and workload safety

A PodDisruptionBudget (PDB) is a Kubernetes policy that limits how many pods of a given application can be simultaneously unavailable due to voluntary disruptions: node drains during upgrades, manual evictions, or cluster autoscaler scale-down events. PDBs do not protect against involuntary disruptions such as node failures.

Without a PDB, draining a node during an upgrade might evict all replicas of a Deployment at once if they happen to be co-located on the same node. A PDB prevents this by blocking the drain until the availability constraint is satisfied.

# Ensure at least 2 replicas are always available
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: my-app-pdb
  namespace: default
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: my-app

# Alternative: allow at most 1 replica to be unavailable at a time
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: my-app-pdb
  namespace: default
spec:
  maxUnavailable: 1
  selector:
    matchLabels:
      app: my-app

# Check PDBs in the default namespace
kubectl get pdb

# Describe a specific PDB (shows current disruptions allowed)
kubectl describe pdb my-app-pdb

Handling deprecated API versions

Each Kubernetes minor version deprecates some API versions and removes them after a version or two. Applying a manifest that references a removed API to an upgraded cluster returns a 404 error and your deployment fails silently.

This is one of the most common causes of post-upgrade incidents. The fix is to audit your manifests before upgrading, not after.

# Install pluto — detects deprecated Kubernetes APIs in manifest files
# https://github.com/FairwindsOps/pluto

pluto detect-files -d ./k8s-manifests/

# Check for deprecated APIs in a live cluster
pluto detect-api-resources --target-versions k8s=v1.32.0

# Check Helm releases for charts using deprecated APIs
helm list --all-namespaces
helm get manifest my-release | grep "^apiVersion:"

When upgrading across multiple minor versions, check the Kubernetes changelog for each intermediate version. Each minor release can add new deprecations or removals. Updating Helm charts to versions that support the current API group is usually the fastest fix.

Common mistakes when upgrading GKE clusters

Triggering a manual upgrade

If you are not enrolled in a release channel, or want to upgrade ahead of the automatic schedule, use these commands:

# List available versions for the cluster's current channel or region
gcloud container get-server-config \
  --region=europe-west2 \
  --format='yaml(channels)'

# Upgrade the control plane to a specific version (always upgrade this first)
gcloud container clusters upgrade my-cluster \
  --region=europe-west2 \
  --master \
  --cluster-version=1.32.2-gke.1200

# After the control plane upgrade completes, upgrade the node pool
gcloud container clusters upgrade my-cluster \
  --region=europe-west2 \
  --node-pool=default-pool \
  --cluster-version=1.32.2-gke.1200

# Monitor upgrade progress
gcloud container operations list \
  --filter="TARGET=my-cluster AND STATUS=RUNNING"

The control plane upgrade typically takes 5–10 minutes and does not affect running pods. Node pool upgrades take longer depending on pool size, the configured upgrade strategy, and how long pods take to reschedule.

Manual node operations during upgrades

Understanding the underlying kubectl operations is useful for troubleshooting a stuck upgrade or performing a manual drain before maintenance.

Cordon marks a node as unschedulable. No new pods will be placed on it, but existing pods continue running:

# Prevent new pods being scheduled on a node
kubectl cordon gke-my-cluster-default-pool-abc123

# Verify the node is cordoned (status shows SchedulingDisabled)
kubectl get nodes

Drain evicts all running pods from the node (except DaemonSet pods). Kubernetes respects PodDisruptionBudgets during drains. If a drain would violate a PDB, it blocks until the constraint can be satisfied:

# Drain a node (evicts pods, respects PDBs, ignores DaemonSets)
kubectl drain gke-my-cluster-default-pool-abc123 \
  --ignore-daemonsets \
  --delete-emptydir-data

# After maintenance, bring the node back into service
kubectl uncordon gke-my-cluster-default-pool-abc123

—delete-emptydir-data allows eviction of pods using emptyDir volumes, which lose their data on eviction. Without this flag, the drain blocks on any pod using an emptyDir volume.

Step-by-step safe upgrade workflow

Use this sequence for any production cluster upgrade, whether triggered manually or observed as an automatic upgrade in progress.

1. Inspect versions and release channel

gcloud container clusters describe my-cluster \
  --region=europe-west2 \
  --format='value(currentMasterVersion,currentNodeVersion,releaseChannel.channel)'

2. Review compatibility and deprecated APIs

Run Pluto against your manifests and review the GKE Console upgrade warnings. Check the Kubernetes changelog for each minor version between your current and target versions.

3. Validate workload resilience

Confirm all production Deployments have more than one replica. Check that PDBs exist for critical services. Review Services and Ingress configurations for anything that could behave differently after an API or networking change.

4. Choose upgrade strategy

Decide between surge (default, lower cost) and blue-green (rollback capability, higher cost) for each node pool. Set max-unavailable=0 for any pool running production workloads.

5. Schedule maintenance

If using a release channel, confirm your maintenance window allows the upgrade. If upgrading manually, pick a low-traffic window and notify stakeholders.

6. Test in staging

Upgrade a non-production cluster to the target version first. Deploy and test your workloads against the upgraded version. Soak for at least a full business day.

7. Upgrade the control plane (if manual)

gcloud container clusters upgrade my-cluster \
  --region=europe-west2 \
  --master \
  --cluster-version=1.32.2-gke.1200

Wait for the operation to complete before proceeding. The API server will be briefly unavailable.

8. Upgrade node pools

gcloud container clusters upgrade my-cluster \
  --region=europe-west2 \
  --node-pool=default-pool \
  --cluster-version=1.32.2-gke.1200

Watch your monitoring dashboards throughout. Check for pod restarts and scheduling failures as each node is replaced.

9. Monitor workloads during and after

Keep GKE monitoring and Kubernetes logging open throughout the upgrade. Watch for error rate spikes, pod restarts, and any OOMKilled or CrashLoopBackOff events.

10. Validate and soak

Run the validation checks below. Then soak for 24–48 hours under normal production load before considering the upgrade closed.

How to validate the cluster after an upgrade

After the upgrade completes, run these checks before declaring it successful:

# Confirm control plane and node versions are aligned
kubectl version
kubectl get nodes -o wide

# Check that all pods are running and no restarts are spiking
kubectl get pods --all-namespaces
kubectl get pods --all-namespaces --field-selector=status.phase!=Running

# Look for CrashLoopBackOff or ImagePullBackOff
kubectl get pods --all-namespaces | grep -E "CrashLoop|ImagePull|Error|Pending"

# Check recent events for warnings
kubectl get events --all-namespaces --sort-by='.lastTimestamp' | tail -30

# Verify PDBs are not blocking anything
kubectl get pdb --all-namespaces

Beyond the kubectl checks, verify:

• Application health. Exercise your actual application endpoints, not just pod status. A pod can be Running while returning 500s.
• Ingress and services. Confirm that Ingress controllers and Services are routing correctly. Load balancer IP assignments and backend health checks can sometimes need a few minutes after a node pool replacement.
• API deprecation errors. Check your application logs and the Kubernetes API server audit logs for any 404s caused by removed API versions.
• Metrics and alerts. Review your Cloud Monitoring dashboards for error rate, latency, and resource utilisation. If you have alerting configured, confirm no new alerts fired during the upgrade window.
• Node and control plane version parity. Confirm all nodes in each pool are on the target version. Stale nodes from a partially completed upgrade will show up in kubectl get nodes.