GCP Load Balancer Backend Unhealthy? Fix Health Checks Fast

Simple explanation

A backend is any server, VM, container, or service that sits behind a load balancer and handles actual requests. In GCP, backends are grouped in managed instance groups, unmanaged instance groups, or network endpoint groups (NEGs).

A health check is a small probe the load balancer sends to each backend on a regular interval. It might be an HTTP GET to /health, a TCP connection attempt, or an SSL handshake. The backend must respond within a timeout and return an acceptable status code.

When a backend fails enough consecutive health checks, the load balancer marks it unhealthy and stops routing traffic to it. This protects users from hitting a broken server. But it also means a misconfigured health check or a missing firewall rule can make a perfectly working backend look dead.

The load balancer is doing its job correctly. The fix is almost always in the firewall, the health check config, or the application itself.

How GCP health checks work

The load balancer sends a probe to each backend at a fixed interval (default: 5 seconds). Each probe must get a valid response within the configured timeout (default: 5 seconds).

Unhealthy threshold: If a backend fails this many consecutive probes (default: 2), it is marked UNHEALTHY and removed from rotation.

Healthy threshold: After you fix the problem, the backend must pass this many consecutive probes (default: 2) before the load balancer considers it healthy again and starts sending traffic to it.

A health check failure is not the same as an application error. A health check failure means the load balancer could not reach or verify the backend at all. An application returning 500 to user requests is a different problem: the backend is reachable, but the app has a bug. The load balancer can mark a backend as healthy (health check passes) while the app still returns 5xx errors to real user traffic.

When to use this guide

Use this guide if you see any of these symptoms:

• The Google Cloud console shows backends as UNHEALTHY in your backend service or load balancer details
• Users get 502 Bad Gateway or 503 Service Unavailable from the load balancer
• A Cloud Run or serverless NEG backend is failing health checks
• Some instances in a managed instance group pass while others fail
• gcloud compute backend-services get-health returns UNHEALTHY for one or more backends
• You just set up a new HTTP load balancer and traffic is not flowing

Fast diagnosis: start here

Run this command to see the current status of every backend in your backend service:

# Global backend service (external HTTP/S load balancers)
gcloud compute backend-services get-health BACKEND_SERVICE_NAME --global

# Regional backend service (internal or regional external load balancers)
gcloud compute backend-services get-health BACKEND_SERVICE_NAME --region=REGION

Then match your situation to the right starting point:

Fix 1: Firewall rules for health checks

GCP load balancer health checks originate from Google infrastructure using these IP ranges:

• 35.191.0.0/16
• 130.211.0.0/22

Your VPC firewall must have an ingress allow rule from both ranges on the port your health check probes. Without this rule, health check traffic is silently dropped and every backend appears unhealthy.

This applies to external and internal HTTP(S) load balancers that use compute health checks. For internal TCP/UDP load balancers, you may also need to allow 35.191.0.0/16 and 209.85.152.0/22 and 209.85.204.0/22. Always verify the required ranges for your specific load balancer type in the GCP documentation.

Create the firewall rule

# Allow health check probes on specific ports
# Adjust --rules to match YOUR health check port
gcloud compute firewall-rules create allow-health-checks \
  --network=my-vpc \
  --direction=INGRESS \
  --action=ALLOW \
  --source-ranges=35.191.0.0/16,130.211.0.0/22 \
  --rules=tcp:80,tcp:443 \
  --target-tags=load-balanced-backend

Verify the rule applies to your backends

The —target-tags flag scopes this rule to VMs with the matching network tag. Your backend VMs must carry the same tag. If the tags do not match, the rule exists but does not apply to your backends.

# Check which tags are on your VM
gcloud compute instances describe VM_NAME --zone=ZONE \
  --format="value(tags.items)"

# Add the tag if missing
gcloud compute instances add-tags VM_NAME --zone=ZONE \
  --tags=load-balanced-backend

Verify the port matches

The firewall rule must allow the exact port the health check probes. If your health check is configured for port 8080 but the firewall rule only allows 80 and 443, the probes are still blocked.

# Check which port the health check probes
gcloud compute health-checks describe HEALTH_CHECK_NAME \
  --format="value(httpHealthCheck.port, httpsHealthCheck.port, tcpHealthCheck.port)"

Fix 2: Health check port, path, protocol, timeout, and thresholds

Once the firewall rule is correct, the next most common cause is a mismatch between what the health check expects and what the application actually serves.

Wrong port

If the health check probes port 80 but your app listens on 8080, the probe connects to nothing and times out. This is surprisingly common after changing application ports.

# Check the health check port
gcloud compute health-checks describe HEALTH_CHECK_NAME --format="yaml"

# Update to match your application port
gcloud compute health-checks update http HEALTH_CHECK_NAME --port=8080

Wrong path

The health check path must return a 2xx status code. A path that returns 404 (not found) or 403 (forbidden) fails the check. Double-check that the path exists and is accessible without authentication.

Redirects (301 / 302)

# Test from a VM in the same network to see what the path actually returns
curl -I http://BACKEND_IP:PORT/health

Protocol mismatch

If your app serves HTTPS on port 443 but the health check is configured as HTTP, the probe sends a plain HTTP request to an HTTPS endpoint. The response will be garbled or a connection reset. Match the health check protocol to what the app actually serves on that port.

# Switch from HTTP to HTTPS health check
gcloud compute health-checks update https HEALTH_CHECK_NAME \
  --port=443 \
  --request-path=/health

Timeout too short

If the app takes 3 seconds to respond to the health check path and the timeout is 2 seconds, the check always fails, even though the app eventually returns 200. Increase the timeout to be comfortably longer than the worst-case response time for the health endpoint.

# Increase timeout to 10 seconds
gcloud compute health-checks update http HEALTH_CHECK_NAME --timeout=10s

Unhealthy threshold too low

With —unhealthy-threshold=1, a single slow response removes the backend immediately. This causes flapping under load. Use at least 3 for production workloads to tolerate occasional slow responses.

# Set a more forgiving threshold
gcloud compute health-checks update http HEALTH_CHECK_NAME \
  --unhealthy-threshold=3 \
  --healthy-threshold=2

Healthy threshold delay after a fix

After you fix the problem, the backend does not become healthy instantly. It must pass the healthy threshold number of consecutive checks. With a healthy threshold of 2 and a check interval of 5 seconds, expect 10 to 15 seconds. If you raised the healthy threshold, it takes proportionally longer.

Fix 3: Backend service, instance group, and NEG mismatches

Managed instance groups

Managed instance groups (MIGs) run identical VMs from an instance template. If the template is wrong (wrong startup script, wrong container image, wrong port), every instance in the group fails health checks.

• Verify the instance template specifies the correct container image or startup script
• Check that the template includes the correct network tag for the health check firewall rule
• Confirm the app in the template listens on the port the health check probes
• If you updated the template, make sure you rolled out new instances. Old instances still use the old template.

Unmanaged instance groups

Unmanaged instance groups contain manually added VMs that may have different configurations. If some VMs are unhealthy, check each one individually. Common causes: app not running, wrong port, missing network tag, or the VM is in a subnet with different firewall rules.

Zonal NEGs

Zonal network endpoint groups contain specific IP:port pairs. If the NEG references the wrong IP or port, health checks fail. Verify that each endpoint matches a running backend.

# List endpoints in a zonal NEG
gcloud compute network-endpoint-groups list-network-endpoints NEG_NAME \
  --zone=ZONE

Serverless NEGs and Cloud Run

Cloud Run backends use serverless NEGs. GCP manages health checking internally for these, so the troubleshooting is different:

• Verify the serverless NEG points to the correct Cloud Run service name and region
• Check that the Cloud Run service is deployed and not in a failed state
• Look at Cloud Run logs for container startup failures. See the Cloud Run Container Failed to Start guide.
• Confirm the Cloud Run service is in the same region as the serverless NEG
• Check that the Cloud Run service accepts unauthenticated requests if the load balancer does not add authentication headers

# Describe a serverless NEG
gcloud compute network-endpoint-groups describe NEG_NAME \
  --region=REGION

# Check Cloud Run service status
gcloud run services describe SERVICE_NAME --region=REGION \
  --format="value(status.conditions)"

Wrong backend attached or wrong region

A backend service can have backends in multiple regions. If you attached the wrong instance group or NEG, or attached one from the wrong region, the health check targets the wrong servers. Verify the attached backends:

gcloud compute backend-services describe BACKEND_SERVICE_NAME \
  --global \
  --format="yaml(backends)"

Application listening on localhost only

Backend unhealthy vs 502 vs 503

These look similar but have different causes and different fixes:

Use Logs Explorer to see exactly which status codes the load balancer is returning and which backend handled each request.

Step-by-step commands to verify backend health

Inspect backend health

# Global backend service
gcloud compute backend-services get-health BACKEND_SERVICE_NAME --global

# Regional backend service
gcloud compute backend-services get-health BACKEND_SERVICE_NAME --region=REGION

Inspect health check configuration

# List all health checks
gcloud compute health-checks list

# Describe a specific health check
gcloud compute health-checks describe HEALTH_CHECK_NAME --format="yaml"

Inspect backend service configuration

# See which backends are attached and which health check is used
gcloud compute backend-services describe BACKEND_SERVICE_NAME \
  --global \
  --format="yaml(backends, healthChecks)"

Inspect NEG configuration

# List all NEGs
gcloud compute network-endpoint-groups list

# Describe a specific NEG
gcloud compute network-endpoint-groups describe NEG_NAME --zone=ZONE

# For serverless NEGs
gcloud compute network-endpoint-groups describe NEG_NAME --region=REGION

Inspect relevant logs

Use Logs Explorer or the gcloud CLI to check load balancer logs and application logs:

# Load balancer access logs, filtered for 502 and 503 errors
gcloud logging read \
  'resource.type="http_load_balancer" AND (httpRequest.status=502 OR httpRequest.status=503)' \
  --project=PROJECT_ID \
  --limit=20 \
  --format="table(timestamp, httpRequest.requestUrl, httpRequest.status)"

# Application logs from a specific instance
gcloud logging read \
  'resource.type="gce_instance" AND resource.labels.instance_id="INSTANCE_ID"' \
  --project=PROJECT_ID \
  --limit=30

Verify app response directly from a VM in the same network

# SSH into a VM in the same VPC and test the health check path
# This tells you whether the app itself is responding correctly
curl -v http://BACKEND_INTERNAL_IP:PORT/healthz

# Check what the response headers and status code are
curl -I http://BACKEND_INTERNAL_IP:PORT/healthz

For deeper network-level troubleshooting, use VPC Connectivity Tests to trace the exact path health check traffic takes through your network and identify where it gets blocked.