Load Balancer Backend Unhealthy in Azure

How Load Balancer health probes work

Azure Standard Load Balancer sends periodic probes to each backend VM to determine if it can receive traffic. Two probe types are available:

• TCP probe: opens a TCP connection to the specified port. If the connection succeeds (the three-way handshake completes), the backend is healthy. The application does not need to send any data.
• HTTP/HTTPS probe: sends an HTTP GET request to the specified path and port. The backend is healthy only if it returns HTTP 200. Any other response code — including 301, 404, or 500 — marks it as unhealthy.

All probes originate from the special Azure platform IP 168.63.129.16. This IP is not routable and cannot be blocked by route tables, but it can be blocked by NSG rules. NSGs are the most common cause of probe failures.

Check current probe configuration on a load balancer:

az network lb probe list \
  --resource-group myResourceGroup \
  --lb-name myLoadBalancer \
  --output table

Sample output:

IntervalInSeconds    Name         NumberOfProbes    Port    Protocol
-------------------  -----------  ----------------  ------  ----------
15                   httpProbe    2                 80      Http

View backend pool members:

az network lb address-pool list \
  --resource-group myResourceGroup \
  --lb-name myLoadBalancer \
  --output json

Cause 1: NSG blocking the health probe source IP

The most common cause of all backends showing as unhealthy simultaneously is an NSG rule that blocks inbound traffic from 168.63.129.16. This happens when an engineer adds a hardening rule like DenyAllInbound without including an exception for the Azure platform IP.

Check the NSG rules on the backend subnet or NIC:

az network nsg rule list \
  --resource-group myResourceGroup \
  --nsg-name myBackendNSG \
  --output table

Look for any Deny rule with a lower priority number (higher precedence) that could block the probe traffic. Add an explicit allow rule for the health probe source:

az network nsg rule create \
  --resource-group myResourceGroup \
  --nsg-name myBackendNSG \
  --name AllowAzureLoadBalancerInBound \
  --priority 100 \
  --protocol Tcp \
  --direction Inbound \
  --source-address-prefix AzureLoadBalancer \
  --source-port-range "*" \
  --destination-address-prefix "*" \
  --destination-port-range "*" \
  --access Allow

The service tag AzureLoadBalancer matches all traffic from 168.63.129.16 including health probes and diagnostics. Using the service tag is cleaner than hardcoding the IP and future-proofs the rule if Azure adds additional platform IPs.

Cause 2: Application not listening on the probe port

Even with the NSG correctly configured, health probes fail if the application is not listening on the port the probe targets. This happens after deployments where the application crashes on startup, or when the probe port in the load balancer configuration does not match the actual application port.

Verify what the application is listening on from inside the VM:

# On the backend VM (Linux)
sudo ss -tlnp | grep LISTEN

# Expected output for an app on port 8080:
# tcp  0  0 0.0.0.0:8080  0.0.0.0:*  LISTEN  12345/node

If the port is not in the LISTEN state, the application has crashed or not started. Check application logs:

sudo systemctl status myapp.service
sudo journalctl -u myapp.service -n 50 --no-pager

Simulate the exact HTTP health probe request:

# For an HTTP probe on port 8080 at path /health
curl -v -o /dev/null -w "%{http_code}" http://127.0.0.1:8080/health

The response code must be exactly 200 for the probe to succeed. If the application returns 301 (redirect to HTTPS), configure the probe to use HTTPS or add a dedicated health endpoint that returns 200 directly without a redirect.

Cause 3: Single backend VM unhealthy — VM stopped or application crashed

If only one VM in the backend pool is unhealthy while others are healthy, the issue is specific to that VM. Check the VM power state first:

az vm get-instance-view \
  --resource-group myResourceGroup \
  --name myBackendVM2 \
  --query "instanceView.statuses[?starts_with(code, 'PowerState/')].displayStatus" \
  --output tsv

If the VM shows “VM deallocated” or “VM stopped”, start it:

az vm start \
  --resource-group myResourceGroup \
  --name myBackendVM2

After the VM starts, the load balancer health probe retries automatically within the configured probe interval (default 15 seconds). The backend becomes healthy after the configured number of consecutive successful probes (default 2 probes = 30 seconds minimum wait time).

Application Gateway unhealthy backends

Application Gateway has a more sophisticated health check than Standard Load Balancer. View the detailed backend health status:

az network application-gateway show-backend-health \
  --resource-group myResourceGroup \
  --name myAppGateway \
  --output json

The output shows each backend server with a detailed status message:

{
  "backendAddressPools": [
    {
      "backendHttpSettingsCollection": [
        {
          "servers": [
            {
              "address": "10.1.0.4",
              "health": "Unhealthy",
              "healthProbeLog": "Connect error or unhealthy status code was received",
              "ipConfiguration": {}
            }
          ]
        }
      ]
    }
  ]
}

Host header problem: Application Gateway health probes include a Host header. By default it is set to the backend IP address. If the application uses virtual hosting and rejects requests where the Host header is an IP address, the probe will get a non-200 response.

Fix by configuring the health probe to send the correct hostname:

az network application-gateway probe create \
  --resource-group myResourceGroup \
  --gateway-name myAppGateway \
  --name myCustomProbe \
  --protocol Http \
  --host "myapp.example.com" \
  --path "/health" \
  --interval 30 \
  --timeout 30 \
  --threshold 3 \
  --port 80

az network application-gateway http-settings update \
  --resource-group myResourceGroup \
  --gateway-name myAppGateway \
  --name myHTTPSettings \
  --probe myCustomProbe

WAF blocking the health probe: If Application Gateway has WAF enabled, the WAF rules may match the probe request and return 403 instead of passing it through. Check the WAF firewall logs in Azure Monitor for blocked requests matching the probe path, then add a WAF exclusion rule for the health probe URI.

Alerting on backend health metrics

Set up a metric alert to detect backend health degradation before users report it:

LB_ID=$(az network lb show \
  --resource-group myResourceGroup \
  --name myLoadBalancer \
  --query id -o tsv)

az monitor metrics alert create \
  --name "LBUnhealthyBackendAlert" \
  --resource-group myResourceGroup \
  --scopes "$LB_ID" \
  --condition "avg VipAvailability < 100" \
  --window-size 5m \
  --evaluation-frequency 1m \
  --severity 2 \
  --description "Load balancer VIP availability dropped — backend may be unhealthy"

The VipAvailability metric drops to 0 when all backends are unhealthy. The DipAvailability metric tracks individual backend health. Use DipAvailability split by BackendIPAddress to identify which specific backend is failing when multiple instances are in the pool.