Troubleshooting GCP VPC Connectivity: Fix Firewall, Route, and Peering Failures

When to use this guide

Use this page when you see any of these symptoms in GCP:

• A VM cannot reach another VM in the same VPC or a peered VPC
• A private VM cannot call Google APIs (Cloud Storage, BigQuery, Pub/Sub)
• A private VM cannot reach the public internet for package updates or external API calls
• VPC peering is configured but connections time out between the two networks
• A service works from one VM but not from another, with no obvious difference in configuration
• Connectivity was working and stopped after a firewall rule or route change
• DNS resolves correctly but TCP connections still fail

If your problem is specifically DNS resolution (hostnames not resolving, NXDOMAIN errors), start with the DNS Resolution Problems guide instead. If a load balancer health check is failing, see Load Balancer Backend Unhealthy. This page covers IP-level connectivity failures.

Simple explanation

Every packet sent from a GCP VM passes through two checkpoints before it reaches its destination. First, the routing table decides where the packet should go by picking the next hop. Second, the firewall rules decide whether the packet is allowed to proceed. If either checkpoint fails, the packet is dropped and the connection times out silently.

Most VPC connectivity failures come down to one of these five causes:

1. A firewall rule is blocking ingress traffic (the most common cause)
2. No route exists to the destination network
3. Private Google Access is not enabled on the subnet (for Google API calls)
4. Cloud NAT is missing or misconfigured (for outbound internet from private VMs)
5. VPC peering is incomplete or firewall rules were not updated after peering

How GCP connectivity failures happen

GCP evaluates connectivity at multiple layers. Understanding the order helps you diagnose faster because each layer produces different symptoms.

Layer 1: Routing

The VPC route table maps destination IP ranges to next hops. Every VPC starts with a default route (0.0.0.0/0) pointing to the internet gateway and subnet routes for each subnet. If you delete the default route or a custom route with higher priority intercepts traffic, packets aimed at external destinations have nowhere to go.

Layer 2: Firewall rules

After routing determines the path, firewall rules decide whether the packet is allowed. GCP denies all ingress by default (implicit deny at priority 65535). Custom VPCs start with no allow rules at all. The default VPC has pre-populated allow rules for SSH, ICMP, and internal traffic, but custom VPCs do not.

Layer 3: Network features

Features like Private Google Access, Cloud NAT, and VPC peering add connectivity that does not exist by default. A private VM cannot reach Google APIs without Private Google Access enabled. A private VM cannot reach the internet without Cloud NAT. Two VPCs cannot communicate without peering configured on both sides.

Layer 4: OS-level and application

If the GCP network layer allows the traffic, the packet still needs to pass the operating system firewall (iptables, ufw, Windows Firewall) and reach a listening application on the correct port. A service that is not running or is bound to 127.0.0.1 instead of 0.0.0.0 will reject connections even when GCP networking is correct.

Quick triage: symptom to likely cause

Start here. Find the symptom that matches your situation, then jump to the relevant section.

Connectivity Tests: your first debugging step

Before reading firewall rules or route tables manually, run a Connectivity Test. It simulates the network path between a source and destination and reports the result at each hop: forwarded, blocked by firewall, or dropped by a route. It names the specific rule responsible, which makes the fix obvious.

# Run a connectivity test from one VM to another on port 443
gcloud network-management connectivity-tests create my-test \
  --source-ip=10.0.0.2 \
  --destination-ip=10.0.0.3 \
  --destination-port=443 \
  --protocol=TCP \
  --project=PROJECT_ID

# View the test results
gcloud network-management connectivity-tests describe my-test

# List all tests
gcloud network-management connectivity-tests list

You can also run Connectivity Tests from the GCP Console under Network Intelligence Center. The console version gives a visual trace of the path with colour-coded results at each hop.

How to read the results

• ”Blocked by firewall rule [rule-name]” means the named rule is dropping the traffic. Check the rule’s priority, targets, and source ranges.
• ”Blocked by implicit deny” means no explicit rule allows the traffic. You need to create an allow rule.
• ”No route to destination” means the VPC has no route matching the destination IP. Add a route or check for deleted/overridden routes.
• ”Forwarded” at every hop means the network path is clear. The problem is likely at the OS or application level on the destination VM.

Clean up tests after you are done to avoid clutter:

# Delete a connectivity test
gcloud network-management connectivity-tests delete my-test

VM cannot reach another VM in the same VPC

This is the most common VPC connectivity problem. The cause is almost always a missing ingress firewall rule. Custom VPCs do not have the default-allow-internal rule that the default VPC ships with. Without it, VMs in the same VPC cannot communicate.

What to check first

# List firewall rules for the VPC, sorted by priority
gcloud compute firewall-rules list \
  --filter="network:my-vpc" \
  --sort-by=priority \
  --format="table(name,direction,priority,sourceRanges,targetTags,allowed)"

# Check what network tags the failing VM has
gcloud compute instances describe my-vm \
  --zone=us-central1-a \
  --format="value(tags.items)"

# Check the VM's service account (used for service-account-based rules)
gcloud compute instances describe my-vm \
  --zone=us-central1-a \
  --format="value(serviceAccounts[].email)"

What to look for

• An ingress allow rule covering the internal CIDR range (for example, 10.0.0.0/8) on the required port and protocol
• If the rule uses network tags, confirm the destination VM has the matching tag
• If a deny rule exists with a lower priority number, it may override the allow rule
• Check that both VMs are in the same VPC. VMs in different VPCs need VPC peering

How to fix it

# Create an allow-internal rule for a custom VPC
gcloud compute firewall-rules create allow-internal \
  --network=my-vpc \
  --direction=INGRESS \
  --action=ALLOW \
  --rules=tcp:0-65535,udp:0-65535,icmp \
  --source-ranges=10.0.0.0/8

Confirm the fix

# Test connectivity with ping (ICMP)
ping -c 3 10.0.0.3

# Test a specific TCP port
nc -zv 10.0.0.3 443

# Or run another Connectivity Test
gcloud network-management connectivity-tests create verify-fix \
  --source-ip=10.0.0.2 \
  --destination-ip=10.0.0.3 \
  --destination-port=443 \
  --protocol=TCP \
  --project=PROJECT_ID

Private VM cannot reach Google APIs

A VM with only a private IP cannot reach Google APIs (Cloud Storage, BigQuery, Pub/Sub, Container Registry) unless Private Google Access is enabled on the VM’s subnet. Without it, API calls from gsutil, bq, or client libraries time out or return connection errors.

What to check first

# Check whether Private Google Access is enabled on the subnet
gcloud compute networks subnets describe my-subnet \
  --region=us-central1 \
  --format="value(privateIpGoogleAccess)"
# Returns True or False

# Verify the VM has no external IP
gcloud compute instances describe my-vm \
  --zone=us-central1-a \
  --format="value(networkInterfaces[].accessConfigs[].natIP)"
# Empty output = no external IP

How to fix it

# Enable Private Google Access on the subnet
gcloud compute networks subnets update my-subnet \
  --region=us-central1 \
  --enable-private-ip-google-access

Confirm the fix

# From the VM, test access to a Google API
gsutil ls gs://my-bucket

# Or test the API endpoint directly
curl -s -o /dev/null -w "%{http_code}" https://storage.googleapis.com

Private VM cannot reach the public internet

A VM with no external IP needs Cloud NAT to initiate outbound connections to the internet. If apt update, pip install, or curl to an external URL times out, Cloud NAT is likely missing or misconfigured.

What to check first

# List Cloud Routers (Cloud NAT requires one)
gcloud compute routers list \
  --filter="network:my-vpc" \
  --format="table(name,region,network)"

# List Cloud NAT gateways attached to a router
gcloud compute routers nats list \
  --router=my-router \
  --region=us-central1

# Check that the default internet route exists
gcloud compute routes list \
  --filter="destRange:0.0.0.0/0 AND network:my-vpc"

Common causes

• No Cloud NAT gateway exists. Create one on a Cloud Router in the same region and VPC as the VM.
• Wrong region. Cloud NAT is regional. A gateway in europe-west1 does not serve VMs in us-central1.
• Subnet not covered. If the gateway was created with —nat-custom-subnet-ip-ranges, the VM’s subnet may not be included.
• Default route deleted. Cloud NAT depends on the 0.0.0.0/0 default route via the internet gateway. If it was removed during a network hardening exercise, Cloud NAT has nowhere to send traffic.
• Custom route override. A custom route with a higher priority (lower number) that matches 0.0.0.0/0 can redirect traffic away from the internet gateway to a VPN or appliance.
• Egress firewall rule. If your VPC has a deny-all egress rule, it blocks outbound traffic before Cloud NAT can process it. Add an explicit allow-egress rule.

How to fix it

# Create a Cloud Router (if one does not exist)
gcloud compute routers create my-nat-router \
  --network=my-vpc \
  --region=us-central1

# Create a Cloud NAT gateway covering all subnets
gcloud compute routers nats create my-nat-gateway \
  --router=my-nat-router \
  --region=us-central1 \
  --nat-all-subnet-ip-ranges \
  --auto-allocate-nat-external-ips

Confirm the fix

# From the VM, test outbound internet access
curl -s -o /dev/null -w "%{http_code}" https://www.google.com

# Or try a package update
sudo apt update

VPC peering connectivity fails

VPC peering connects two VPCs for private IP communication, but getting it to work involves two independent steps that both must succeed: the peering itself and the firewall rules in both directions.

What to check first

# Check peering status (must be ACTIVE on both sides)
gcloud compute networks peerings list --network=my-vpc

# Check that routes from the peer are visible
gcloud compute routes list \
  --filter="network:my-vpc AND nextHopPeering:*"

# Check firewall rules in the destination VPC
gcloud compute firewall-rules list \
  --filter="network:peer-vpc" \
  --sort-by=priority

Common causes

• Peering INACTIVE. Both sides must create the peering connection. If only one side exists, the status stays INACTIVE. Check the other project.
• Overlapping CIDR ranges. Two VPCs with overlapping subnet CIDRs cannot be peered. GCP rejects the peering request.
• Firewall rules not updated. Peering exchanges routes but does not modify firewall rules. Traffic from the peered VPC arrives with source IPs from the peer’s subnet range. You need ingress allow rules in the destination VPC that cover those ranges.
• Non-transitive routing. If VPC A peers with VPC B and VPC B peers with VPC C, traffic from A cannot reach C through B. A direct peering between A and C is required.

How to fix it

# If peering is INACTIVE, create the missing side
gcloud compute networks peerings create peer-b-to-a \
  --network=network-b \
  --peer-project=project-a \
  --peer-network=network-a \
  --auto-create-routes \
  --project=project-b

# Add a firewall rule to allow traffic from the peered VPC
gcloud compute firewall-rules create allow-peered-traffic \
  --network=my-vpc \
  --direction=INGRESS \
  --action=ALLOW \
  --rules=tcp:0-65535,udp:0-65535,icmp \
  --source-ranges=10.20.0.0/16 \
  --description="Allow ingress from peered VPC subnet range"

Confirm the fix

# Run a Connectivity Test across the peering
gcloud network-management connectivity-tests create peering-test \
  --source-ip=10.0.0.2 \
  --destination-ip=10.20.0.5 \
  --destination-port=443 \
  --protocol=TCP \
  --project=PROJECT_ID

Route vs firewall confusion

Routes and firewall rules are independent systems that both affect connectivity. A common source of confusion is assuming that one implies the other.

• A route exists but traffic is blocked. The route delivers the packet to the right place, but a firewall rule denies it at the destination. The route is working correctly. Fix the firewall.
• A firewall rule allows the traffic but the packet never arrives. No route matches the destination IP, so the packet is dropped before the firewall is even evaluated. Add the missing route.
• A custom route overrides the default route. A route with a higher priority (lower number) matching 0.0.0.0/0 can redirect all internet-bound traffic to a VPN gateway or firewall appliance. This is intentional in some architectures but breaks internet access if unexpected.

# List all routes in the VPC, sorted by priority
gcloud compute routes list \
  --filter="network:my-vpc" \
  --format="table(name,destRange,nextHopGateway,nextHopIp,nextHopInstance,priority)"

# Check for competing routes to the same destination
gcloud compute routes list \
  --filter="destRange:0.0.0.0/0 AND network:my-vpc" \
  --sort-by=priority

If Connectivity Tests says “forwarded” at every hop but the connection still fails, the problem is not in the GCP network layer. Check the OS-level firewall on the destination VM and verify the application is listening on the correct port and interface.

DNS-related connectivity symptoms

Some connectivity failures look like firewall or route problems but are actually DNS. If a hostname does not resolve, the application cannot connect. The error often appears as a generic timeout rather than a clear DNS failure, which makes it easy to misdiagnose.

Quick DNS check

# Test DNS resolution from the VM
nslookup my-service.internal

# Compare: can you reach the IP directly?
curl -v http://10.0.0.5:8080
# If the IP works but the hostname does not, the problem is DNS

Common DNS-related causes of connectivity failure

• Private DNS zone not associated with the VPC. A private Cloud DNS zone only resolves within VPCs explicitly associated with it. If the zone is missing the VM’s VPC, queries return NXDOMAIN.
• VPC peering does not share DNS. Peering exchanges routes, not DNS visibility. VMs in a peered VPC cannot resolve the other VPC’s private DNS names without additional DNS peering or zone association.
• Stale DNS cache. After changing a DNS record, clients continue resolving the old IP until the TTL expires. The old IP may point to a VM that no longer exists or a service that has moved.

For full DNS troubleshooting steps, see DNS Resolution Problems.

Firewall rule debugging reference

The commands in this section are useful for any connectivity investigation where you suspect a firewall rule is the cause.

# List all firewall rules sorted by priority
gcloud compute firewall-rules list \
  --sort-by=priority \
  --format="table(name,direction,priority,sourceRanges,targetTags,allowed)"

# List rules that apply to a specific VPC
gcloud compute firewall-rules list \
  --filter="network:my-vpc" \
  --sort-by=priority

# Describe a specific rule in detail
gcloud compute firewall-rules describe my-rule

# Check what network tags a VM has
gcloud compute instances describe my-vm \
  --zone=us-central1-a \
  --format="value(tags.items)"

# Check the VM's service account
gcloud compute instances describe my-vm \
  --zone=us-central1-a \
  --format="value(serviceAccounts[].email)"

Key things to verify when reading firewall rules:

• Priority order. Lower number = higher precedence. A deny at priority 500 overrides an allow at priority 1000.
• Target matching. A rule targeting tag:backend-vm only applies to VMs with that tag. If the VM does not have the tag, the rule has no effect on it.
• Direction. Ingress rules control inbound traffic. Egress rules control outbound traffic. Check the correct direction for your scenario.
• Implicit deny. If no explicit rule matches, the implied deny-all-ingress at priority 65535 applies.

Private Google Access vs Cloud NAT

These two features solve different problems for private VMs, but they are frequently confused. Understanding the difference saves you from enabling the wrong one.

Rule of thumb: enable Private Google Access on every subnet with private VMs. Add Cloud NAT only when those VMs also need to reach non-Google destinations on the internet.

VPC Peering vs Shared VPC

Both connect multiple GCP projects for private networking, but they work differently. Choosing the wrong model leads to connectivity problems that are hard to fix later.

Use VPC peering when networks are independently managed or span different organisations. Use Shared VPC when a central team should control the network for many projects within the same organisation.

Troubleshooting checklist

Run through this list in order when diagnosing any VPC connectivity failure:

1. Run a Connectivity Test between the source and destination
2. If blocked by firewall: identify the rule, check priority, targets, and source ranges
3. If no route: check the VPC route table for the destination prefix
4. If reaching Google APIs from a private VM: check Private Google Access on the subnet
5. If reaching the internet from a private VM: verify Cloud NAT exists in the same region
6. If crossing a VPC peering: confirm peering is ACTIVE on both sides and firewall rules allow the peer’s CIDR
7. If hostname does not resolve: switch to the DNS troubleshooting guide
8. If the network path is clear: check the OS firewall and verify the service is listening