GCP Pub/Sub Delivery Failures: Backlogs, Push Errors, Dead Letters

How Pub/Sub delivery works in plain English

When Pub/Sub is healthy, the flow looks like this: your application publishes a message to a topic. Every subscription attached to that topic gets its own copy of the message. If the subscription uses pull, the subscriber fetches the message and acknowledges it. If the subscription uses push, Pub/Sub sends the message via HTTP POST to an endpoint, and a 2xx response counts as acknowledgement. Either way, once acknowledged, the message is done. For a deeper comparison of these delivery modes, see Pub/Sub Push vs Pull.

A backlog is the set of messages that have been published but not yet acknowledged. A small backlog is normal. A backlog that keeps growing means your subscriber cannot keep up, is not running, or is failing to acknowledge messages.

Pull failures happen when the subscriber is not pulling, not acknowledging, or losing the ack race against the deadline. Push failures happen when the HTTP endpoint returns an error, is unreachable, or rejects the request due to authentication. The symptoms and fixes are different for each.

A dead-letter topic is a safety net. After a message fails delivery approximately a configured number of times, Pub/Sub forwards it to the dead-letter topic instead of retrying forever. Without one, messages that can never be processed expire silently after the retention window.

The delivery pipeline at a glance

Every Pub/Sub message follows this path. When delivery fails, exactly one stage in this pipeline is broken:

1. Publisher sends a message to a topic
2. The topic fans out the message to every attached subscription
3. The subscription delivers to the subscriber (pull) or push endpoint (push)
4. The subscriber processes the message and sends an acknowledgement
5. If acknowledgement fails repeatedly, the message goes to the dead-letter topic (if configured) or stays in the backlog until retention expires

Understanding this pipeline helps you narrow the problem. A zero publish count means step 1 is broken. A growing backlog with active publishing means steps 3–5 are broken. The Pub/Sub Messaging Model page explains the fan-out and ordering mechanics in detail.

Use this page when…

• Your subscription backlog keeps growing and never drains
• Your pull subscriber appears healthy but messages repeat endlessly
• Your push endpoint returns 403, 404, or 500 errors
• Messages are landing in your dead-letter topic
• Published messages never appear in the subscription at all
• The oldest unacked message age keeps rising
• Your subscriber processes messages but the backlog does not shrink
• A Cloud Function or Cloud Run service triggered by Pub/Sub is not receiving events

Fast triage checklist

Use this table to jump straight to the most likely cause based on what you are seeing:

Check monitoring first

Before changing any configuration, check Cloud Monitoring metrics. These tell you exactly where the pipeline is broken:

# List all available Pub/Sub metrics for your project
gcloud monitoring metric-descriptors list \
  --filter="metric.type:pubsub.googleapis.com" \
  --format="value(type)"

Key metrics and what they tell you

• topic/send_message_operation_count. How many messages your publisher is sending. Zero means no messages are being published and the problem is on the publisher side, not the subscriber side.

• subscription/num_undelivered_messages. The current backlog size. A number that keeps growing means subscribers are falling behind or not running.

• subscription/oldest_unacked_message_age. The age (in seconds) of the oldest unacknowledged message. A high and rising number means the subscriber is stalled. A stable low number means delivery is healthy.

• subscription/push_request_count. Push delivery attempts broken down by HTTP response code. High counts at non-2xx codes mean your push endpoint is rejecting messages.

• subscription/pull_request_count. Pull requests from subscribers. Zero means no subscriber is pulling.

• subscription/dead_letter_message_count. Messages forwarded to the dead-letter topic. A growing count means your subscriber consistently fails to process certain messages.

You can set up alerting policies on these metrics so you catch delivery failures before the backlog becomes critical. For custom counters based on log patterns, see log-based metrics.

# Quick check: see the current backlog and oldest message age
gcloud pubsub subscriptions describe my-subscription \
  --format="table(name,numUndeliveredMessages,messageRetentionDuration)"

# Check push delivery response codes in the last hour
gcloud monitoring time-series list \
  --filter='metric.type="pubsub.googleapis.com/subscription/push_request_count"
    AND resource.labels.subscription_id="my-push-subscription"' \
  --interval-start-time=$(date -u -d '-1 hour' +%Y-%m-%dT%H:%M:%SZ) \
  --format="table(metric.labels.response_code, points.value.int64Value)"

Publisher-side delivery failures

If topic/send_message_operation_count is zero or dropping, messages are not being published. The problem is in your publishing application, not Pub/Sub.

Messages not being published at all

Check your publisher application logs first. Common causes:

• Publisher client not initialised or not connected. Verify the Pub/Sub client library is creating a publisher successfully and that the topic name is correct. A typo in the topic name causes a NOT_FOUND error on every publish.

• Authentication failure. The service account running the publisher needs roles/pubsub.publisher on the topic. Missing credentials or wrong service account produce PERMISSION_DENIED errors. See troubleshooting authentication errors for credential diagnostics.

• Pub/Sub API not enabled. The Pub/Sub API must be enabled in the project. A PERMISSION_DENIED or SERVICE_DISABLED error on publish calls means the API is off.

# Check if the Pub/Sub API is enabled
gcloud services list --enabled --filter="name:pubsub.googleapis.com"

# Verify the topic exists
gcloud pubsub topics describe my-topic

# Check IAM policy on the topic
gcloud pubsub topics get-iam-policy my-topic

Publisher timeouts and throughput issues

• DEADLINE_EXCEEDED errors. The publish call timed out before Pub/Sub confirmed receipt. This can happen under network pressure, high message volume, or when the publisher machine is CPU/memory constrained. Increase the publish timeout in your client configuration or reduce batch size.

• Quota limits. Pub/Sub has per-project quotas on publish throughput. If you hit them, publish calls return RESOURCE_EXHAUSTED. Check your quota usage in the Cloud Console under IAM & Admin > Quotas.

• Large messages. Pub/Sub has a 10 MB maximum message size. Messages exceeding this limit are rejected. If your payloads are large, store the data in Cloud Storage and publish a reference instead.

How to confirm publish is working

# Publish a test message and confirm it succeeds
gcloud pubsub topics publish my-topic \
  --message="test message $(date -u +%Y-%m-%dT%H:%M:%SZ)"

# If this succeeds, your topic is fine and the problem is in your publisher code
# If this fails, read the error message — it tells you what is wrong

Pull subscription failures

Pull subscriptions require the subscriber to actively fetch messages. If nothing is pulling, messages accumulate in the backlog until the retention window expires (default 7 days).

No active subscriber

The most basic pull failure: no process is pulling from the subscription. Check whether your subscriber application, container, or Cloud Function is actually running. A crashed pod, a stopped VM, or an undeployed function means zero pull activity and a growing backlog.

# Manually pull messages to confirm the subscription works
gcloud pubsub subscriptions pull my-subscription \
  --limit=5 \
  --auto-ack

# If messages come back, the subscription is fine — your subscriber is not pulling
# Pull without acknowledging (messages redeliver after ack deadline)
gcloud pubsub subscriptions pull my-subscription \
  --limit=5

Subscriber not acknowledging messages

Your subscriber pulls messages and processes them, but never sends an acknowledgement. Pub/Sub redelivers unacknowledged messages after the ack deadline expires. This creates a loop: the same messages arrive again and again, the backlog never drains, and you see duplicate processing in your logs.

Fix: ensure every code path that successfully processes a message calls ack(). Check error handling paths too. A caught exception that skips the ack causes silent redelivery.

Ack deadline too short

If your processing involves database writes, HTTP calls to slow APIs, or heavy computation, the deadline expires before processing finishes. Pub/Sub assumes the message was not processed and redelivers it.

# Check the current ack deadline
gcloud pubsub subscriptions describe my-subscription \
  --format="value(ackDeadlineSeconds)"

# Increase the ack deadline to 60 seconds
gcloud pubsub subscriptions update my-subscription \
  --ack-deadline=60

For variable processing times, use modifyAckDeadline in your client code to extend the deadline dynamically while processing is still in progress.

Slow processing causing backlog growth

If messages arrive faster than your subscriber can process them, the backlog grows even though everything is technically working. Solutions: increase the number of subscriber instances, increase concurrency within each subscriber, or optimise processing time.

Missing IAM permissions

The service account running your subscriber needs roles/pubsub.subscriber on the subscription. Without it, every pull request fails with PERMISSION_DENIED.

# Check who has subscriber access
gcloud pubsub subscriptions get-iam-policy my-subscription

# Grant subscriber role to your service account
gcloud pubsub subscriptions add-iam-policy-binding my-subscription \
  --member="serviceAccount:my-sa@my-project.iam.gserviceaccount.com" \
  --role="roles/pubsub.subscriber"

Push subscription failures

Push subscriptions deliver messages via HTTP POST to an endpoint URL. Pub/Sub retries with exponential backoff on any non-success response. If the endpoint consistently returns errors, the backlog grows and push_request_count shows high error rates.

Understanding push response codes

Pub/Sub treats responses 200, 201, 202, 204, and 102 as successful delivery. Everything else is a failure that triggers a retry:

• 403 Forbidden: almost always an IAM issue. The Pub/Sub service agent does not have permission to invoke the endpoint. This is the most common push failure for private Cloud Run and Cloud Functions endpoints.

• 404 Not Found: the endpoint URL is wrong, the service is not deployed, or the route does not exist. Verify the URL and confirm the service is running.

• 401 Unauthorized: push authentication is configured but the endpoint is rejecting the JWT token. Check that the audience in the push config matches what your endpoint expects.

• 5xx Server Error: your endpoint application is crashing. Check the endpoint logs (not Pub/Sub logs) to find the application error. Use Logs Explorer filtered to your Cloud Run or Cloud Functions service.

• Timeout / unreachable: the endpoint did not respond within the push timeout, or Pub/Sub cannot reach it. Check firewall rules, VPC configuration, and whether the endpoint is publicly accessible (or correctly configured for internal access).

# Check the push endpoint URL and auth configuration
gcloud pubsub subscriptions describe my-push-subscription \
  --format="yaml(pushConfig)"

# Update the push endpoint URL
gcloud pubsub subscriptions modify-push-config my-push-subscription \
  --push-endpoint=https://my-service-abc123-uc.a.run.app/pubsub

Push authentication and JWT issues

When push authentication is enabled, Pub/Sub includes a signed JWT in the Authorization header. Your endpoint must validate this token. Common problems:

• The audience in the push config does not match what the endpoint checks
• The service account specified in the push config does not exist or is disabled
• The endpoint validates the token issuer but uses the wrong expected value

Private Cloud Run and Cloud Functions endpoints

This is the single most common push delivery failure. If your Cloud Run service or Cloud Function requires authentication (the default), the Pub/Sub service agent must have roles/run.invoker (for Cloud Run) or roles/cloudfunctions.invoker (for Cloud Functions) granted on the service. Without it, every push returns 403. For more on how Cloud Run authentication works, see the Cloud Run Security Model guide.

# Find your project number (needed for the service agent email)
gcloud projects describe PROJECT_ID --format="value(projectNumber)"

# Grant Pub/Sub permission to invoke a private Cloud Run service
gcloud run services add-iam-policy-binding my-cloud-run-service \
  --region=us-central1 \
  --member="serviceAccount:service-PROJECT_NUMBER@gcp-sa-pubsub.iam.gserviceaccount.com" \
  --role="roles/run.invoker"

# Grant Pub/Sub permission to invoke a Cloud Function (gen2)
gcloud functions add-invoker-policy-binding my-function \
  --region=us-central1 \
  --member="serviceAccount:service-PROJECT_NUMBER@gcp-sa-pubsub.iam.gserviceaccount.com"

Firewall and endpoint reachability

Push delivery comes from Google-owned IP ranges. If your endpoint is behind a firewall or in a VPC with restrictive ingress rules, Pub/Sub cannot reach it. For Cloud Run and Cloud Functions with default settings, this is not an issue. For custom endpoints behind a load balancer or on a VM, ensure the firewall allows HTTPS traffic from Google’s IP ranges.

Reading push failure patterns from logs

Pub/Sub itself does not log individual push attempts in Cloud Logging by default. To diagnose push failures:

1. Check push_request_count metric grouped by response_code to see the pattern
2. Check the endpoint service logs (Cloud Run, Cloud Functions, or your own server) for the corresponding errors
3. If your endpoint is a Cloud Function triggered by Pub/Sub, see debugging Cloud Functions failures for log query guidance
4. Enable audit logging for Pub/Sub if you need to trace individual delivery attempts

Dead-letter topics and retry behaviour

What dead-letter topics do

A dead-letter topic is a separate topic that receives messages your subscription could not deliver after repeated attempts. Instead of retrying forever until the retention window expires, Pub/Sub forwards the message to the dead-letter topic so you can inspect it, fix the problem, and reprocess.

Maximum delivery attempts are approximate

When you set —max-delivery-attempts=10, Pub/Sub forwards the message after approximately 10 failed deliveries. The actual count may be slightly higher or lower due to the distributed nature of the system. Do not design logic that depends on an exact attempt count.

IAM requirements for dead-lettering

The Pub/Sub service agent needs two permissions:

• roles/pubsub.publisher on the dead-letter topic (to publish forwarded messages)
• roles/pubsub.subscriber on the source subscription (to acknowledge forwarded messages)

# Create a dead-letter topic and a monitoring subscription
gcloud pubsub topics create my-subscription-dead-letter
gcloud pubsub subscriptions create my-dead-letter-monitor \
  --topic=my-subscription-dead-letter

# Configure dead-letter policy on the source subscription
gcloud pubsub subscriptions update my-subscription \
  --dead-letter-topic=my-subscription-dead-letter \
  --max-delivery-attempts=10

# Grant the service agent permission to publish to the dead-letter topic
gcloud pubsub topics add-iam-policy-binding my-subscription-dead-letter \
  --member="serviceAccount:service-PROJECT_NUMBER@gcp-sa-pubsub.iam.gserviceaccount.com" \
  --role="roles/pubsub.publisher"

# Grant the service agent permission to ack on the source subscription
gcloud pubsub subscriptions add-iam-policy-binding my-subscription \
  --member="serviceAccount:service-PROJECT_NUMBER@gcp-sa-pubsub.iam.gserviceaccount.com" \
  --role="roles/pubsub.subscriber"

Retry policy vs push backoff

Pub/Sub has two separate retry mechanisms that are easy to confuse:

• Retry policy (configured on the subscription): controls the minimum and maximum delay between redelivery attempts. Applies to both pull and push subscriptions. Default is immediate redelivery with no delay.

• Push backoff: an automatic exponential backoff Pub/Sub applies to push endpoints that return errors. The backoff increases with consecutive failures and can reach minutes between attempts. This happens on top of any configured retry policy.

For pull subscriptions, the retry policy adds delay between redeliveries. For push subscriptions, the exponential backoff from consecutive failures is the dominant delay. You can configure the retry policy to add a minimum backoff for pull redeliveries:

# Set a retry policy with minimum 10s and maximum 600s backoff
gcloud pubsub subscriptions update my-subscription \
  --min-retry-delay=10s \
  --max-retry-delay=600s

When to use a dead-letter topic

Configure a dead-letter topic when:

• Message loss is unacceptable and you need to capture every failed message for investigation
• Some messages are “poison pills” that will never process successfully and would block other messages
• You want to separate healthy delivery from failed delivery for monitoring purposes

Inspecting dead-lettered messages

# Pull messages from the dead-letter monitoring subscription
gcloud pubsub subscriptions pull my-dead-letter-monitor \
  --limit=10 \
  --format="table(message.data.decode(base64), message.attributes)"

# Each dead-lettered message includes these attributes:
# - CloudPubSubDeadLetterSourceSubscription: which subscription forwarded it
# - CloudPubSubDeadLetterSourceDeliveryCount: approximate delivery attempts

Read the message body and the source subscription attribute. This tells you which subscription failed and what the message contained. Fix the subscriber, then decide whether to replay the dead-lettered messages or reprocess them manually.

Recover after the fix

After you fix the root cause, verify that delivery is actually recovering:

Confirm the backlog is draining

# Watch the backlog size decrease over time
gcloud pubsub subscriptions describe my-subscription \
  --format="value(name, numUndeliveredMessages)"

# Check oldest unacked message age — it should be dropping

The num_undelivered_messages metric should trend down and oldest_unacked_message_age should stabilise or drop. If neither changes, the fix did not work or there is a second problem.

Validate push delivery recovery

After fixing a push endpoint, check push_request_count grouped by response code. You should see 2xx responses increasing and error responses dropping. Push backoff may cause a delay before Pub/Sub ramps delivery back up after a prolonged failure period.

Verify subscriber acknowledgements

Check subscription/ack_message_count in Cloud Monitoring. A healthy subscriber shows a steady ack rate that matches or exceeds the publish rate.

Decide whether to replay messages

If messages were lost during the outage or if you need to reprocess messages from a specific point in time, Pub/Sub supports seek operations. You can seek to a timestamp to replay all messages published after that time, or seek to a snapshot to replay from a saved state. Use seek carefully because it redelivers messages that were already successfully processed. Your subscriber must handle duplicates safely.

# Seek to a timestamp to replay messages from that point forward
gcloud pubsub subscriptions seek my-subscription \
  --time="2026-03-27T10:00:00Z"

# Create a snapshot for future replay points
gcloud pubsub snapshots create my-snapshot \
  --subscription=my-subscription

Push vs pull failure patterns

For a full comparison of when to use each delivery type, see Pub/Sub Push vs Pull.

Checking subscription configuration

Many delivery failures trace back to misconfigured subscriptions. Check these fields:

# Describe a subscription — inspect all key fields
gcloud pubsub subscriptions describe my-subscription

# Key fields to check:
# - topic: the topic this subscription receives from
# - pushConfig.pushEndpoint: URL for push subscriptions (empty for pull)
# - ackDeadlineSeconds: how long before unacked messages are redelivered
# - messageRetentionDuration: how long undelivered messages are kept
# - deadLetterPolicy: dead-letter topic and max delivery attempts
# - retryPolicy: min and max backoff for redelivery

# List all subscriptions attached to a topic
gcloud pubsub topics list-subscriptions my-topic

How to confirm it is fixed

After applying your fix, verify these indicators over the next 15–30 minutes:

• num_undelivered_messages is trending down (backlog draining)
• oldest_unacked_message_age has stabilised or is dropping
• push_request_count shows 2xx responses (for push subscriptions)
• dead_letter_message_count has stopped growing
• ack_message_count shows steady acknowledgements matching the publish rate
• Subscriber application logs show successful processing without repeated errors

If the backlog is not draining after 15 minutes, there may be a second issue or the fix did not fully resolve the problem. Re-check the triage table and metrics.