Google Cloud Pub/Sub Overview: Topics, Subscriptions, and Delivery Explained
Google Cloud Pub/Sub is a fully managed asynchronous messaging service that lets one service send messages without knowing which services will receive them. Publishers write to a topic. Each subscription attached to that topic gets its own independent copy of every message. Pub/Sub buffers messages in between, so a slow or temporarily offline subscriber does not cause message loss or affect other subscribers. The delivery guarantee is at-least-once by default, which means duplicate deliveries are possible and your consumers need to handle them.
Simple explanation
Think of Pub/Sub as a post office for your cloud services. A service drops a letter (message) into a named mailbox (topic). The post office makes a copy of that letter for every subscriber who registered for that mailbox. Each subscriber picks up their copy on their own schedule. If a subscriber is away, the post office holds the letter until they return or until it expires.
The key insight: the sender never knows who the recipients are, and the recipients never know who sent the message. That separation is what makes Pub/Sub useful.
Why Pub/Sub exists
Direct service-to-service calls create tight coupling. If Service A calls Service B directly over HTTP, three things must be true: A must know B’s address, B must be available when A calls, and a traffic spike from A directly stresses B. If B goes down, A either fails or has to implement its own retry logic.
Pub/Sub removes all three constraints:
- Decoupling. A publishes to a topic. B subscribes to that topic. Neither knows about the other. You can replace, scale, or redeploy B without touching A.
- Buffering. Pub/Sub holds messages when B is slow or temporarily down. B processes them when it recovers. No messages are lost within the retention window.
- Fan-out. Adding a third consumer requires no changes to the publisher. Create a new subscription and it immediately starts receiving every future message from the topic.
- Retries. If a subscriber fails to acknowledge a message, Pub/Sub automatically redelivers it. You do not need to build retry logic in the publisher.
- Async processing. The publisher returns immediately after Pub/Sub accepts the message. Heavy processing happens later, on the subscriber side, at whatever pace the subscriber can handle.
These properties make Pub/Sub the foundation of most event-driven architectures on Google Cloud.
How Pub/Sub works
The message lifecycle follows a clear sequence:
- Publisher sends a message. Your application publishes a message (data + optional attributes) to a named topic using the Pub/Sub API or a client library.
- Topic receives the message. Pub/Sub accepts the message, assigns it a unique
messageIdand apublishTimetimestamp, and stores it durably. - Each subscription gets its own copy. Every subscription attached to the topic receives an independent copy of the message. Subscriptions are isolated from each other: a backlog in one does not affect delivery to others.
- Subscriber processes the message. The subscriber receives the message (via push or pull) and does its work: writing to a database, triggering a workflow, updating a cache, or whatever logic it needs.
- Subscriber acknowledges. After successful processing, the subscriber sends an acknowledgement (ack) back to Pub/Sub. The message is then removed from that subscription.
- Pub/Sub redelivers when needed. If the subscriber does not ack within the ack deadline, Pub/Sub assumes processing failed and redelivers the message. This is how at-least-once delivery works.
For a deeper look at message flow, fan-out patterns, ordering, and exactly-once delivery, see Pub/Sub Messaging Model.
Core concepts
Topics
A topic is a named channel that receives messages from publishers. Topics have no knowledge of their subscribers. You can think of a topic as the category or event type: order-created, user-signed-up, sensor-reading. A topic can have zero or many subscriptions.
Subscriptions
A subscription is a named resource attached to a topic. Each subscription independently tracks which messages have been delivered and acknowledged. A topic with three subscriptions means every published message is delivered three times: once to each subscription. Subscriptions only receive messages published after the subscription was created.
Subscriptions do not receive messages published before they existed. If you create a topic, publish 1,000 messages, and then create a subscription, that subscription starts empty. Always create subscriptions before you begin publishing, or enable topic-level retention and use seek-to-timestamp to replay past messages.
Publishers and subscribers
A publisher is any application that sends messages to a topic. A subscriber is any application that receives messages from a subscription. These are roles, not dedicated infrastructure. A single service can be both a publisher on one topic and a subscriber on another. Common subscriber targets include Cloud Run services and Cloud Functions.
Message data and attributes
A Pub/Sub message has two parts: a data field (arbitrary bytes, up to 10 MB, base64-encoded in the API) and optional attributes (string key-value pairs). Most applications encode a JSON payload as UTF-8 bytes in the data field.
Attributes are useful for routing and filtering without parsing the data payload. You can attach metadata like event type, source service, or schema version. Subscriptions support server-side filtering on attribute values, so subscribers only receive messages matching specific criteria.
Acknowledgements and ack deadline
When a subscriber finishes processing a message, it sends an acknowledgement (ack) back to Pub/Sub. Once acked, the message is removed from that subscription. If the subscriber does not ack within the ack deadline (configurable between 10 seconds and 600 seconds per subscription, default 10 seconds), Pub/Sub redelivers the message.
The default ack deadline is only 10 seconds. If your processing involves database writes, external API calls, or any I/O that could stall, you will hit this deadline regularly. Either set a longer ack deadline on the subscription or extend it programmatically using modifyAckDeadline before it expires.
For troubleshooting redelivery issues, see Pub/Sub Message Delivery Failures.
At-least-once delivery and idempotency
Pub/Sub guarantees that every message is delivered at least once to each subscription. Duplicates can occur when a subscriber does not ack in time, when a subscriber restarts, or during infrastructure events. This is not a rare edge case. It happens during normal operation.
Your consumers must be idempotent: processing the same message twice should produce the same outcome as processing it once. Common approaches include using the message’s messageId as a deduplication key, checking whether the event was already processed before applying side effects, and using conditional writes.
Exactly-once delivery is available as an opt-in subscription setting. It has higher latency, lower throughput, and higher cost than at-least-once. Most teams find that designing idempotent consumers with standard at-least-once delivery is cheaper and more resilient.
Message retention and replay
Pub/Sub has two separate retention mechanisms:
- Subscription message retention. Unacknowledged messages in a subscription are retained for up to the subscription’s message retention duration, which defaults to 7 days.
- Topic-level message retention. An optional setting that retains all published messages at the topic level, regardless of acknowledgement state. Configurable from 10 minutes to 31 days. When enabled, you can use Seek to replay messages to any subscription attached to that topic.
These are independent settings. Topic-level retention is what enables replay. Without it, once a message is acknowledged, it is gone from that subscription.
If you ever need to reprocess messages (for example, after deploying a bug fix), enable topic-level retention. Without it, you cannot replay acknowledged messages.
Push vs pull, ordering keys, dead-letter topics, and schemas
Pub/Sub supports two delivery modes: pull (the subscriber requests messages) and push (Pub/Sub sends messages to an HTTPS endpoint). The choice depends on your subscriber’s architecture. See Pub/Sub Push vs Pull for a full comparison.
Ordering keys guarantee that messages with the same key are delivered in publish order within a single region. Dead-letter topics capture messages that a subscriber repeatedly fails to process, preventing them from blocking the subscription indefinitely. Schemas let you enforce a contract (Avro or Protocol Buffers) on message data at publish time. These features are covered in detail in Pub/Sub Messaging Model.
When to use Pub/Sub
Event-driven microservices. Services communicate through events published to topics rather than direct HTTP calls. Each service subscribes to the events it cares about. See Event-Driven Architectures for the full pattern.
Async background processing. Offload slow work from your request path. A web server publishes a message and returns immediately. A background worker picks it up and processes it at its own pace. Cloud Functions triggered by Pub/Sub are a common choice for this pattern.
Fan-out to multiple consumers. A single order-created event can be consumed by an analytics pipeline, a notification service, and a billing system independently. Each has its own subscription and processes at its own speed. No coordination needed between them.
Streaming into analytics. Application events flow from Pub/Sub into a Dataflow pipeline for transformation, then into BigQuery for analysis. This is one of the most common streaming pipeline patterns on GCP.
State-change notifications. Publish events when resources change state (file uploaded, job completed, config updated). Other services subscribe to react to those changes without polling.
An e-commerce service publishes an order-created event to a Pub/Sub topic. Three subscriptions exist on that topic: one feeds the analytics pipeline that updates the sales dashboard, one triggers the email notification to the customer, and one initiates the billing workflow. Each processes the event independently. If the email service is down for five minutes, the analytics and billing subscriptions are unaffected, and the email subscription processes its backlog when the service recovers.
When not to use Pub/Sub
Synchronous request/response. If a caller needs an immediate answer (an API returning data to a user), Pub/Sub adds unnecessary latency and complexity. Use direct HTTP or gRPC calls.
Single deferred task execution. If you need to send exactly one task to exactly one worker, with rate limiting or scheduled execution, Cloud Tasks is a better fit. Pub/Sub is designed for broadcast and fan-out, not targeted task dispatch.
Strict ordering across all messages. Pub/Sub supports ordering within a single ordering key, not global ordering across all messages. If you need total ordering across an entire stream, you may need a different architecture or a single ordering key (which limits throughput).
Pub/Sub messages are capped at 10 MB. For large files or datasets, store the data in Cloud Storage and publish a Pub/Sub message containing the object path. The subscriber reads the reference and fetches the actual data separately.
Pub/Sub vs Cloud Tasks
This is one of the most common decision points for beginners. Both handle asynchronous work, but they solve different problems.
| Pub/Sub | Cloud Tasks | |
|---|---|---|
| Delivery model | Fan-out: each subscription gets a copy | Single delivery to one target |
| Publisher knows receiver? | No | Yes (target URL specified at enqueue time) |
| Multiple consumers | Built-in via multiple subscriptions | Not supported; one task, one handler |
| Rate limiting | Flow control on subscriber side only | Built-in per-queue rate limiting |
| Scheduled dispatch | Not supported | Built-in (schedule tasks for future execution) |
| Task-level deduplication | Not built-in (consumer must be idempotent) | Built-in via task name |
Rule of thumb: Use Pub/Sub when you are broadcasting events to unknown or multiple consumers. Use Cloud Tasks when you are dispatching a specific job to a specific handler. For a full comparison, see Pub/Sub vs Cloud Tasks.
Common beginner mistakes
- Not designing for duplicate delivery. At-least-once delivery is not a theoretical edge case. Duplicates happen during normal operation. If your consumer is not idempotent, you will end up with corrupted data, double charges, or duplicate notifications. Design all consumers to be idempotent from the start.
- Misunderstanding the ack deadline. The default ack deadline is 10 seconds. If your processing takes longer, Pub/Sub redelivers the message before you finish, causing duplicates and wasted work. Either set a longer ack deadline on the subscription or extend it programmatically with
modifyAckDeadline. - Confusing topic retention with subscription retention. Subscription message retention controls how long unacknowledged messages stay in the backlog. Topic-level message retention is a separate, optional feature that retains all messages (including acknowledged ones) for replay. You need topic-level retention enabled to use Seek for replaying past messages.
- Creating subscriptions after publishing. Subscriptions only receive messages published after the subscription was created. If you create a topic, publish 1,000 messages, then create a subscription, that subscription gets none of those messages. Always create subscriptions before you start publishing, or use seek-to-timestamp with topic-level retention to replay from a past point.
- Sending large payloads directly. Messages are capped at 10 MB. For large payloads, store the data in Cloud Storage and publish a reference (the GCS object path) as the message. The subscriber fetches the data separately.
- Choosing Pub/Sub when Cloud Tasks is the right tool. If you need to send one task to one handler with rate limiting or scheduled execution, Cloud Tasks is purpose-built for that. Pub/Sub’s fan-out model adds unnecessary complexity for single-target task dispatch.
Summary
- Pub/Sub is a managed asynchronous messaging service that decouples publishers from subscribers via topics and subscriptions.
- Publishers send messages to topics. Each subscription attached to a topic gets its own independent copy. Fan-out is built in.
- Delivery is at-least-once by default. Duplicates happen in normal operation. All consumers must be idempotent.
- Unacknowledged messages are retained for up to 7 days by default. Topic-level retention (optional, up to 31 days) enables message replay via Seek.
- The default ack deadline is 10 seconds. Extend it programmatically for slow-processing consumers.
- Use Pub/Sub for event fan-out and streaming. Use Cloud Tasks for single-target task dispatch with rate limiting.
Frequently asked questions
What is Google Cloud Pub/Sub used for?
Pub/Sub is used for asynchronous messaging between services. Common use cases include event-driven microservices, streaming data into analytics systems like BigQuery, distributing notifications, buffering work for background processing, and fanning out a single event to multiple independent consumers.
What is the difference between a topic and a subscription?
A topic is a named channel that receives messages from publishers. A subscription is a named resource attached to a topic that delivers copies of those messages to subscribers. One topic can have many subscriptions, and each subscription receives its own independent copy of every message. Publishers send to topics; subscribers read from subscriptions.
Does Pub/Sub guarantee exactly-once delivery?
By default, Pub/Sub provides at-least-once delivery, meaning duplicates are possible. Exactly-once delivery is available as an opt-in subscription setting, but it has higher latency and lower throughput. Most production systems use at-least-once delivery and design consumers to be idempotent instead.
How long are messages kept in Pub/Sub?
Unacknowledged messages in a subscription are retained for up to the subscription message retention duration, which defaults to 7 days. Separately, you can enable topic-level message retention (10 minutes to 31 days) to retain all published messages regardless of acknowledgement state, which enables seek and replay.
When should I use Pub/Sub instead of Cloud Tasks?
Use Pub/Sub when multiple consumers need the same message (fan-out), when you need streaming delivery, or when publishers should not know about consumers. Use Cloud Tasks when you need to dispatch a single task to a single handler with features like rate limiting, scheduled execution, and task-level deduplication.