Event-Driven Architecture in GCP: Pub/Sub Patterns for Data Pipelines
This page covers event-driven architecture from a data-and-analytics perspective. The focus is on using Pub/Sub to feed analytics pipelines: clickstream ingestion, telemetry, and order events flowing into BigQuery. If you are looking for the broader architectural view (Eventarc, orchestration patterns, service decoupling), see Event-Driven Systems in GCP.
This page is for beginners who want to understand when and why to use event-driven patterns for data workloads, what GCP services are involved, and what mistakes to avoid.
Simple explanation
Think of a factory conveyor belt. Instead of your application hand-delivering analytics data to every system that needs it, it drops a short message onto the belt (Pub/Sub). Different machines sit along the belt: one enriches the data, another aggregates it, another loads it into a warehouse. Each machine works at its own speed. If one machine stops, the belt keeps moving and messages queue up until the machine restarts. No machine needs to know about the others.
That is the core idea: produce events once, consume them many ways, and let the pipeline absorb spikes without breaking your application.
How event-driven architecture works in GCP
A typical analytics-oriented event flow has four layers:
1. PRODUCE 2. ROUTE 3. CONSUME 4. SINK
───────────── ────────────── ──────────────────── ──────────
App / service ──▶ Pub/Sub topic ──┬──▶ Subscription A │
│ (Dataflow streaming job) ──▶ BigQuery
│
├──▶ Subscription B
│ (Cloud Run service) ──────▶ Operational DB
│
└──▶ Subscription C
(Cloud Function) ─────────▶ Alert / SlackFan-out is the key concept. One Pub/Sub topic can have many subscriptions. Each subscription receives every message independently. Adding a new consumer means creating a new subscription. The producer does not change, and existing consumers are not affected.
Fan-out happens at the subscription level, not the topic level. If two Cloud Run instances share one subscription, they compete for messages (load balancing). If they each have their own subscription, they both receive every message (fan-out). This distinction matters when you design your pipeline. See Pub/Sub Messaging Model for the full picture.
For a deeper look at delivery modes, see Pub/Sub Push vs Pull.
GCP services that usually make up the pattern
| Service | What it does here | When to use it |
|---|---|---|
| Pub/Sub | Event bus: accepts events from producers, fans out to subscribers | Always. It is the backbone of every event-driven pipeline in GCP. |
| Dataflow | Stream processing: reads from a Pub/Sub subscription, transforms, aggregates, windows, and writes to a sink | When events need enrichment, joins, windowed aggregation, or high-throughput continuous processing before landing in a warehouse. |
| BigQuery | Analytical warehouse: the final destination for processed events | When you need SQL-queryable analytics, dashboards, or long-term event storage. |
| Cloud Run | Stateless HTTP consumer: receives events via a Pub/Sub push subscription | When event processing is custom application logic (validation, enrichment, API calls) and you want container-based deploys. |
| Cloud Functions | Lightweight event handler: triggered directly by Pub/Sub | When the processing logic is small and self-contained (send an alert, transform a single record, call an API). |
| Eventarc | Routes platform events (file uploaded to Cloud Storage, Audit Log entry) to Cloud Run or Cloud Functions | When you need to react to GCP infrastructure events, not application events. See Event-Driven Systems in GCP for details. |
Real example: event-driven analytics pipeline
A SaaS product tracks user activity. Every click, page view, and feature usage generates an event. The team needs real-time dashboards and the ability to run ad-hoc queries on the full event history.
Web app / mobile SDK
publishes JSON events
|
v
Pub/Sub topic: user-activity
|
├──▶ Subscription: analytics-stream (pull)
| Dataflow streaming job
| - parses and validates JSON
| - enriches with user metadata from Firestore
| - windows into 1-minute micro-batches
| - writes to BigQuery table: analytics.user_events
|
├──▶ Subscription: anomaly-detector (push)
| Cloud Run service
| - checks event patterns against rules
| - fires alert if anomaly detected
|
└──▶ Subscription: raw-archive (pull)
Cloud Function
- writes raw JSON to Cloud Storage (cold archive)The web app publishes one event per user action. Three independent consumers handle it:
- A Dataflow streaming pipeline enriches events and loads them into BigQuery for dashboards and SQL queries.
- A Cloud Run service watches for anomalies in real time.
- A Cloud Function archives the raw JSON to Cloud Storage for compliance.
Adding a fourth consumer (say, a machine-learning feature store) means creating a new subscription on the same topic. No changes to the web app or any existing consumer. This is the power of fan-out: your producers never need to know what happens downstream.
When to use this
- High-volume event ingestion. Clickstream, telemetry, IoT sensor data, or transaction logs that arrive continuously and need to flow into an analytics system.
- Multiple downstream consumers. The same event needs to feed a dashboard, a data lake, an alerting system, and an operational service, all independently.
- Real-time analytics. You need dashboards or alerts that reflect events within seconds, not hours.
- Decoupling producers from analytics pipelines. Application teams should not need to know or care how the analytics pipeline works. They publish events; the data team subscribes.
- Asynchronous processing. The user does not need an immediate answer from the analytics system. They click a button; the event flows through the pipeline in the background.
- Absorbing traffic spikes. Pub/Sub queues events during bursts so downstream consumers can process at a steady rate without being overwhelmed.
When not to use this
- Synchronous read/check flows. A user asking “what is my account balance?” needs an immediate answer. That is a direct API call, not an event.
- Simple one-to-one background work. If you need to send one task to one worker with rate limiting and scheduling, Cloud Tasks is simpler than Pub/Sub.
- Small, simple systems. Two services calling each other directly is easier to build, test, and debug than adding an event bus in between.
- Strong consistency requirements. If every consumer must see the same state at the same instant, the eventual consistency of event-driven systems works against you. Consider a synchronous transactional approach instead.
- Your team is new to distributed systems. Tracing a user action across five asynchronous consumers is harder than following a synchronous call chain. Start with direct calls and introduce events incrementally as you build confidence.
Choosing between synchronous calls and event-driven architecture is like choosing between a phone call and a letter. A phone call gives you an immediate answer but ties up both people at the same time. A letter lets both sides work on their own schedule, but you have to accept a delay before you hear back. For analytics workloads the delay is almost always acceptable, so the “letter” approach wins.
Event-driven architecture vs synchronous API calls
| Dimension | Event-driven (Pub/Sub) | Synchronous (HTTP / gRPC) |
|---|---|---|
| Coupling | Loose: producer does not know consumers | Tight: caller must know the endpoint |
| Latency | Milliseconds to seconds (async) | Milliseconds (immediate response) |
| Consistency model | Eventually consistent | Strongly consistent (within the call) |
| Scaling pattern | Consumers scale independently; Pub/Sub buffers during spikes | Caller and callee must both handle the load simultaneously |
| Debugging complexity | Higher: requires distributed tracing and correlation IDs | Lower: a single request/response chain |
| Best-fit use cases | Analytics pipelines, fan-out, high-volume ingestion, decoupled consumers | User-facing reads, validations, lookups, low-latency workflows |
Pub/Sub vs Cloud Tasks
This is a common point of confusion. The short version:
- Pub/Sub = event fan-out. One message, many independent consumers. The producer does not know or control who receives it.
- Cloud Tasks = task dispatch. One task, one explicit target endpoint. The sender controls the destination, rate, and schedule.
For analytics pipelines, Pub/Sub is almost always the right choice because you typically need multiple consumers (a streaming pipeline, an alerting service, an archive) processing the same events independently.
Cloud Tasks is better when you need to send a specific piece of work to a specific worker, like scheduling a report generation job or rate-limiting API calls to a third-party service.
For a full comparison with code examples and a decision table, see Pub/Sub vs Cloud Tasks.
Common mistakes
- Not designing consumers for idempotency. Pub/Sub guarantees at-least-once delivery. Under normal conditions (subscriber timeouts, network hiccups, pod restarts), the same message will arrive more than once. If your Dataflow job or Cloud Run handler inserts a duplicate row or double-counts a metric, your analytics are wrong. Use upserts keyed on event ID, or deduplicate explicitly before writing.
- Schema drift in event payloads. When the producing team adds, renames, or removes fields without coordinating with consumers, pipelines break silently. A Dataflow job drops records it cannot parse. BigQuery loads nulls into columns that should have values. Version your event schemas and validate payloads at the producer before publishing.
- No dead-letter handling. Without a dead-letter topic, a malformed message is retried until the retention period expires, then dropped. That is silent data loss. Configure a dead-letter topic on every production subscription and alert on its message count.
- Publishing events from rolled-back transactions. If you publish a Pub/Sub message inside a database transaction and the transaction rolls back, the message was already sent but the state change it describes never happened. Downstream consumers process a phantom event. Use the outbox pattern: write the event to an outbox table in the same transaction, then publish from a separate process after commit.
- Weak observability. A user action that flows through three Pub/Sub topics and five services is invisible without distributed tracing. Always include a
correlation_idin event attributes, propagate it through every downstream service, and include it in every log entry. Use Cloud Trace to visualise the chain. - Using event-driven architecture for the wrong workloads. Not every integration benefits from an event bus. A synchronous lookup (“is this coupon valid?”) or a simple request-response API is easier to build and debug than routing it through Pub/Sub. Choose event-driven patterns where you actually need decoupling, fan-out, or async processing.
Pub/Sub’s default delivery guarantee is at-least-once, not exactly-once. There is an optional exactly-once mode at the subscription level, but it adds cost and reduces throughput. The safe default is to assume every consumer will see duplicates and design accordingly. If your consumer sends an email, charges a card, or increments a counter, it must check whether it already processed that event ID before acting.
Frequently asked questions
What is event-driven architecture in GCP?
Event-driven architecture is a design where services publish events to Pub/Sub topics instead of calling each other directly. Consumers subscribe independently and process events at their own pace. In GCP, the core pattern is producers publishing to Pub/Sub, with Dataflow, Cloud Run, Cloud Functions, and BigQuery acting as downstream consumers and sinks.
When should I use Pub/Sub instead of Cloud Tasks?
Use Pub/Sub when one event needs to reach multiple independent consumers (fan-out) or when you are feeding a data pipeline. Use Cloud Tasks when you need to deliver a single task to a single endpoint with rate limiting, scheduling, or built-in deduplication. Pub/Sub broadcasts; Cloud Tasks dispatches.
Is Pub/Sub exactly-once?
By default, Pub/Sub guarantees at-least-once delivery, meaning a message may arrive more than once. Pub/Sub does offer an exactly-once delivery mode at the subscription level, but it adds cost and reduces throughput. The standard design expectation is that all consumers are idempotent, meaning safe to process the same event twice.
How do Dataflow and BigQuery fit into event-driven systems?
Dataflow is a stream-processing engine that reads events from a Pub/Sub subscription, transforms or aggregates them (windowing, joins, enrichment), and writes the results to a sink like BigQuery. BigQuery is the analytical warehouse where processed events land for SQL queries, dashboards, and reporting. Together they form the analytics leg of an event-driven pipeline.
What is eventual consistency?
In an event-driven system there is always a delay between when a producer publishes an event and when all consumers finish processing it. During that window, different parts of the system may show different data. Eventual consistency means the system will converge to a correct state, but not instantly. Design your application and UI to tolerate this delay.