GCP Streaming Pipelines | Dataflow, Pub/Sub, and BigQuery

What this means in simple terms

Most data processing waits. You collect a day’s worth of logs, run a job overnight, and check the results in the morning. That is batch processing, and it works well for many use cases.

Streaming is for when you cannot wait. Instead of collecting data and processing it later, a streaming pipeline processes each event within seconds of it happening. Think of a fraud detection system that needs to flag a suspicious transaction before it clears, or an IoT dashboard that shows live sensor readings.

The challenge is that streaming data never ends. You cannot aggregate “all the data” because there is always more coming. So streaming pipelines use windows to group events into time-based chunks, watermarks to estimate when a window’s data is complete, and triggers to decide when to emit results. These three concepts are what make streaming pipelines fundamentally different from batch.

What a streaming pipeline in GCP actually is

A GCP streaming pipeline follows a consistent pattern: producers generate events, Pub/Sub buffers and distributes them, Dataflow processes them continuously using Apache Beam, and results land in BigQuery or another sink.

The data is unbounded, meaning there is no defined end to the stream. Unlike a batch pipeline that reads a file from start to finish, a streaming pipeline reads from a source that produces events indefinitely. A Pub/Sub subscription is unbounded. A CSV file in Cloud Storage is bounded.

How streaming pipelines work in GCP

Here is how data flows through a typical GCP streaming pipeline, step by step:

1. Events are produced. Applications, devices, or services generate events such as a purchase, a sensor reading, or a page view. Each event carries a payload and an event timestamp.
2. Pub/Sub buffers and distributes them. Producers publish to a Pub/Sub topic. Pub/Sub provides at-least-once delivery and decouples producers from consumers. If Dataflow falls behind, Pub/Sub holds messages until they can be processed.
3. Dataflow processes events continuously. A Dataflow job runs an Apache Beam pipeline that reads from the Pub/Sub subscription, applies transforms, and never terminates.
4. Event time is read from the message. Each event’s timestamp is extracted from the message payload or Pub/Sub publish time. This is the event time that windowing is based on.
5. Windows group events into time-based chunks. Because the stream never ends, windows define finite boundaries for aggregation. A 5-minute fixed window groups all events with event times between 14:00 and 14:05.
6. Watermarks estimate completeness. Dataflow tracks the watermark, its best estimate of how far through event time it has progressed. When the watermark passes the end of a window, Dataflow considers that window’s data mostly complete.
7. Triggers decide when results are emitted. The default trigger fires once when the watermark passes the window boundary. Custom triggers can fire early (for speculative results), on every late element, or on a combination of conditions.
8. Results are written to a sink. The processed output goes to BigQuery, Cloud Storage, another Pub/Sub topic, or any supported destination.

Producer → Pub/Sub topic → Dataflow (Beam) → BigQuery
                                ↓
                        Dead-letter topic
                        (inspect & replay)

When to use streaming pipelines

Streaming is the right choice when the business genuinely needs results in seconds or minutes:

• Real-time fraud scoring to block a transaction before it clears, not in tomorrow’s report
• Operational alerting to trigger an alert when error rates spike, not when a batch job runs hours later
• Live dashboards to show current state, not yesterday’s state
• User-facing personalisation to adjust recommendations based on what a user is doing right now
• IoT anomaly detection to identify a failing sensor before it causes damage

The common thread is latency-driven decision making. If acting on stale data has a meaningful cost (financial, operational, or user experience), streaming is worth the added complexity.

When batch is a better choice

Batch pipelines are simpler to build, cheaper to run, and easier to debug. If your use case does not require sub-minute latency, batch is almost always the better starting point.

• Daily reports: analysts check dashboards once a day. A nightly batch job is sufficient.
• Historical analysis: backfilling months of data is a batch problem, not a streaming one.
• Cost-sensitive workloads: batch loads into BigQuery from Cloud Storage are free. Streaming writes are not.
• Simple transformations: if you are just moving data from A to B on a schedule, a batch pipeline avoids windowing and watermark complexity entirely.

Streaming pipelines vs batch pipelines

For a broader comparison of pipeline design approaches, see ETL vs ELT.

Windowing explained

Windows divide an unbounded stream into finite chunks so you can aggregate data. Beam provides three main window types.

Fixed (tumbling) windows split the stream into non-overlapping intervals of equal size. All events with event times between 14:00 and 14:05 go in one window; 14:05 to 14:10 in the next. Use these for clean, non-overlapping time buckets like hourly counts or 5-minute averages.

Sliding windows overlap. A 1-hour window that slides every 10 minutes produces a new window every 10 minutes, each covering the most recent 60 minutes. Use these for rolling averages and moving metrics.

Session windows group events by gaps in activity. A window opens when an event arrives and closes after a configurable period of inactivity (the gap duration). Use these for user session analytics where bursts of activity matter more than fixed time boundaries.

Watermarks, triggers, and late data explained

These three concepts work together to answer one question: when should the pipeline emit results for a window?

Watermarks

The watermark is Dataflow’s estimate of the current event-time frontier. It tracks how far through event time the pipeline has progressed. When the watermark passes the end boundary of a window, Dataflow considers that window’s input mostly complete.

The watermark is not a guarantee. It is a heuristic. Data can still arrive after the watermark has passed, and that data is considered late.

Triggers

Triggers control when a window’s results are emitted. The default trigger fires once when the watermark passes the window boundary. But you can configure more sophisticated behaviour:

• Early triggers emit speculative results before the watermark passes the window. Useful for dashboards that need fast initial results even if they are incomplete.
• Late triggers emit updated results when late data arrives after the watermark has passed.
• Composite triggers combine conditions. For example: emit early every 30 seconds, then emit again on each late element.

Allowed lateness

Allowed lateness defines how long after the watermark passes a window boundary Dataflow should continue accepting late data for that window. Without allowed lateness, late data is silently discarded.

For example, setting allowed lateness to 2 hours means that if a window closes at 14:05 and the watermark passes, Dataflow will still accept events for that window until 16:05. When a late event arrives within the allowed lateness period, the late trigger fires and an updated result is emitted.

Example: Pub/Sub to Dataflow to BigQuery

This example reads IoT sensor events from Pub/Sub, assigns event-time timestamps, applies 5-minute fixed windows with 2 hours of allowed lateness, and writes average temperatures per sensor to BigQuery using the Storage Write API.

import json
import apache_beam as beam
from apache_beam.options.pipeline_options import PipelineOptions
from apache_beam.transforms.window import FixedWindows
from apache_beam.transforms.trigger import AfterWatermark, AfterProcessingTime, AccumulationMode
import apache_beam.transforms.window as window

options = PipelineOptions([
    '--runner=DataflowRunner',
    '--project=my-app-prod',
    '--region=europe-west2',
    '--streaming',
    '--temp_location=gs://my-app-prod-data/temp',
])

def assign_event_timestamp(element):
    """Use the timestamp from the event payload as the element's event time."""
    import apache_beam as beam
    record = json.loads(element)
    event_time = record['event_timestamp']
    return beam.window.TimestampedValue(record, event_time)

with beam.Pipeline(options=options) as pipeline:
    (
        pipeline
        | 'Read from Pub/Sub' >> beam.io.ReadFromPubSub(
            topic='projects/my-app-prod/topics/sensor-readings'
          )
        | 'Assign event time' >> beam.Map(assign_event_timestamp)
        | 'Window 5 min' >> beam.WindowInto(
            FixedWindows(5 * 60),
            trigger=AfterWatermark(
                early=AfterProcessingTime(60),
            ),
            accumulation_mode=AccumulationMode.ACCUMULATING,
            allowed_lateness=2 * 60 * 60
          )
        | 'Key by sensor' >> beam.Map(lambda r: (r['sensor_id'], r['temperature']))
        | 'Average per sensor' >> beam.CombinePerKey(beam.combiners.MeanCombineFn())
        | 'Write to BigQuery' >> beam.io.WriteToBigQuery(
            'my-app-prod:analytics.sensor_averages',
            method=beam.io.WriteToBigQuery.Method.STORAGE_WRITE_API,
            write_disposition=beam.io.BigQueryDisposition.WRITE_APPEND
          )
    )

Cost and operational considerations

Streaming pipelines are more expensive to run than batch equivalents. Understand the trade-offs before committing.

• Dataflow compute: streaming jobs run 24/7. Workers are billed per second. A job with 4 workers running continuously costs far more than a batch job that runs for 10 minutes per hour. Use —max_num_workers to cap autoscaling costs.
• BigQuery writes: the Storage Write API charges per byte written. Batch loads from Cloud Storage are free. For high-volume streams, write costs can be significant. See BigQuery Pricing Explained for current rates.
• Pub/Sub: charged per message and per byte delivered. At high throughput, Pub/Sub costs are non-trivial. Batching messages on the producer side reduces per-message overhead.
• State and allowed lateness: Dataflow keeps window state in memory for the duration of the allowed lateness period. Longer allowed lateness means more state, more memory, and higher worker costs.
• Operational overhead: streaming jobs need monitoring, alerting, and on-call attention. A stuck pipeline means stale data. Plan for alerting on pipeline lag and throughput from day one.

For broader cost strategies, see Cost Optimisation Strategies.

Related topics and next steps

Now that you understand streaming pipelines, these pages go deeper into the individual components:

• Dataflow Overview covers how Dataflow manages workers, autoscaling, and job lifecycle
• Apache Beam Basics explains PCollections, transforms, and the programming model behind Dataflow
• Pub/Sub Overview covers topics, subscriptions, and delivery guarantees
• Pub/Sub Messaging Model explains push vs pull, ordering, and exactly-once delivery
• Event-Driven Architectures covers designing systems around events rather than direct calls
• Designing Data Pipelines walks through architecture patterns, dead-letter handling, and testing
• BigQuery Performance Optimisation helps you optimise queries and storage for the data your pipeline produces
• ETL vs ELT covers where transformation happens and why it matters