Apache Beam Basics on GCP: Pipelines, PCollections, Transforms, and Dataflow

Simple explanation

Apache Beam is how you describe a data pipeline in code. You say: read data from here, transform it like this, and write the results over there. Beam turns that description into a graph of steps.

Dataflow is how Google Cloud runs that pipeline at scale. It takes the graph Beam built and distributes the work across as many machines as needed, handling failures and scaling automatically.

The two work together but are separate things. Beam is the programming model. Dataflow is one of several execution engines (called runners) that can execute Beam code.

Why Apache Beam exists

Before Beam, building a data pipeline meant choosing an execution engine first and writing code tightly coupled to it. If you wrote a Spark job, your code only ran on Spark. If you wanted to move to a different engine, you rewrote the pipeline from scratch. Testing required spinning up the same engine locally or deploying to a cluster and waiting.

Beam solves this by separating two concerns:

• Pipeline logic: what data to read, how to transform it, and where to write the results.
• Execution: how to distribute that work across machines, manage parallelism, and handle failures.

You write the pipeline logic once. The runner handles execution. During development, you run the pipeline locally with the DirectRunner and get results in seconds. In production, you switch to DataflowRunner and the same code runs across hundreds of workers. If you later need to run on Apache Flink, you change the runner again without touching pipeline logic.

Beam also provides a single programming model that works for both batch and streaming data. Instead of learning one framework for files and another for message streams, you use the same PCollections and transforms for both. This unified model is one of the main reasons GCP teams adopt Beam for data pipeline design.

How Apache Beam works

A Beam pipeline follows a consistent flow from source to sink:

1. Read data from a source. The pipeline reads input from a file in Cloud Storage, a BigQuery table, a Pub/Sub subscription, or any supported connector.
2. Store it in a PCollection. The data becomes a PCollection: a distributed, parallel dataset that Beam manages across workers.
3. Apply transforms. You chain PTransforms (Map, Filter, GroupByKey, Combine, and others) to parse, clean, enrich, aggregate, or reshape the data. Each transform takes a PCollection as input and produces a new PCollection as output.
4. Send the pipeline to a runner. The runner receives the full pipeline graph and decides how to execute it. The DirectRunner runs everything locally. The DataflowRunner submits the graph to Dataflow for distributed execution.
5. Write results to a sink. The final PCollection is written to an output destination: a file, a BigQuery table, a Pub/Sub topic, or a database.

Core concepts

Pipeline

A Pipeline is the top-level object that represents your entire data processing job. It connects sources, transforms, and sinks into a directed acyclic graph (DAG). In Python, you create a pipeline as a context manager using with beam.Pipeline() as p. Everything inside the with block adds steps to the graph. The pipeline does not execute until you exit the context.

Think of a pipeline as a blueprint. You describe every step the data goes through before any data actually moves. The runner reads the blueprint and executes it.

PCollection

A PCollection (parallel collection) is the data flowing through a pipeline. Every transform takes one or more PCollections as input and produces a new PCollection as output. PCollections are the edges in the pipeline graph.

Example: reading a CSV file produces a PCollection of strings (one per line). Applying a Map transform to parse each line produces a new PCollection of parsed records.

PTransform

A PTransform is an operation applied to a PCollection. Beam provides built-in transforms for the most common operations:

• Map: apply a function to each element individually.
• Filter: keep only elements that match a condition.
• FlatMap: like Map, but each input element can produce zero or more output elements.
• GroupByKey: group elements by key, producing key-value pairs where each key maps to all its values.
• Combine: aggregate values (sum, count, average) more efficiently than GroupByKey followed by manual aggregation.
• Flatten: merge multiple PCollections of the same type into one.

You chain transforms together using the | operator. Each transform produces a new PCollection, so you can build long processing chains.

DoFn

A DoFn (Do Function) is a user-defined function that processes elements one at a time. The ParDo transform applies a DoFn to every element in a PCollection, in parallel across workers. You define a class that extends beam.DoFn and implement a process method that receives one element and yields zero or more output elements.

Bounded vs unbounded data

A bounded PCollection has a defined beginning and end. Examples: a CSV file, a database export, a set of Avro files in Cloud Storage. Beam processes bounded data as a batch job that starts, processes everything, and finishes.

An unbounded PCollection has no end. Examples: messages arriving on a Pub/Sub topic, events from a Kafka stream. Beam processes unbounded data as a streaming pipeline that runs continuously, using windowing and watermarks to group data into finite chunks for processing.

The same transforms work on both bounded and unbounded PCollections. Whether your pipeline runs as batch or streaming depends on the data source, not on your transform code.

Runners

A runner is the execution engine that takes your pipeline graph and runs it. Beam supports multiple runners:

• DirectRunner: runs on your local machine using threads. Use it for development and testing. It faithfully emulates Beam semantics, so bugs caught locally will behave the same on Dataflow.
• DataflowRunner: submits the pipeline to Google Cloud Dataflow for fully managed distributed execution.
• FlinkRunner / SparkRunner: runs on Apache Flink or Spark clusters for teams that already operate those engines.

You switch runners by changing a pipeline option. The pipeline code itself does not change. This portability is one of Beam’s core design goals.

Apache Beam and Dataflow

This is the single most important distinction for beginners to understand:

During development, you use the DirectRunner to test pipelines locally with small data samples. This gives you fast feedback without provisioning any cloud resources. When you are ready for production, you switch to the DataflowRunner. Dataflow provisions workers, distributes the work, autoscales based on load, and shuts everything down when the job finishes (for batch) or scales down during quiet periods (for streaming).

The pipeline code is identical in both cases. Only the runner option changes. For details on what Dataflow manages, see the Dataflow Overview.

A basic pipeline in Python

This pipeline reads order records from a CSV file in Cloud Storage, filters out cancelled orders, extracts the order total, and writes the results to a new file.

import apache_beam as beam

with beam.Pipeline() as pipeline:
    (
        pipeline
        | 'Read orders' >> beam.io.ReadFromText(
            'gs://my-shop-data/orders/*.csv',
            skip_header_lines=1
        )
        | 'Parse CSV rows' >> beam.Map(lambda line: line.split(','))
        | 'Remove cancelled' >> beam.Filter(lambda fields: fields[3] != 'CANCELLED')
        | 'Extract total' >> beam.Map(lambda fields: f"{fields[0]},{fields[4]}")
        | 'Write results' >> beam.io.WriteToText(
            'gs://my-shop-data/output/active-orders'
        )
    )

What each part does

with beam.Pipeline() as pipeline: creates a Pipeline object as a context manager. Everything inside the block builds the pipeline graph. The pipeline runs when the with block exits. Without extra options, this uses the DirectRunner by default.

| (pipe operator): applies a transform to the PCollection produced by the previous step. Each | takes the output PCollection from the left side and feeds it into the transform on the right side.

>> (right-shift operator): assigns a human-readable name to the step. The string before >> becomes the step label in pipeline graphs and job logs. Every step name must be unique within the pipeline.

ReadFromText: reads files from the given path and produces a PCollection of strings, one per line. The skip_header_lines=1 skips the CSV header row.

beam.Map: applies a function to each element. Here, it splits each line on commas to produce a list of fields.

beam.Filter: keeps only elements where the function returns True. This drops any row where the status field (index 3) is CANCELLED.

Second beam.Map: extracts the order ID (index 0) and total (index 4) into a comma-separated string for the output file.

WriteToText: writes the final PCollection to files in Cloud Storage.

from apache_beam.options.pipeline_options import PipelineOptions

options = PipelineOptions(
    runner='DataflowRunner',
    project='my-gcp-project',
    region='us-central1',
    temp_location='gs://my-shop-data/temp'
)

with beam.Pipeline(options=options) as pipeline:
    # Exact same pipeline code as above
    ...

The pipeline logic does not change. Only the options change. This is the portability that Beam provides.

When to use Apache Beam

Beam is a strong choice when your pipeline needs one or more of these characteristics:

• Batch ETL at scale. Reading files from Cloud Storage, transforming them, and loading the results into BigQuery. This is the most common ETL pattern on GCP.
• Real-time streaming. Consuming messages from Pub/Sub, transforming each message, and writing to BigQuery or Cloud Storage continuously. Beam handles windowing, late data, and exactly-once processing.
• Unified batch and streaming. When the same business logic needs to run on both historical data (batch) and live data (streaming), Beam lets you write that logic once.
• Complex multi-step transformations. Joining multiple data sources, applying business rules, computing aggregations, and writing to multiple sinks in a single pipeline.
• Portability. When you want the option to run the same pipeline on Dataflow today and on Flink or Spark later without rewriting.
• Local testing before cloud deployment. The DirectRunner lets you validate pipeline logic in seconds before submitting to Dataflow.

When Apache Beam may be overkill

Cases where Beam is probably not the right starting point:

• Small one-off scripts. Transforming a single file under a few hundred MB? A Python script with pandas or a BigQuery SQL query is faster to write and run.
• Low-throughput event handling. Processing a small number of events per second? A Pub/Sub subscription with a Cloud Run service or Cloud Function is simpler than a full streaming Beam pipeline.
• Standard patterns covered by Dataflow templates. Google provides pre-built Dataflow templates for common jobs like Pub/Sub-to-BigQuery and Cloud Storage-to-BigQuery. If a template fits, you skip writing Beam code entirely. See the Dataflow Overview for template options.
• Existing Spark jobs. If your team already has working Spark pipelines, migrating them to Dataproc is often simpler than rewriting in Beam. See Running Spark in GCP.
• SQL-only transformations. If everything you need is SQL on data already in BigQuery, scheduled queries or BigQuery scripting avoids the need for a pipeline framework entirely.

Apache Beam vs Dataflow vs Spark/Dataproc

These three names come up together constantly. Here is how they relate: