How to Design Data Pipelines in AWS: Batch vs Streaming, Glue, Kinesis, and Redshift

Designing data pipelines in simple terms

A data pipeline is any automated system that moves data from a source to a destination and transforms it along the way. Every pipeline, no matter how complex, follows the same five-part model:

1. Source: where data originates. An application database, an API, a file drop in S3, or an event stream.
2. Transport: the service that carries data between stages. Kinesis Data Streams for real-time events, S3 for batch files.
3. Transform: the step that cleans, reshapes, or enriches data. Glue for managed Spark ETL, Redshift SQL or dbt for warehouse transforms, Lambda for lightweight event processing.
4. Destination: where transformed data lands for consumption. Redshift for analytics queries, S3 for a data lake, DynamoDB for operational lookups.
5. Monitoring: the observability layer that tells you when something breaks. CloudWatch alarms on job failures, DLQ message counts, and stream consumer lag.

A collection of scripts that moves data between systems is not a pipeline. A pipeline handles failures explicitly, can be rerun safely on any past date, and is monitored independently of the applications that produce the data. Without those three things, every failure becomes a manual incident.

How an AWS data pipeline works

Data enters from one or more sources, passes through a transport layer, gets transformed, and lands in a destination where analysts or applications consume it. Monitoring runs alongside every stage.

Source (app DB, API, file drop, event stream)
        |
        v
Transport (S3 for batch, Kinesis for streaming)
        |
        v
Transform (Glue, Lambda, Redshift SQL, or EMR)
        |
        v
Destination (Redshift, S3 data lake, or operational store)
        |
        v
Monitoring (CloudWatch alarms, dashboards, DLQ alerts)

At a high level, here is what each service does in a pipeline:

• S3: stores raw and processed files; acts as the staging layer and long-term raw archive for both batch and streaming pipelines
• Kinesis Data Streams: ingests high-volume event streams in real time, decoupling producers from consumers
• Kinesis Data Firehose: buffers and delivers streaming records to S3 or Redshift automatically, with no consumer code required
• AWS Glue: serverless managed Spark for ETL; reads from S3, transforms data, writes Parquet or ORC back to S3 or Redshift
• Lambda: lightweight event-driven transforms; starts in milliseconds but limited to 15 minutes and no Spark
• Redshift: the analytics warehouse; the destination for most reporting and BI, and increasingly where ELT transforms run via SQL
• Step Functions: orchestrates multi-step workflows with retry policies, timeouts, catch blocks, and a visual audit trail
• EventBridge: schedules pipeline triggers (EventBridge Scheduler) and routes events between services without coupling producers to consumers
• CloudWatch: collects metrics and logs from every service; the foundation for pipeline alarms and dashboards

The first decisions to make

Batch vs streaming

Batch pipelines process a bounded set of data on a schedule: hourly, daily, or weekly. A nightly job reads yesterday’s order files from S3, transforms them with Glue, and loads them into Redshift. Latency is measured in hours. Complexity is low.

Streaming pipelines process data continuously as it arrives. Events flow from Kinesis Data Streams into Lambda or Firehose and land in S3 or Redshift within seconds. Latency is measured in seconds. Complexity is significantly higher: you need to handle late data, out-of-order events, shard management, and consumer checkpointing. See the streaming pipelines with Kinesis guide for a full breakdown.

Where data should land

Redshift is the default destination for analytics and BI. If your pipeline powers dashboards, reports, or ad-hoc SQL queries by analysts, data should land in Redshift.

S3 is the right destination when you need a data lake: raw files in open formats (Parquet, ORC, JSON) that multiple tools and teams can read. S3 is also the staging layer for batch loads into Redshift, and the archive for replay and backfills.

Operational stores (DynamoDB, RDS, ElastiCache) are the destination only when downstream applications need low-latency lookups rather than analytical queries. Most teams need both: an analytical destination (Redshift or S3) and an operational store. See the data warehouses vs data lakes guide for a fuller comparison.

How transformations should run

Glue is the default for managed Spark ETL. It reads files from S3, applies Python or Scala transformations, and writes structured output back to S3 or Redshift without cluster provisioning. Good for large files and complex joins.

Redshift SQL and dbt-style ELT work well when data is already in Redshift and the transform is expressible in SQL. Load raw data first, then transform in-warehouse. This ELT approach avoids a separate compute layer entirely and lets analysts iterate on transformation logic without engineering involvement.

Lambda is right for lightweight per-record transforms: parsing, filtering, routing, or enriching individual events. Lambda starts in milliseconds, but it has a 15-minute timeout and no Spark. It is not designed for processing large files or complex multi-table joins.

EMR gives you managed Spark and Hadoop clusters with more control than Glue: custom Spark configurations, support for Hadoop-native tools (Hive, Presto, HBase), and the ability to run existing PySpark code with minimal changes. Use EMR when Glue’s managed environment is not flexible enough, or when you are migrating an existing Spark workload from on-premises.

When to use orchestration

Step Functions is for multi-step workflows with dependencies: run a Glue job, wait for it to finish, run a Redshift COPY, run dbt, send a success notification. Step Functions gives you retry policies per step, catch blocks for expected failures, and a visual console showing exactly which step failed.

EventBridge Scheduler is for single-step triggers: run this Glue job at 02:00 UTC every day. If your pipeline has one meaningful step, EventBridge Scheduler is all you need. Step Functions is overhead.

Glue vs Lambda vs EMR vs Redshift SQL

The most common confusion when designing an AWS data pipeline is choosing the transformation engine. Here is a direct comparison to help you decide:

If you are starting from scratch with no existing Spark code, Glue is the default for file-based ETL and Redshift SQL is the default for warehouse-native transforms. See the ETL vs ELT guide for a deeper comparison.

Reference architectures

Batch pipeline: S3 → Glue → Redshift → BI

This is the most common production batch pattern. Raw data lands in S3, a Glue job transforms it, and the results are loaded into Redshift for BI queries.

When it fits: daily reporting, data warehouse refreshes, log analytics, partner file ingestion. Use this pattern when one-hour-old data is acceptable.

[Source systems: app DB, partner SFTP, API exports]
           |
           | (file drops, JDBC extracts)
           v
  S3 raw zone
  s3://my-lake/raw/orders/year=2026/month=05/day=11/

           |
           | (EventBridge Scheduler at 02:00 UTC)
           v
  AWS Glue ETL Job
  - Read JSON from S3 raw zone
  - Clean, validate, cast types
  - Write Parquet to S3 processed zone

           |
           v
  S3 processed zone
  s3://my-lake/processed/orders/year=2026/month=05/day=11/

           |
           | (Redshift COPY command)
           v
  Redshift: staging.orders_raw

           |
           | (dbt run or Step Functions SQL step)
           v
  Redshift: analytics.daily_order_summary

           |
           v
  BI tools (QuickSight, Tableau)

Why this design: S3 acts as both the landing zone and the permanent raw archive. Glue reads and transforms without touching the source system again. The Redshift COPY command is faster than row-by-row inserts because it reads Parquet files in parallel across nodes. See the loading data into Redshift guide for COPY patterns and best practices.

Where failures usually happen: The Glue job is the most common failure point. Bad schema from upstream, type mismatches, or S3 permission issues all show up here. The Redshift COPY fails when Parquet schema does not match the table. The dbt or SQL step fails when Redshift is under load.

Reruns and backfills: Pass the date as a parameter to the Glue job. The pipeline writes to date-partitioned S3 prefixes and uses MERGE in Redshift, so you can rerun for any past date. The output overwrites the previous partition without affecting others.

# Trigger the Glue job for a specific date (backfill-friendly)
aws glue start-job-run \
  --job-name process-daily-orders \
  --arguments '{
    "--date": "2026-05-11",
    "--source": "s3://my-lake/raw/orders/year=2026/month=05/day=11/",
    "--target": "s3://my-lake/processed/orders/"
  }'

# Load to Redshift after Glue succeeds
aws redshift-data execute-statement \
  --cluster-identifier my-cluster \
  --database mydb \
  --sql "COPY staging.orders_raw FROM 's3://my-lake/processed/orders/year=2026/month=05/day=11/' IAM_ROLE 'arn:aws:iam::123456789012:role/RedshiftRole' FORMAT AS PARQUET;"

Streaming pipeline: Kinesis → Lambda or Firehose → S3/Redshift

This pattern processes events in near-real time as they arrive. Two common sub-paths: Lambda-based processing and Firehose direct delivery.

When it fits: fraud detection, live operational dashboards, event-driven monitoring, real-time inventory updates. Use streaming when latency must be seconds, not hours.

[Application servers / APIs]
         |
         | (kinesis put-record)
         v
Kinesis Data Streams  (order-events, 4 shards)

         |
         |--- Path A: Lambda processing
         |         |
         |         v
         |    Lambda (process-order-events)
         |    - Validate and enrich each record
         |    - Buffer to S3 or send to Firehose
         |    - DLQ (SQS) for failed records
         |
         |--- Path B: Firehose direct delivery
               |
               v
         Kinesis Data Firehose
         - Buffer for 60s or 5 MB
         - Write Parquet to S3 automatically

         |
         v
S3 landing zone  (s3://my-lake/kinesis-landing/orders/)

         |
         | (micro-batch COPY every 5 minutes)
         v
Redshift: streaming.orders_realtime

Why this design: Kinesis decouples the producer (application) from the consumer. Firehose is the simpler path: no Lambda code needed for delivery, just configuration. Lambda is right when you need per-record validation, enrichment, or routing before records reach S3.

Where failures usually happen: Lambda failures on malformed events (always configure a DLQ). Kinesis iterator age growing when a consumer falls behind shard throughput. Redshift COPY failing when the landing prefix has files from multiple incompatible schema versions.

Reruns and backfills: Keep raw Kinesis records archived to S3. Kinesis retains records for 24 hours to 365 days depending on configuration. For longer replay windows, archive raw events to S3 and replay from there. See the streaming pipelines with Kinesis guide for shard management and replay details.

# Create a Kinesis Data Stream
aws kinesis create-stream \
  --stream-name order-events \
  --shard-count 4

# Put a record to the stream
aws kinesis put-record \
  --stream-name order-events \
  --partition-key "order-123" \
  --data '{"orderId":"order-123","amount":49.99,"placed_at":"2026-05-11T10:30:00Z"}'

When to use this approach

• Daily reporting and BI. A batch pipeline loads yesterday’s data into Redshift overnight. Analysts query fresh tables each morning with QuickSight or Tableau.
• Data warehouse refresh. Multiple source systems export files to S3. A Glue job consolidates and transforms them nightly. Redshift becomes the single analytical source of truth.
• Log analytics. Application logs land in S3 via CloudWatch Logs or a log agent. A Glue job partitions and converts them to Parquet for Athena or Redshift queries.
• Fraud and event-driven monitoring. A Kinesis pipeline processes payment events within seconds, flags anomalies, and writes scores to an operational store and Redshift for deeper analysis. See the event-driven architectures guide for how to structure these flows.
• Operational dashboards. A streaming pipeline feeds a Redshift table that refreshes every few minutes. A BI tool queries it for near-real-time operational views.
• Hybrid batch and streaming. A streaming pipeline writes raw events to S3 and a live Redshift table for operational dashboards. A separate nightly batch job runs heavier aggregations on the same underlying data. Streaming gives low latency; batch gives completeness and lower cost for heavy computation.

When not to use this approach

• Transforms are simple SQL. If your pipeline is “load files into Redshift, then run SQL to clean and reshape,” Redshift scheduled queries or dbt handle the transform step without Glue or a separate compute service. The simpler the transform, the less you need a full ETL layer.
• Stakeholders only need daily or hourly freshness. Avoid Kinesis streaming when a scheduled EventBridge trigger and a Glue job give everyone what they need. Streaming adds 24/7 compute cost and operational complexity for latency that may not matter.
• One-step scheduled loads. If your pipeline is a single Redshift COPY from S3, you do not need Step Functions. An EventBridge Scheduler rule that triggers the COPY is the right level of complexity.
• Small data volumes. Glue and EMR have startup overhead. A Glue job takes 2 to 3 minutes to initialize. For small files (a few MB), a Lambda function reading and writing to S3 is faster, cheaper, and simpler.
• Downstream apps need low-latency serving. Redshift is an analytical warehouse, not a transactional database. If downstream applications need millisecond lookups on pipeline results, write to DynamoDB or ElastiCache, not just Redshift.

Idempotency and safe reruns

Every pipeline step should be idempotent: running it twice on the same input produces the same result as running it once. This is non-negotiable because retries are guaranteed to happen in distributed systems.

• Write with overwrite, not append. In Spark or Glue, use mode(‘overwrite’) instead of mode(‘append’). A retry replaces the previous partial output rather than adding duplicates.

• Use MERGE (UPSERT) in Redshift instead of INSERT. A MERGE statement inserts new rows and updates existing ones by key. Running it twice with the same input data produces the same result: no duplicates.

• Partition by processing date. A job that writes to a date partition can be rerun for any past date without affecting other partitions. Overwrite only the target partition.

• Track processed watermarks for streaming. Store the last-processed sequence number or timestamp in DynamoDB or S3. On restart, resume from that checkpoint rather than reprocessing from the beginning.

-- Idempotent Redshift MERGE: insert new orders, update changed ones
MERGE INTO analytics.orders AS target
USING staging.orders_raw AS source
ON target.order_id = source.order_id
WHEN MATCHED THEN
    UPDATE SET
        amount = source.amount,
        status = source.status,
        updated_at = source.updated_at
WHEN NOT MATCHED THEN
    INSERT (order_id, customer_id, amount, status, placed_at, updated_at)
    VALUES (source.order_id, source.customer_id, source.amount,
            source.status, source.placed_at, source.updated_at);

Error handling and dead-letter queues

Pipelines fail. Sources send bad data. Downstream systems are temporarily unavailable. Designing for failure from the start is cheaper than retrofitting it after the first incident.

• DLQ on every Lambda event source. For Lambda consumers reading from Kinesis or SQS, configure a dead-letter queue. Failed records after N retries go to the DLQ (an SQS queue or S3 prefix) instead of blocking the stream. Set a CloudWatch alarm on NumberOfMessagesSent for the DLQ and alert immediately when it exceeds zero.

• Poison pill handling in Glue and Spark. A single malformed record should not kill an entire Glue job. Use try/except in PySpark UDFs, or write rejected records to a separate S3 quarantine prefix for review rather than failing the whole job.

• Step Functions for workflow-level failures. Step Functions gives you retry policies and exponential backoff per step, catch blocks for expected failures (Glue errors, Redshift timeouts), and a visual console to see exactly which step failed. This is far more useful than reading raw logs when debugging a multi-step pipeline.

• Kinesis shard blocking. If a Lambda consumer fails on a Kinesis shard and keeps retrying, it blocks all subsequent records in that shard. Configure bisectBatchOnFunctionError on the event source mapping to split failing batches in half and isolate the bad record faster.

• Alerting on failures. Set CloudWatch alarms on Glue job failure metrics, Lambda error rates, and DLQ message counts. An on-call engineer should be paged within minutes of a pipeline failure, not discover it from a stale dashboard the next morning.

Monitoring and alerting with CloudWatch

Every pipeline component emits metrics to CloudWatch. Monitor pipeline health separately from application health. A pipeline that stops processing silently is worse than one that fails loudly.

Key metrics to alarm on:

• Glue: glue.driver.aggregate.numFailedTasks, job duration, bytes read and written
• Lambda: Errors, Throttles, Duration, ConcurrentExecutions
• Kinesis: GetRecords.IteratorAgeMilliseconds (consumer lag), PutRecord.Success
• SQS DLQ: NumberOfMessagesSent; alarm immediately when this exceeds 0
• Redshift: HealthStatus, CPUUtilization, query queue depth
• Step Functions: ExecutionsFailed, ExecutionThrottled

# Create a CloudWatch alarm for Glue job failures
aws cloudwatch put-metric-alarm \
  --alarm-name "glue-orders-job-failure" \
  --alarm-description "Alert when daily orders Glue job fails" \
  --metric-name "glue.driver.aggregate.numFailedTasks" \
  --namespace "Glue" \
  --dimensions Name=JobName,Value=process-daily-orders \
  --period 300 \
  --statistic Sum \
  --comparison-operator GreaterThanThreshold \
  --threshold 0 \
  --evaluation-periods 1 \
  --alarm-actions arn:aws:sns:us-east-1:123456789012:pipeline-alerts

Backfills and replay

Every pipeline will need to reprocess historical data at some point: after a bug fix, a schema change, or a new business requirement. Design for this from day one.

• Keep raw data in S3 before any transformation. If you only keep transformed data and a transformation bug ships, reprocessing means re-extracting from the source system, which may not be possible weeks later. Raw S3 archives are cheap insurance.

• Parameterize Glue jobs by date. Pass —date as a job argument. A job that reads from s3://my-lake/raw/orders/year=YYYY/month=MM/day=DD/ can be rerun for any past date without code changes.

• Write to date partitions and overwrite. A backfill run overwrites the target partition for the specific date. Other partitions are untouched.

• Test the backfill path early. Run the pipeline for a past date in your staging environment before production. The first time you need a backfill is not the time to discover it does not work.

• For streaming backfills, replay from S3. Archive raw Kinesis events to S3 as they arrive. For backfills beyond Kinesis retention, replay from the S3 archive rather than from the stream directly.

Data quality and schema evolution

Pipelines break when upstream producers change event structure without notice. Build schema validation into the ingestion step rather than discovering changes when a Glue job throws a type error.

• Validate schemas at ingestion. Check incoming records against a known schema (JSON Schema, Avro, or a Glue Data Catalog definition). Write non-conforming records to a quarantine prefix or DLQ rather than letting them corrupt downstream tables.

• Plan for schema evolution. Add new fields as nullable in Redshift. Version your S3 prefixes when major schema changes break backward compatibility. Test schema changes in staging before production.

• Use the Glue Data Catalog as your schema registry. The catalog tracks table schemas for S3-backed datasets. Glue jobs and Athena queries read schema definitions from the catalog rather than inferring them each time. This catches mismatches earlier and prevents silent type coercions.

• Track row counts and null rates. A Glue job that loads zero rows is a silent failure. Log input row count and output row count as CloudWatch metrics. Alarm when the ratio falls outside expected bounds.

Cost and complexity trade-offs

• Batch is cheaper than streaming for the same workload. A Glue job that processes 10 GB in 10 minutes costs far less than a Kinesis stream plus Lambda plus Firehose running 24/7 to deliver the same data with a 60-second buffer.
• Glue has a 2 to 3 minute startup overhead. For very small jobs (under 100 MB), startup time dominates. A Lambda function is faster and cheaper for small, frequent processing tasks.
• Kinesis shards have a fixed hourly cost. Each shard costs roughly $0.015/hour. Provision based on actual throughput (1 MB/s write or 1,000 records/s per shard) and scale down when throughput drops.
• Step Functions charges per state transition. A simple two-step pipeline with 1,000 daily executions will cost cents. A complex workflow with many parallel branches at high frequency should be cost-checked before production.
• Redshift Serverless eliminates idle cluster cost. If your pipeline loads data nightly and nobody queries Redshift during the day, Redshift Serverless charges only for query time and compute used. Compare against a provisioned cluster at your actual utilisation before choosing.

Architecture decisions checklist

Work through these before building a new pipeline:

1. Batch or streaming? Does the business need seconds of latency, or is hourly acceptable? Default to batch.
2. Where does data land? Redshift for analytics and BI, S3 for a data lake or long-term raw archive, operational store only when downstream apps need low-latency serving.
3. Which transformation engine? Glue for large S3 ETL, Redshift SQL or dbt for warehouse ELT, Lambda for lightweight per-record processing, EMR for existing Spark workloads.
4. Orchestration needed? Multi-step with dependencies → Step Functions. Single-step scheduled → EventBridge Scheduler.
5. Idempotency strategy? Overwrite S3 partitions by date. MERGE in Redshift by unique key. Checkpoint watermarks for streaming.
6. Failure handling? DLQ configured on all Lambda event sources. Poison pill handling in Glue jobs. Step Functions retry policy per step.
7. Monitoring? CloudWatch alarms on job failures, DLQ message count, and Kinesis consumer lag before go-live.
8. Backfill tested? Run the pipeline for a past date partition in staging to confirm the backfill path works before production.