Amazon Kinesis Explained: Data Streams vs Firehose for Beginners

Amazon Kinesis in simple terms

Most applications produce events continuously: user clicks, order placements, sensor readings, application logs. The problem is getting that data reliably from wherever it is produced to wherever it needs to go. A data warehouse. A real-time dashboard. A downstream service that needs to react within seconds. Doing this at scale, with durability and replay, is harder than it sounds.

Kinesis solves three problems: reliably ingesting streaming data at scale, routing it to the right consumer or destination, and making records replayable when something goes wrong downstream. It is used for clickstream analytics, IoT telemetry pipelines, real-time dashboards, and event-driven architectures in AWS.

How Amazon Kinesis works

A Kinesis streaming pipeline follows a consistent six-step pattern from producer to destination:

1. Producer writes a record. Your application, an IoT device, or an AWS service sends a record (up to 1 MB) to a stream using the SDK or CLI. Each record includes a partition key that determines which shard receives it.
2. Record is ingested into a shard. Kinesis hashes the partition key and assigns the record to a shard. Records in a shard are ordered and durable from the moment they are written.
3. Record is retained in the stream. The record sits in the stream for the configured retention period (24 hours by default, up to 365 days). It is available for consumers to read at any point within that window.
4. Consumer reads the record. One or more independent consumers (Lambda functions, custom applications, or Managed Service for Apache Flink) read from the stream, each tracking its own position. Multiple consumers read the same record without interfering.
5. Consumer processes and routes the data. The consumer transforms, enriches, or filters the record and sends results to a downstream service: a database, another stream, or a notification service.
6. Data lands at the destination. Processed data ends up in storage (Amazon S3, Amazon Redshift) or is acted on in real time.

With Amazon Data Firehose, steps 4 and 5 are handled automatically. You skip writing consumer code entirely and Firehose delivers directly to S3, Redshift, OpenSearch, or Splunk.

Kinesis services at a glance

Kinesis is three services, not one. Each solves a different part of the streaming problem:

When to use Kinesis Data Streams

Choose Kinesis Data Streams when your use case needs any of these:

• Multiple independent consumers on the same data. A fraud detection service, an analytics pipeline, and a real-time dashboard can all read the same order-events stream simultaneously without interfering with each other.
• Strict ordering within a group of related events. All events for customer 456 arrive in the same shard in the order they were produced, which is essential for workflows that depend on event sequence.
• Replay and reprocessing. If a downstream service fails or a bug is found in your processing logic, consumers can replay events from any point within the retention window (up to 365 days).
• Custom real-time processing. You need to validate, transform, enrich, or route records within seconds of arrival, not after a 60-second buffer window.
• High-throughput event ingestion. A stream with 10 shards handles 10 MB/second of writes. Add shards to scale.

Good examples: clickstream processing where a real-time analytics consumer and a separate archiving consumer both read the same stream, IoT pipelines where device data drives both alerts and long-term storage, or any event-driven architecture where multiple services react to the same events independently.

When to use Amazon Data Firehose

Firehose is the right choice when your goal is reliable delivery to a storage destination with minimal operational overhead. You configure a destination and Firehose handles batching, compression, format conversion, and retry automatically.

Choose Firehose when:

• Your destination is S3, Redshift, OpenSearch, or Splunk. Firehose connects directly to these services and manages the entire delivery pipeline for you.
• A delay of 60 seconds or more is acceptable. Firehose buffers records before delivering. If sub-minute availability is not a requirement, Firehose is the simpler choice.
• You are building a data lake pipeline. Writing Parquet files to Amazon S3 partitioned by time is the standard first step in a data lake architecture, and Firehose handles this without any code.
• You want to avoid managing a consumer. Firehose removes the polling loop, error handling, and checkpoint management that a Kinesis Data Streams consumer requires.

When to use Managed Service for Apache Flink

Amazon Managed Service for Apache Flink is the right choice when you need stateful computations over a continuous stream: things that require tracking context across multiple records or across time windows.

Use it for:

• Windowed aggregations. Count events per user in 5-minute tumbling windows, or compute rolling averages over a 1-hour sliding window.
• Stream joins. Enrich incoming order events with customer profile data from another stream in real time.
• Continuous filtering and transformation. Reformat, mask PII, or route records to different destinations based on content, all in a persistent managed job.
• Anomaly detection. Track state across a time window to detect unusual patterns like a spike in error rates or unusual transaction amounts.

Flink is more complex to configure than Firehose and more powerful than a simple Lambda consumer. Use it when your transformation logic is too stateful for Lambda but you do not want to self-manage a Flink cluster on EMR.

Common use cases

• Clickstream analytics. Web or mobile events sent to Kinesis Data Streams, consumed by one Lambda for real-time dashboards and another for archiving to S3 via Firehose.
• Application log pipelines. EC2 instances or containers stream logs via the Kinesis Agent. Firehose delivers them to S3 or OpenSearch for monitoring and search.
• IoT telemetry. Thousands of devices send sensor readings to Kinesis Data Streams. Managed Service for Apache Flink aggregates readings per device per window and writes anomaly alerts to a downstream service.
• Event ingestion for near-real-time dashboards. App events flow through Kinesis Data Streams into a Lambda that writes micro-batches to Amazon Redshift every few minutes.
• Streaming data into S3 for a data lake. Firehose writes Parquet-formatted records partitioned by date. AWS Glue crawls the S3 prefix and makes it queryable. See Designing Data Pipelines for the full pattern.

Kinesis Data Streams in depth

Shards

A stream is divided into shards. Each shard is an independent ordered sequence with fixed throughput limits:

• Write: 1 MB/second or 1,000 records/second (whichever is lower)
• Read: 2 MB/second per shard (shared across all standard consumers)

You can split a shard to increase capacity or merge two shards to reduce it. Resharding takes a few minutes to complete and does not interrupt ongoing reads or writes.

Provisioned vs on-demand mode

Kinesis Data Streams offers two capacity modes:

• Provisioned mode: You specify the number of shards. You pay per shard per hour regardless of utilization. Best for predictable, steady throughput where you want cost control.
• On-demand mode: AWS scales shards automatically based on traffic. You pay per GB written and per GB read. Best for unpredictable workloads or early-stage applications.

Partition keys and ordering

When you write a record, you provide a partition key (any string). Kinesis hashes the key and assigns the record to a shard. Records with the same partition key always land in the same shard, in the order they were written.

Use a high-cardinality value like customer ID, device ID, or session ID so records distribute evenly across shards. Avoid constant or low-cardinality keys: if all records share the same partition key, they all go to one shard and you waste all other shard capacity.

Retention

• Default: 24 hours
• Extended: up to 7 days
• Long-term: up to 365 days (additional cost per GB per hour)

Retention determines how long records are available for consumers to read or replay. A consumer that falls behind, or a downstream system that fails, can re-read records from any point within the retention window. For production pipelines, extend retention to at least 7 days to give yourself headroom to investigate and replay without data loss.

Consumers and enhanced fan-out

Multiple consumer applications can read the same stream simultaneously. Each consumer tracks its own sequence number independently. Standard consumers share the 2 MB/second read throughput per shard, so five consumers each see around 400 KB/second from a single shard.

Enhanced fan-out gives each registered consumer its own dedicated 2 MB/second throughput per shard. This removes read-side contention when multiple high-throughput consumers need full throughput simultaneously. Enhanced fan-out also reduces end-to-end latency to around 70 ms by using a push model rather than polling.

CLI: create, put, and read records

# Create a stream in on-demand mode (AWS manages shard scaling)
aws kinesis create-stream \
  --stream-name order-events \
  --stream-mode-details StreamMode=ON_DEMAND

# Create a stream in provisioned mode with 2 shards
aws kinesis create-stream \
  --stream-name order-events \
  --shard-count 2

# Write a record with a high-cardinality partition key
aws kinesis put-record \
  --stream-name order-events \
  --partition-key "customer-123" \
  --data '{"order_id": 9876, "customer_id": 123, "total": 59.99, "ts": "2026-01-15T14:30:00Z"}'

# Read records: get a shard iterator, then pull records
SHARD_ITERATOR=$(aws kinesis get-shard-iterator \
  --stream-name order-events \
  --shard-id shardId-000000000000 \
  --shard-iterator-type TRIM_HORIZON \
  --query 'ShardIterator' \
  --output text)

aws kinesis get-records \
  --shard-iterator "$SHARD_ITERATOR" \
  --limit 10

TRIM_HORIZON reads from the oldest available record. Use LATEST for only new records, or AT_TIMESTAMP to start from a specific point in time.

# Check stream status and open shard count
aws kinesis describe-stream-summary \
  --stream-name order-events \
  --query 'StreamDescriptionSummary.{Status:StreamStatus,Shards:OpenShardCount}'

Lambda as a Kinesis consumer

AWS Lambda integrates natively with Kinesis Data Streams. Lambda polls the stream, batches records, and invokes your function. You do not manage shard iterators, polling loops, or checkpointing.

aws lambda create-event-source-mapping \
  --function-name ProcessOrders \
  --event-source-arn arn:aws:kinesis:us-east-1:123456789012:stream/order-events \
  --batch-size 100 \
  --starting-position LATEST \
  --bisect-batch-on-function-error

• —batch-size 100: Lambda receives up to 100 records per invocation
• —starting-position LATEST: start reading from new records only
• —bisect-batch-on-function-error: if a batch fails, split it in half and retry each half separately, isolating a single bad record without blocking the entire shard

Amazon Data Firehose delivery stream

Firehose is the simplest way to reliably deliver streaming data to a storage destination. Write records in; Firehose handles batching, compression, optional format conversion (JSON to Parquet or ORC), and delivery to the destination.

Supported destinations: Amazon S3, Amazon Redshift (staged via S3 with auto COPY), Amazon OpenSearch Service, Splunk, and any custom HTTPS endpoint.

Buffer settings

Firehose buffers incoming records before delivering them. You configure the threshold by size (1 MB to 128 MB) or by time (60 seconds to 900 seconds). When either threshold is reached, Firehose flushes to the destination. Plan for at least 60 seconds of latency before records appear at the other end.

Putting records to Firehose

# Put a single record
aws firehose put-record \
  --delivery-stream-name order-delivery-stream \
  --record '{"Data": "{\"order_id\": 1234, \"customer_id\": 456, \"total\": 89.99}"}'

# Put multiple records at once (more efficient at scale)
aws firehose put-record-batch \
  --delivery-stream-name order-delivery-stream \
  --records '[
    {"Data": "{\"order_id\": 1234, \"customer_id\": 456}"},
    {"Data": "{\"order_id\": 1235, \"customer_id\": 789}"}
  ]'

How a practical AWS streaming pipeline fits together

Pattern 1: Firehose to S3 to Redshift

This is the most common pattern for streaming data into a data warehouse at low operational cost. It trades sub-minute latency for simplicity and removes the need to write consumer code.

Application events
      |
      | (SDK put-record or put-record-batch)
      v
Amazon Data Firehose
(buffers 60s-15min, converts to Parquet)
      |
      v
Amazon S3
s3://my-lake/events/year=2026/month=01/day=15/hour=14/
      |
      | (AWS Glue crawler updates Data Catalog with new partitions)
      v
AWS Glue ETL job (nightly)
(clean, transform, deduplicate)
      |
      v
Amazon Redshift
(analysts and BI tools query here)

Firehose handles delivery and Parquet file creation. AWS Glue crawls the S3 prefix to keep the catalog current. A Glue ETL job loads transformed data into Redshift on a schedule. This pattern is covered in depth in ETL vs ELT and Designing Data Pipelines.

Pattern 2: Kinesis Data Streams to Lambda to downstream services

Use this pattern when you need real-time custom processing: enriching records, triggering alerts, writing to multiple destinations, or routing events based on content.

Application events
      |
      | (kinesis put-record, partition-key: customer ID)
      v
Kinesis Data Streams
(order-events, 4 shards, on-demand mode)
      |
      | (Lambda event source mapping, batch size 100)
      v
AWS Lambda: process-order-events
  Validate and enrich record
  Write operational data to DynamoDB
  Emit enriched event to SNS for fan-out
      |
      v
Amazon SNS -> downstream services (alerts, audit log, ML pipeline)

A second Lambda consumer on the same stream can independently archive all raw records to Amazon S3 in parallel. Each consumer tracks its own position and has no effect on the other.

Kinesis vs SQS vs SNS

These three services are often confused because they all move messages between systems. The differences are fundamental and the choice matters.

Decision framework:

• Multiple services need to independently consume the same events? Use Kinesis Data Streams (pull, consumers track their own position) or SNS (push, immediate fan-out).
• One worker handles each job and the message is discarded after processing? Use SQS.
• Trigger several Lambda functions or SQS queues from one event? Use SNS for fan-out.
• Need ordered events, replay, or high-throughput continuous ingestion? Use Kinesis Data Streams.

For a deeper look at how SNS works, see Amazon SNS Overview and SNS Messaging Model.