Streaming Pipelines in Azure

The anatomy of a streaming pipeline

All streaming pipelines share the same logical structure, regardless of which Azure services you use:

Event Sources          Ingestion              Processing             Serving
──────────────    ────────────────    ─────────────────────    ──────────────
IoT sensors    →  Azure Event Hubs →  Stream Analytics     →   Power BI
Web clickstream   Azure IoT Hub       Spark Structured         Azure SQL DB
Mobile events     Service Bus         Streaming                CosmosDB
Log streams       Kafka (HDI/ADB)     Azure Functions          ADLS Gen2
                                      Azure Data Explorer      Synapse pool

The ingestion layer buffers incoming events so the processing layer can consume them at its own pace. The processing layer applies transformations, filters, aggregations, and enrichments. The serving layer makes results available to downstream consumers — dashboards, APIs, databases, or additional pipelines.

Azure Event Hubs as the streaming ingestion layer

Azure Event Hubs is the standard ingestion buffer for high-throughput streaming in Azure. It accepts millions of events per second, retains them for 1–90 days, and allows multiple consumer groups to read the same stream independently. Think of it as a managed Kafka — and in fact, Event Hubs exposes a Kafka-compatible endpoint so Kafka producers and consumers work without code changes.

For a detailed explanation of Event Hubs’ internal model, see the Azure Event Hubs Overview page. For this page, we focus on how Event Hubs plugs into a streaming processing pipeline.

Azure Stream Analytics: SQL for streaming data

Stream Analytics is the simplest path to streaming analytics. You write a SQL-like query that continuously processes events from an input (Event Hubs, IoT Hub, or Blob Storage) and writes results to an output (Power BI, Azure SQL, Cosmos DB, ADLS, etc.). There is no cluster to manage — Stream Analytics is fully serverless.

-- Stream Analytics query: detect anomalous sensor readings
-- Input: 'SensorEvents' from Event Hubs
-- Output: 'AlertsDB' to Azure SQL Database

SELECT
    IoTHub.ConnectionDeviceId AS device_id,
    AVG(temperature)          AS avg_temp_5min,
    MAX(temperature)          AS max_temp_5min,
    MIN(temperature)          AS min_temp_5min,
    COUNT(*)                  AS reading_count,
    System.Timestamp()        AS window_end
INTO
    AlertsDB
FROM
    SensorEvents TIMESTAMP BY event_time
GROUP BY
    IoTHub.ConnectionDeviceId,
    TumblingWindow(minute, 5)
HAVING
    AVG(temperature) > 80
    OR MAX(temperature) > 95;

This query groups sensor readings into 5-minute tumbling windows and outputs device averages only when temperature is dangerously high. The query evaluates continuously — new results emit to AlertsDB every 5 minutes for any device exceeding the threshold.

Stream Analytics supports three window types:

• Tumbling window — fixed, non-overlapping windows (e.g., 5-minute blocks)
• Hopping window — overlapping windows with a fixed hop (e.g., compute the last 10 minutes every 5 minutes)
• Sliding window — a window that slides forward as new events arrive, evaluating continuously

Spark Structured Streaming: full code flexibility

When your streaming transformation logic is too complex for SQL (scoring events with an ML model, joining a stream with a slowly-changing dimension, maintaining complex stateful aggregations), use Spark Structured Streaming on Synapse or Databricks.

Spark Structured Streaming reads from Event Hubs or Kafka in micro-batches, processes with the full PySpark API, and writes to Delta Lake or other sinks with exactly-once semantics.

from pyspark.sql import SparkSession
from pyspark.sql.functions import col, from_json, window, avg, count
from pyspark.sql.types import StructType, StructField, StringType, DoubleType, TimestampType

spark = SparkSession.builder.getOrCreate()

# Define the schema of event payloads
event_schema = StructType([
    StructField("device_id",    StringType(),    True),
    StructField("temperature",  DoubleType(),    True),
    StructField("humidity",     DoubleType(),    True),
    StructField("event_time",   TimestampType(), True)
])

# Read from Azure Event Hubs
# Connection string comes from Spark config or Azure Key Vault secret scope
connection_string = spark.conf.get("spark.eventhubs.connectionString")

eh_conf = {
    "eventhubs.connectionString": sc._jvm.org.apache.spark.eventhubs.EventHubsUtils.encrypt(
        connection_string
    )
}

raw_stream = (
    spark.readStream
    .format("eventhubs")
    .options(**eh_conf)
    .load()
)

# Parse the binary body into JSON
parsed = raw_stream.select(
    from_json(col("body").cast("string"), event_schema).alias("data"),
    col("enqueuedTime")
).select("data.*", "enqueuedTime")

# Compute 5-minute averages with watermark for late data handling
aggregated = (
    parsed
    .withWatermark("event_time", "10 minutes")
    .groupBy(
        col("device_id"),
        window(col("event_time"), "5 minutes")
    )
    .agg(
        avg("temperature").alias("avg_temperature"),
        avg("humidity").alias("avg_humidity"),
        count("*").alias("reading_count")
    )
)

# Write to Delta Lake with checkpointing for fault tolerance
query = (
    aggregated.writeStream
    .format("delta")
    .outputMode("append")
    .option("checkpointLocation", "abfss://checkpoints@storage.dfs.core.windows.net/sensor_agg/")
    .start("abfss://silver@storage.dfs.core.windows.net/sensor_aggregates/")
)

query.awaitTermination()

The checkpoint location is critical for fault tolerance. Structured Streaming uses it to record exactly where in the Event Hubs partition stream it last successfully read. If the job restarts, it picks up from the checkpoint rather than reprocessing old events or missing events that arrived during downtime.

Lambda vs Kappa architecture

When you have both batch historical data and real-time streaming data, you face an architectural choice: the Lambda architecture or the Kappa architecture.

The Lambda architecture maintains two separate systems: a batch layer (ADF + Synapse processing yesterday’s data) and a speed layer (Stream Analytics or Structured Streaming processing today’s data in real time). Results are merged at the serving layer. This is reliable but complex — you maintain two codebases doing similar transformations.

The Kappa architecture uses only a streaming system. Historical reprocessing is done by replaying stored events through the same streaming pipeline. Delta Lake makes this practical in Azure: events are stored in the Event Hubs capture feature (which writes raw events to ADLS as Avro files), and when you need to reprocess history, you replay those captured files through the Structured Streaming job.

Azure Functions for lightweight event processing

For simple event transformations, Azure Functions with an Event Hubs trigger is the lightest-weight option. Each function invocation processes one batch of events from a partition:

# Azure Function triggered by Event Hubs
# Processes events and writes anomalies to Cosmos DB
import azure.functions as func
import json
import logging
from azure.cosmos import CosmosClient

def main(events: func.EventHubEvent, anomalies: func.Out[func.DocumentList]):
    results = []
    for event in events:
        body = json.loads(event.get_body().decode('utf-8'))

        if body.get('temperature', 0) > 90:
            anomaly = {
                "id": event.sequence_number,
                "device_id": body["device_id"],
                "temperature": body["temperature"],
                "detected_at": event.enqueued_time.isoformat(),
                "partition_key": body["device_id"]
            }
            results.append(anomaly)
            logging.warning(f"Anomaly detected: device {body['device_id']}, temp {body['temperature']}")

    if results:
        anomalies.set(func.DocumentList(results))

Azure Functions is best for simple, stateless event processing — filtering, enrichment, and routing. For stateful aggregations, windowing, or complex transformations, use Stream Analytics or Spark Structured Streaming.