Azure Event Hubs Overview

What Azure Event Hubs actually does

Event Hubs solves a fundamental problem in streaming systems: producers emit events at unpredictable rates, but consumers need to process them at a steady pace without being overwhelmed. Event Hubs sits between them, storing incoming events in a durable log for a configurable retention period (1 to 90 days). Consumers read from the log at whatever rate they can handle, resuming exactly where they left off if they restart.

The key characteristics that make Event Hubs suitable for high-throughput scenarios:

• Throughput — millions of events per second; standard tier handles up to 1 MB/s inbound per partition
• Durability — events are replicated across availability zones within the region
• Retention — events are retained for 1–7 days (standard) or up to 90 days (premium/dedicated)
• Multiple consumers — multiple consumer groups can read the same event stream independently without interfering with each other
• Kafka compatibility — Event Hubs exposes a Kafka 1.0 compatible endpoint; existing Kafka clients work without modification

Inside Event Hubs: namespaces, hubs, and partitions

The Event Hubs resource hierarchy has three levels:

An Event Hubs namespace is the top-level resource — a container for one or more event hubs, with a network endpoint and shared capacity allocation. All event hubs in a namespace share the namespace’s throughput units.

An event hub (also called a topic in Kafka terms) is an individual event stream. One namespace can contain multiple event hubs — for example, one for order events, one for inventory updates, and one for user activity.

A partition is an ordered sequence of events within an event hub. Events sent to a partition are appended to the end of its log and remain there until the retention period expires. Partitions enable parallel processing — each consumer in a consumer group reads from a different set of partitions, allowing multiple machines to consume the stream simultaneously.

Namespace: eh-ns-mycompany
│
├── Event Hub: orders (4 partitions)
│   ├── Partition 0: [e1][e2][e5][e8]...
│   ├── Partition 1: [e3][e6][e9]...
│   ├── Partition 2: [e4][e7]...
│   └── Partition 3: [e10]...
│
├── Event Hub: inventory (2 partitions)
│   ├── Partition 0: [e1][e3]...
│   └── Partition 1: [e2][e4]...
│
└── Event Hub: user-activity (8 partitions)
    ├── Partition 0–7: ...

Event Hubs tiers

For most production workloads, Standard with auto-inflate enabled is the right choice. Auto-inflate automatically scales throughput units up (but not down) when utilisation reaches the limit, preventing throttling during traffic spikes.

Sending events to Event Hubs

# Python: send events to Event Hubs using the Azure SDK
from azure.eventhub import EventHubProducerClient, EventData
import json
from datetime import datetime

# Connection string from Azure portal > Event Hubs namespace > Shared Access Policies
connection_string = "Endpoint=sb://eh-ns-mycompany.servicebus.windows.net/;SharedAccessKeyName=..."
event_hub_name = "orders"

def send_order_events(orders: list):
    producer = EventHubProducerClient.from_connection_string(
        conn_str=connection_string,
        eventhub_name=event_hub_name
    )

    with producer:
        # Create a batch — batching is more efficient than sending one event at a time
        event_batch = producer.create_batch()

        for order in orders:
            event = EventData(json.dumps({
                "order_id":    order["id"],
                "customer_id": order["customer_id"],
                "total":       order["total"],
                "timestamp":   datetime.utcnow().isoformat()
            }))
            # Set partition key to route same customer's orders to same partition
            event_batch.add(event)

        producer.send_batch(event_batch)
        print(f"Sent {len(orders)} events to Event Hubs")

# Usage
sample_orders = [
    {"id": 1001, "customer_id": 42, "total": 149.99},
    {"id": 1002, "customer_id": 17, "total": 89.50},
    {"id": 1003, "customer_id": 42, "total": 229.00},
]
send_order_events(sample_orders)

Notice the use of create_batch() and send_batch(). Always batch events when possible — sending individual events one at a time adds connection overhead per event. A batch can contain up to 1 MB of events total.

Event Hubs Capture: archiving raw events to ADLS

Event Hubs Capture automatically saves a copy of all received events to Azure Blob Storage or ADLS Gen2 in Avro format. This gives you an immutable archive of your raw event stream — a bronze layer in the medallion architecture — without writing any consumer code.

Enable Capture when creating or modifying an event hub:

# Enable Capture on an existing event hub, archiving to ADLS Gen2
az eventhubs eventhub update \
  --resource-group rg-streaming \
  --namespace-name eh-ns-mycompany \
  --name orders \
  --enable-capture true \
  --skip-empty-archives true \
  --capture-interval 300 \
  --capture-size-limit 314572800 \
  --destination-name EventHubArchive.AzureBlockBlob \
  --storage-account "https://mydatalake.blob.core.windows.net" \
  --blob-container bronze \
  --archive-name-format "{Namespace}/{EventHub}/year={Year}/month={Month}/day={Day}/hour={Hour}/minute={Minute}/{PartitionId}"

Capture files land in ADLS in Avro format, partitioned by time. These files can be read directly by Spark or through the serverless SQL OPENROWSET function using Avro format support, enabling replay of the full historical event stream for reprocessing or auditing.

Creating an Event Hubs namespace and hub

# Create the namespace (Standard tier, 1 TU with auto-inflate up to 10)
az eventhubs namespace create \
  --resource-group rg-streaming \
  --name eh-ns-mycompany \
  --location eastus \
  --sku Standard \
  --enable-auto-inflate true \
  --maximum-throughput-units 10

# Create an event hub with 4 partitions and 7-day message retention
az eventhubs eventhub create \
  --resource-group rg-streaming \
  --namespace-name eh-ns-mycompany \
  --name orders \
  --partition-count 4 \
  --message-retention 7

# Create a consumer group for the Stream Analytics job
az eventhubs eventhub consumer-group create \
  --resource-group rg-streaming \
  --namespace-name eh-ns-mycompany \
  --eventhub-name orders \
  --name stream-analytics-cg

# Create a consumer group for the Spark Structured Streaming job
az eventhubs eventhub consumer-group create \
  --resource-group rg-streaming \
  --namespace-name eh-ns-mycompany \
  --eventhub-name orders \
  --name spark-cg