Event Hubs Messaging Model: Partitions and Consumer Groups

Partitions: ordered logs within an event hub

A partition in Event Hubs is a single ordered, durable log. Events are appended to the end of the log as producers send them. The log is immutable — you cannot delete or modify individual events. Events expire automatically after the retention period (1–90 days depending on tier).

Each event in a partition has:

• Sequence number — a monotonically increasing integer unique within the partition
• Offset — the byte position within the partition log
• Enqueued time — the UTC timestamp when the event was received by Event Hubs
• Partition key — an optional string set by the producer to control which partition an event is routed to
• Body — the event payload (up to 1 MB, or up to 1 MB per batch for standard tier)

Partition 0:
offset:  [0]  [45]  [92]  [134]  [178]  [223]  ...
seq_no:  [0]  [1]   [2]   [3]    [4]    [5]    ...
data:    [e1] [e2]  [e3]  [e4]   [e5]   [e6]   ...
time:    [T1] [T1]  [T2]  [T3]   [T3]   [T4]   ...

The partition key is hashed to determine which partition receives the event. Events with the same partition key always go to the same partition — guaranteeing ordering within the scope of that key. For example, routing all events for the same customer_id to the same partition ensures that a consumer processing that partition sees that customer’s events in order.

Consumer groups: independent views of the stream

A consumer group is a view of an event hub. Each consumer group maintains its own independent set of offsets — one per partition. This means two consumer groups can read the same event hub simultaneously, and each processes every event independently. Changes to one group’s offset do not affect the other.

Event Hub: orders (4 partitions)
│
├── Consumer Group: stream-analytics-cg
│   ├── Partition 0 offset: 892 (Stream Analytics last read here)
│   ├── Partition 1 offset: 1103
│   ├── Partition 2 offset: 1044
│   └── Partition 3 offset: 967
│
├── Consumer Group: spark-structured-streaming-cg
│   ├── Partition 0 offset: 880 (Spark is slightly behind)
│   ├── Partition 1 offset: 1095
│   ├── Partition 2 offset: 1021
│   └── Partition 3 offset: 941
│
└── Consumer Group: $Default (built-in, usually unused in production)
    └── Partition 0–3 offsets: not tracked (no active consumers)

In this diagram, Stream Analytics and a Spark Structured Streaming job both read the same orders event hub independently. Stream Analytics is slightly ahead because it processes in smaller micro-batches. Neither affects the other’s position.

Checkpointing: recording your position

Checkpointing is how a consumer records its current position in each partition. When a consumer checkpoints offset 500 in partition 0, it is declaring “I have successfully processed all events up to offset 500 in partition 0.” If the consumer restarts, it reads the checkpoint and resumes from offset 501 rather than starting over.

In the Azure Event Hubs SDK, checkpoints are stored in Azure Blob Storage. The EventProcessorClient manages checkpointing automatically — you call update_checkpoint() inside your event handler:

from azure.eventhub.aio import EventHubConsumerClient
from azure.eventhub.extensions.checkpointstoreblobaio import BlobCheckpointStore
from azure.storage.blob.aio import ContainerClient
import asyncio
import json

async def process_events(partition_context, events):
    """Process a batch of events from one partition"""
    for event in events:
        body = json.loads(event.body_as_str())
        print(
            f"Partition {partition_context.partition_id} | "
            f"Offset {event.offset} | "
            f"order_id={body.get('order_id')}"
        )

    # Checkpoint after processing all events in the batch
    # This records the offset of the last event in the batch
    await partition_context.update_checkpoint()
    print(f"Checkpointed partition {partition_context.partition_id}")


async def main():
    # Store checkpoints in Azure Blob Storage
    checkpoint_store = BlobCheckpointStore.from_connection_string(
        conn_str="DefaultEndpointsProtocol=https;AccountName=...",
        container_name="event-hub-checkpoints"
    )

    consumer = EventHubConsumerClient.from_connection_string(
        conn_str="Endpoint=sb://eh-ns-mycompany.servicebus.windows.net/;...",
        consumer_group="spark-structured-streaming-cg",
        eventhub_name="orders",
        checkpoint_store=checkpoint_store
    )

    async with consumer:
        # This call blocks and continuously receives events
        await consumer.receive_batch(
            on_event_batch=process_events,
            max_batch_size=100,
            max_wait_time=5  # Wait up to 5 seconds for a batch to fill
        )

asyncio.run(main())

The checkpoint store in Blob Storage contains one small JSON blob per partition per consumer group. Each blob records the offset of the last successfully processed event. This is what makes the consumer fault-tolerant — crash at any time, and the next instance resumes from the last checkpoint.

Partition assignment with multiple consumer instances

When you run multiple instances of a consumer application (for example, 4 instances consuming a 4-partition event hub), the EventProcessorClient automatically distributes partitions among the running instances using a distributed lease mechanism stored in Blob Storage.

Each instance tries to claim partitions by writing lease blobs. If an instance crashes or stops renewing its leases, the other instances detect the orphaned leases and take over those partitions. This provides automatic failover with minimal event reprocessing (from the last checkpoint).

4 partitions, 4 consumer instances:

Instance 1 → Partition 0  (offset: 892)
Instance 2 → Partition 1  (offset: 1103)
Instance 3 → Partition 2  (offset: 1044)
Instance 4 → Partition 3  (offset: 967)

Instance 2 crashes:

Instance 1 → Partition 0, Partition 1  (Partition 1 resumed from offset 1103)
Instance 3 → Partition 2
Instance 4 → Partition 3

The number of consumer instances that can run in parallel is bounded by the partition count. More instances than partitions means idle instances waiting for work. Plan your partition count at event hub creation time to match your expected consumer parallelism.

Starting positions

When a consumer group connects to an event hub for the first time (no checkpoint exists), you must tell it where to start reading. Three options:

from azure.eventhub import EventPosition

# Start from the beginning of the retention window
starting_positions = {
    partition_id: EventPosition("@latest")
    for partition_id in ["0", "1", "2", "3"]
}
# or
starting_positions = {
    "0": EventPosition(offset="-1"),   # beginning
    "1": EventPosition(offset="-1"),
}

# Start from a specific timestamp (process only events from the last 2 hours)
from datetime import datetime, timedelta, timezone
two_hours_ago = datetime.now(timezone.utc) - timedelta(hours=2)
starting_positions = {
    partition_id: EventPosition(two_hours_ago)
    for partition_id in ["0", "1", "2", "3"]
}

Delivery guarantees: at-least-once

Event Hubs provides at-least-once delivery. This means every event will be delivered to a consumer at least once, but may be delivered more than once (if the consumer processes an event but crashes before checkpointing). It does not provide exactly-once delivery.

To handle duplicates, your downstream storage or processing must be idempotent:

• Idempotent writes — use UPSERT (INSERT OR UPDATE) instead of INSERT when writing to databases; the same event processed twice produces the same result
• Deduplication — include the event’s sequence number or a unique event_id in your schema and deduplicate before writing to the final store
• Delta Lake merge — use MERGE INTO on a unique key when writing Spark streaming output to Delta Lake