Event-Driven Systems in AWS: Service Selection, Production Patterns, and Architecture Trade-offs

Simple explanation

Think of a fire station dispatch center. When a 911 call comes in, the dispatcher broadcasts the incident over the radio. Every relevant unit (fire trucks, ambulances, police) hears the broadcast and responds based on its own role. The dispatcher does not call each unit individually or wait for all of them to confirm. If a unit misses the transmission, the message is re-sent. The dispatcher keeps a log of every call and its outcome.

AWS event-driven systems work the same way. A service (the producer) announces that something happened. A broker receives the announcement and routes it to the right consumers. Each consumer handles the event independently. If a consumer fails, the system retries and parks the event in a dead-letter queue instead of losing it silently.

The key insight for system designers: the producer does not know who is listening, how many consumers exist, or what they do with the event. That decoupling is what lets event-driven systems scale and evolve independently. Adding a new consumer requires no change to the producer at all.

How event-driven systems work in AWS

Every event-driven system in AWS follows the same five-stage flow regardless of which services you use:

1. Producer emits an event. An order service writes the order to DynamoDB and publishes an order.placed event. Its job is done. It does not wait or care what happens next.
2. Broker or router receives the event. EventBridge, SNS, SQS, or Kinesis receives the event and decides what to do with it based on rules, subscriptions, or shard assignments.
3. Event is distributed to consumers. EventBridge rules filter by content and route to multiple targets. SNS pushes to all subscribers simultaneously. SQS holds the event until a consumer is ready. Kinesis writes it to a shard for ordered, replayable processing.
4. Consumers process the event independently. Each consumer (typically a Lambda function, ECS task, or long-running worker) handles its copy of the event without knowing about other consumers.
5. Retries, DLQs, and observability handle failures. Failed consumers are retried automatically. After retry limits are exhausted, events route to a dead-letter queue. Distributed tracing with X-Ray connects the full event path across services in one view.

Producer (Lambda / EC2 / ECS)
    │
    └─▶ EventBridge  (routes by rule)
              │
              ├─▶ SNS Topic  (fans out to all subscribers)
              │       ├─▶ SQS Queue A  ──▶  Lambda consumer A  ──▶  DLQ A
              │       ├─▶ SQS Queue B  ──▶  Lambda consumer B  ──▶  DLQ B
              │       └─▶ SQS Queue C  ──▶  ECS worker C       ──▶  DLQ C
              │
              └─▶ Kinesis Stream  (ordered, high-volume)
                      └─▶  Lambda consumer D (per shard)

Events vs commands

This distinction shapes every design decision in event-driven systems. An event records that something happened. The producer broadcasts it and does not know or care who reacts. A command tells a specific system to do a specific thing. The sender has a dependency on the receiver being available and responding correctly.

Both exist in well-designed systems. Default to events for cross-service side effects. Use direct calls for operations where the caller genuinely needs an immediate result: input validation, synchronous user responses, payment confirmation before showing “order confirmed”.

AWS event services compared: choosing the right service

This table is the core of service selection. These services overlap and are frequently combined. The right choice depends on what behavior you specifically need from the broker layer.

When to use event-driven architecture in AWS

These are the scenarios where event-driven architecture pays off clearly:

• Fan-out side effects after a primary action. An order is placed. The inventory service, email service, and analytics pipeline each need to react. SNS to SQS fan-out handles this cleanly without the order service knowing about any of them. Adding a fraud-detection service later means adding one SNS subscription with no changes to the order service.
• Async background jobs. Sending confirmation emails, resizing uploaded images, generating reports. Any work that does not need to complete before returning a response to the user belongs in an async queue. SQS plus Lambda is the standard pattern.
• Decoupling slow or unreliable downstream systems. If a third-party CRM, payment gateway, or legacy internal service is slow or intermittently unavailable, SQS buffers events and retries, isolating the upstream service from downstream instability.
• Reacting to AWS service events. EventBridge receives native events from over 200 AWS services: EC2 state changes, S3 object creation, RDS snapshots, CodePipeline status updates. No polling, no custom glue code required.
• SaaS and webhook integrations. EventBridge has native integrations with Stripe, Zendesk, Datadog, and other SaaS providers, making it the cleanest way to react to external system events without writing custom webhook handlers.
• High-volume streaming workloads. Clickstream analytics, IoT sensor data, financial market feeds, application log aggregation. Any continuous high-volume data stream belongs in Kinesis rather than an SQS queue. See the data pipeline design guide for how Kinesis fits into broader pipeline architectures.

When NOT to use event-driven architecture

• User-facing flows that need immediate results. “Is this username taken?” or “Was my payment accepted?” need synchronous responses. Emitting an event and polling for a reply event is complex and slow. Use a direct API call.
• Strict transactional consistency. EDA is inherently eventually consistent. There is always a window where different services see different data. Domains like financial transactions, inventory reservations, and seat booking often need strong consistency guarantees that pure event-driven patterns make significantly harder to achieve.
• Small or simple systems. Two services calling each other directly are easier to understand, test, and debug than the same two services connected through an event bus with queues, DLQs, and retry policies. Add event-driven architecture when coupling pain actually exists, not as a speculative architectural investment.
• Teams without observability and retry discipline in place. EDA failures are silent by default. If distributed tracing, structured logging, and DLQ monitoring are not already operational, the complexity will outweigh the architectural benefits. Build observability infrastructure first.

Common production patterns in AWS

These are the patterns that appear repeatedly in well-designed AWS event-driven systems. Each solves a specific problem and has a specific use case.

SNS to SQS fan-out

What it is: An SNS topic has multiple SQS queue subscriptions. One message published to the topic delivers a copy to every subscribed queue. Each queue has its own independent consumer and dead-letter queue.

When to use it: Any time one event needs to trigger multiple independent downstream services where each consumer needs its own buffering, retry handling, and failure isolation.

Concrete example: An order.placed event publishes to an SNS topic. Three SQS queues subscribe: one for inventory, one for email, one for analytics. If the email Lambda fails and retries three times before hitting the DLQ, inventory and analytics processing are completely unaffected.

SNS: order-placed
  ├── SQS: inventory-queue  ──▶  Lambda  ──▶  DLQ: inventory-dlq
  ├── SQS: email-queue      ──▶  Lambda  ──▶  DLQ: email-dlq
  └── SQS: analytics-queue  ──▶  Lambda  ──▶  DLQ: analytics-dlq

EventBridge rule-based routing

What it is: EventBridge receives events and applies JSON-based rules to route specific events to specific targets. Rules can filter on event source, type, and any field in the event payload.

When to use it: When different event types or values need to go to different consumers, when you want to react to AWS service events without polling, or when you need to route SaaS events (Stripe payments, Zendesk tickets) into your AWS system without custom webhook infrastructure.

Concrete example: Orders over $500 route to a priority fulfillment queue; orders under $500 route to the standard queue. Two EventBridge rules handle the split. No code changes are required when the threshold changes.

{
  "source": ["myapp.orders"],
  "detail-type": ["order.placed"],
  "detail": {
    "amount": [{ "numeric": [">", 500] }]
  }
}

SQS buffering in front of Lambda or ECS workers

What it is: A producer writes directly to an SQS queue. Lambda (or an ECS worker) polls the queue and processes messages in batches. SQS absorbs traffic spikes; the consumer scales based on queue depth.

When to use it: When a downstream service cannot absorb peak load directly: image processing, report generation, bulk data exports, or any workload with uneven traffic and a non-trivial processing time per message.

Concrete example: A user uploads a video. The upload handler writes a job message to an SQS queue. Lambda processes the queue with a batch size of 10 and a visibility timeout set longer than the maximum processing time. Lambda scales automatically based on queue depth. Failed transcoding jobs move to the DLQ for manual inspection.

Kinesis for ordered, high-throughput streams

What it is: A Kinesis Data Stream partitions records across shards by partition key. Records within a shard are ordered. Multiple consumer applications read independently from their own positions in the stream, with full replay capability within the retention window.

When to use it: High-volume continuous data where order matters within a partition: user activity streams, financial transactions, IoT sensor readings, application logs being fed to a data lake. Use Kinesis instead of SQS when multiple consumers need to independently replay the same records, or when throughput exceeds what SQS FIFO can handle. See the data pipeline design guide for how Kinesis connects to broader pipeline architectures.

Step Functions for orchestrated multi-step workflows

What it is: A Step Functions state machine explicitly defines each step of a workflow, with built-in retries, error handling, branching, and parallel execution. The state machine is the single source of truth for where every workflow execution stands.

When to use it: Multi-step business processes where you need to know what happened, in what order, and where failures occurred: payment processing, order fulfillment, compliance workflows, user onboarding with multiple verification steps.

Concrete example: An order fulfillment workflow: (1) reserve inventory, (2) charge payment, (3) create shipment, (4) send confirmation. If the payment step fails, Step Functions retries with exponential backoff and, after hitting the retry limit, triggers a compensating transaction to release the reserved inventory. Every execution is visible in the AWS console with a full step-by-step history.

Choreography vs orchestration: making the choice

This is the most consequential design decision in event-driven architecture. Both approaches distribute work across services. The difference is where coordination lives and how visible it is.

Step Functions is clearly the right choice when:

• The process has multiple sequential steps that must complete in a specific order.
• You need to compensate (undo) earlier steps if a later step fails (the saga pattern).
• You need to know exactly where a specific execution failed and why, without digging through logs across multiple services.
• The workflow has complex branching, parallel execution, or human approval steps.
• Business stakeholders or support teams need visibility into workflow state without reading application logs.

Choreography is clearly the right choice when:

• Multiple services need to react to one event independently and the reactions do not need to happen in a specific sequence.
• You want to add new consumers in the future without changing any existing code.
• The reactions are non-critical (analytics, cache invalidation, notifications) where partial or delayed failure is acceptable.
• Each consuming service genuinely owns its processing logic with no dependency on the outcomes of other consumers.

Reliability and failure handling in production

Event-driven systems fail in subtle ways. The failure does not usually crash a service. It silently drops events, duplicates processing, or delivers messages out of order. This section covers the patterns that prevent those failures from becoming data loss or customer-visible bugs.

Idempotency

Most AWS event services deliver at-least-once. SQS, EventBridge, and Kinesis can all deliver the same message more than once. A consumer that is not idempotent will process duplicates and produce wrong results: double-charged payments, double-sent emails, double-decremented inventory.

Common approaches to idempotency:

• Deduplication IDs: Include a unique event ID in the payload. The consumer checks whether it has already processed this ID before doing any work. Store processed IDs in DynamoDB with a TTL matched to your retry window.
• Conditional writes: Use DynamoDB conditional expressions or database UPSERT semantics. Write only if the record does not already exist, or update only if the current state is what you expect.
• Naturally idempotent operations: “Set inventory count to 42” is idempotent. “Decrement inventory by 1” is not. Redesigning operations to be inherently safe to repeat eliminates the problem at the source.

Retries and backoff

Configure explicit retry policies. Do not accept defaults without understanding them. Key settings to check:

• SQS visibility timeout: Must be longer than the maximum consumer execution time. If Lambda takes up to 60 seconds to process a message, set visibility timeout to at least 90 seconds. Otherwise a slow Lambda invocation causes the message to become visible again and a second Lambda picks it up before the first one finishes.
• Lambda maximum retry attempts: For asynchronous Lambda invocations, the default is 2 retries. Set it explicitly: 0 for operations you cannot make idempotent, higher for safe idempotent consumers.
• EventBridge retry policy: EventBridge retries with exponential backoff for up to 24 hours by default. Shorten this window if events become stale quickly for your use case.

Dead-letter queues

Every SQS queue that feeds a consumer should have a DLQ. Without one, messages that fail processing are retried until the retention period expires and then silently discarded. Here is the Terraform configuration:

resource "aws_sqs_queue" "email_queue" {
  name                       = "email-queue"
  visibility_timeout_seconds = 90
  message_retention_seconds  = 86400  # 1 day

  redrive_policy = jsonencode({
    deadLetterTargetArn = aws_sqs_queue.email_dlq.arn
    maxReceiveCount     = 3
  })
}

resource "aws_sqs_queue" "email_dlq" {
  name                      = "email-queue-dlq"
  message_retention_seconds = 1209600  # 14 days
}

After 3 failed processing attempts, messages move to the DLQ where they are retained for 14 days for inspection and reprocessing. Set a CloudWatch alarm on ApproximateNumberOfMessagesVisible for every DLQ. Any non-zero value means events are being dropped and needs immediate investigation.

Ordering

Options when ordering matters:

• SQS FIFO: Guarantees order within a message group and exactly-once processing. Throughput is limited to approximately 3,000 messages per second with batching. Use FIFO when ordering is required and volume is within this limit.
• Kinesis shards: Records within a shard are strictly ordered. Route related events to the same shard using a consistent partition key (for example, order ID). Use Kinesis when ordering is required at high volume.
• Design order-independence: The cleanest long-term solution. Design consumers so the processing result is correct regardless of event arrival order. This eliminates ordering as an architectural constraint entirely.

Observability

Silent failures are the defining operational risk of event-driven systems. An event dropped by a failed consumer looks like normal operation unless you are actively monitoring for it. Minimum production requirements:

• Structured logs with a correlation ID on every event. The same ID should appear in the logs of every service that handled that event.
• X-Ray distributed tracing enabled on Lambda and API Gateway to trace the full event path across services in one view.
• CloudWatch alarms on DLQ depth (ApproximateNumberOfMessagesVisible > 0) for every queue in production.
• CloudWatch alarms on Lambda error rate for every event consumer function.