Designing Highly Available Systems in Azure

The fundamentals of high availability

High availability is expressed as a percentage uptime over a rolling period. Azure SLAs are contractual commitments that define how much downtime is acceptable — 99.9% allows about 8.7 hours per year, 99.99% allows about 52 minutes, and 99.999% allows about 5 minutes. Your target SLA drives every design decision downstream.

Two metrics govern recovery from failures. Recovery Time Objective (RTO) is the maximum acceptable duration between a failure and full restoration of service. Recovery Point Objective (RPO) is the maximum acceptable data loss, measured in time. A system with RTO of 15 minutes and RPO of 5 minutes must recover within 15 minutes and must not lose more than 5 minutes of data.

Achieving high availability requires eliminating single points of failure at every tier. A load balancer in front of a single VM does not make the system highly available — the VM is still a single point of failure. You need at least two instances at every tier, and those instances must be placed such that a single failure event cannot take both down simultaneously.

Availability zones and availability sets

Azure regions that support availability zones contain at least three physically separate zones, each with independent power, cooling, and networking infrastructure. Spreading your VMs across zones protects against datacentre-level failures. When one zone loses power, the VMs in the other zones continue running and the load balancer routes traffic exclusively to healthy instances.

To deploy a VM into a specific zone, you specify the zone number at creation time. The Standard Load Balancer and Application Gateway v2 both support zone-redundant frontend IPs that survive zone failures automatically.

# Create a VM in availability zone 1
az vm create \
  --resource-group myRG \
  --name web-vm-zone1 \
  --image Ubuntu2204 \
  --size Standard_D2s_v3 \
  --zone 1 \
  --admin-username azureuser \
  --generate-ssh-keys

# Create a second VM in zone 2
az vm create \
  --resource-group myRG \
  --name web-vm-zone2 \
  --image Ubuntu2204 \
  --size Standard_D2s_v3 \
  --zone 2 \
  --admin-username azureuser \
  --generate-ssh-keys

# Create a zone-redundant Standard Load Balancer
az network lb create \
  --resource-group myRG \
  --name myPublicLB \
  --sku Standard \
  --frontend-ip-name myFrontEnd \
  --backend-pool-name myBackEndPool

Availability sets are the older construct. They spread VMs across up to three fault domains (separate physical racks with independent power and networking) and up to 20 update domains (groups that Azure updates one at a time during planned maintenance). Availability sets do not protect against datacentre failures, only against rack-level and update-collision failures. Use them only when your VM size is not available in a zone-enabled region.

Health probes and automatic failover

A load balancer distributes traffic only to instances that pass health probes. Azure Load Balancer supports TCP and HTTP probes. Application Gateway supports HTTP and HTTPS probes with custom paths and response code matching. Azure Traffic Manager uses DNS-level health checks that can redirect users to a different region when one endpoint becomes unhealthy.

Health probe design matters significantly. A probe that checks only whether a port is open (TCP probe) will pass even if the application is deadlocked or returning 500 errors. An HTTP probe against a dedicated health endpoint that verifies database connectivity, cache connectivity, and core business logic gives a much more accurate picture of whether the instance can actually serve traffic.

# Add a health probe to an existing load balancer (HTTP on port 80, path /health)
az network lb probe create \
  --resource-group myRG \
  --lb-name myPublicLB \
  --name myHealthProbe \
  --protocol Http \
  --port 80 \
  --path /health \
  --interval 15 \
  --threshold 2

The probe interval and unhealthy threshold together determine how quickly Azure detects and removes a failed instance. With a 15-second interval and a threshold of 2, Azure marks an instance unhealthy after 30 seconds of failed probes. This is a reasonable default. Setting these values too low risks removing healthy instances due to transient blips; setting them too high delays recovery.

High availability at the database tier

The database is usually the hardest tier to make highly available because it holds state. Azure offers several managed options that handle replication internally.

Azure SQL Database Business Critical maintains three synchronous replicas across availability zones within the same region. Failover is automatic and typically completes in under 30 seconds with no data loss. The read replicas are also exposed as read-only endpoints, reducing load on the primary.

Azure Database for PostgreSQL Flexible Server supports zone-redundant high availability, where a standby replica in a different zone is kept synchronous with the primary. Failover typically completes in 60-120 seconds.

Azure Cosmos DB replicates data across zones and regions automatically. With session or strong consistency, reads and writes remain consistent. With multi-region writes enabled, any region can accept writes and the SLA rises to 99.999%.

Graceful degradation and circuit breakers

Even with multiple replicas and automatic failover, there will be brief periods during which a dependency is unavailable. Systems that crash entirely when a downstream service is slow or unavailable turn a partial outage into a total one. Graceful degradation means designing each component to continue operating — perhaps with reduced functionality — when its dependencies misbehave.

A circuit breaker pattern prevents a slow downstream service from exhausting your connection pool and taking down your own service. When failure rate exceeds a threshold, the circuit breaker trips to “open” state and returns errors immediately without attempting the real call. After a timeout, it enters “half-open” state and allows a few trial requests through. If those succeed, it closes again.

In Azure, you can implement circuit breakers in application code using libraries like Polly (for .NET) or Resilience4j (for Java). Azure API Management has built-in circuit breaker policies. Azure Service Bus and Event Hubs automatically queue messages so that downstream processing can restart after a failure without losing events.

Safe deployment practices

Many outages are self-inflicted — caused by a bad deployment rather than infrastructure failure. High availability requires treating deployments themselves as potential failure events and managing them accordingly.

Blue-green deployments maintain two identical environments. Traffic shifts from blue (old) to green (new) all at once, but the old environment is kept live so rollback is immediate. Canary deployments gradually shift a small percentage of traffic to the new version, monitoring error rates and latency before proceeding.

Azure App Service deployment slots implement blue-green natively. Azure Kubernetes Service supports canary deployments via weighted ingress rules. Azure Front Door allows traffic splitting between backends by percentage, making it straightforward to canary a new version at the global load balancing layer.

# Swap staging slot to production in App Service (blue-green)
az webapp deployment slot swap \
  --resource-group myRG \
  --name myWebApp \
  --slot staging \
  --target-slot production

# List current slots and their status
az webapp deployment slot list \
  --resource-group myRG \
  --name myWebApp \
  --query "[].{Name:name, State:state}" \
  --output table

Real-world scenario: e-commerce application

Consider an e-commerce platform that must not lose orders and must be available 99.99% of the time. The architecture has three tiers: web frontend, order processing API, and order database.

The web frontend runs on Azure App Service Premium with three zone-redundant instances behind Azure Application Gateway v2. The Application Gateway is itself zone-redundant and performs SSL termination, WAF inspection, and cookie-based session affinity. Azure Front Door sits in front, providing global anycast entry points and caching for static content.

The order processing API runs on Azure Container Apps with a minimum of three replicas spread across zones. It uses managed identity to authenticate to downstream services — no credentials stored in config. Azure Service Bus queues orders asynchronously so that a spike in traffic does not cause API timeouts; the queue acts as a shock absorber.

The order database is Azure SQL Database Business Critical with zone-redundant replication. A read replica is configured as the API’s read endpoint for product catalog queries, keeping write capacity reserved for order inserts. Geo-replication to a secondary region provides a fallback for region-level failures, with an RPO of approximately 5 seconds.

Architecture review checklist

Before declaring a system highly available, validate each of the following:

• Every tier has at least two instances in separate availability zones.
• Load balancers are Standard tier (zone-redundant), not Basic tier.
• Health probes check application logic, not just port liveness.
• Databases use zone-redundant HA or geo-replication.
• Retry logic and circuit breakers are implemented in all inter-service calls.
• Deployments use slots, canaries, or staged rollouts — not in-place replacement of all instances simultaneously.
• Dependencies on external services (payment gateways, shipping APIs) have fallback behavior or graceful error messages.
• You have run failure injection tests (Azure Chaos Studio) to verify failover actually works.