Spot VMs for Cost Savings in Azure

Spot VM Pricing

Spot prices fluctuate based on supply and demand for spare capacity. The Azure portal shows the current Spot price and historical eviction rate for each VM size and region combination. Here are representative ranges for common sizes in East US:

Workloads Suitable for Spot VMs

The key requirement for Spot VM suitability is that the workload can be safely interrupted and resumed, or that the work unit is short enough that interruption is acceptable. Good candidates include:

Batch processing and data pipelines

ETL jobs, data transformation, and batch analytics that process data from a queue or file system. When evicted mid-job, the job resumes from the last checkpoint on a replacement VM. The Azure Batch service is designed specifically for this pattern on Spot compute.

CI/CD build agents

Build servers that compile code, run tests, and produce artifacts. A build job that takes 15 minutes can be safely retried on a new Spot VM if the original is evicted. Most CI/CD systems (Azure DevOps, GitHub Actions) have built-in retry mechanisms.

Machine learning training

ML model training jobs that use frameworks with checkpoint support (TensorFlow, PyTorch). Save model state every N epochs to persistent storage. If the Spot VM is evicted, a replacement VM resumes training from the last checkpoint rather than from scratch.

Stateless web application backends

HTTP servers that do not hold user session state locally. If an instance is evicted, the load balancer routes traffic to remaining instances while a replacement is provisioned. This requires multiple instances and a load balancer — it does not work with a single Spot VM serving all traffic.

Development and test environments

Dev VMs where an occasional interruption is acceptable. Teams that can tolerate a VM restart (with the help of auto-start scripts and tooling that resumes their workflow) can use Spot for significant savings on long-running dev machines.

Workloads Not Suitable for Spot VMs

• Production databases: data loss on eviction is unacceptable. Use managed database services or PAYG/RI VMs for database workloads.
• Stateful single-instance services: anything holding in-memory session state or local cache that cannot be quickly reconstructed.
• Latency-sensitive user-facing services without redundancy: a single Spot VM serving user requests will cause service unavailability during eviction.
• Long jobs without checkpointing: a 24-hour ML training run without checkpoints would lose all progress on eviction. Add checkpointing first, then consider Spot.

Creating and Managing Spot VMs

# Create a Spot VM with Deallocate eviction policy
az vm create \
  --resource-group my-rg \
  --name spot-worker-01 \
  --image Ubuntu2204 \
  --size Standard_D4s_v5 \
  --priority Spot \
  --eviction-policy Deallocate \
  --max-price -1 \
  --admin-username azureuser \
  --generate-ssh-keys \
  --no-wait

# --eviction-policy Deallocate: VM stops but disks are retained
# --eviction-policy Delete: entire VM and disks are deleted on eviction
# --max-price -1: pay up to PAYG price, only evict for capacity reasons
# --max-price 0.05: set a maximum hourly price; evict if price exceeds $0.05/hr

# Check current Spot price for a VM size
az rest \
  --method GET \
  --url "https://prices.azure.com/api/retail/prices?\$filter=armRegionName eq 'eastus' and skuName eq 'D4s v5 Spot' and serviceName eq 'Virtual Machines'" \
  --query "Items[].{sku:skuName, spotPrice:retailPrice}" \
  --output table

Creating a Spot VM Scale Set

For batch processing workloads, VM Scale Sets with Spot priority provide automatic scaling alongside Spot pricing. The Scale Set replaces evicted instances automatically.

# Create a Spot VM Scale Set
az vmss create \
  --resource-group my-rg \
  --name spot-batch-scaleset \
  --image Ubuntu2204 \
  --vm-sku Standard_D4s_v5 \
  --instance-count 3 \
  --priority Spot \
  --eviction-policy Deallocate \
  --max-price -1 \
  --upgrade-policy-mode Manual \
  --admin-username azureuser \
  --generate-ssh-keys

# Enable autoscale on the Scale Set
az monitor autoscale create \
  --resource-group my-rg \
  --resource spot-batch-scaleset \
  --resource-type Microsoft.Compute/virtualMachineScaleSets \
  --name spot-autoscale \
  --min-count 1 \
  --max-count 20 \
  --count 3

Handling Eviction Gracefully

The Azure Instance Metadata Service (IMDS) provides a 30-second eviction warning. Applications can poll this endpoint to detect pending eviction and initiate graceful shutdown — saving state, completing or checkpointing current work, and draining connections.

# Poll the IMDS eviction notice endpoint from within the VM
# Returns 200 with JSON body when eviction is imminent
# Returns 404 when no eviction is scheduled

curl -H "Metadata: true" \
  "http://169.254.169.254/metadata/scheduledevents?api-version=2020-07-01" | \
  python3 -c "import sys,json; events=json.load(sys.stdin)['Events']; \
  print('EVICTION PENDING' if any(e['EventType']=='Preempt' for e in events) else 'No eviction')"

# Python eviction handler — run as a background thread or sidecar process
import requests
import time
import signal
import sys

METADATA_URL = "http://169.254.169.254/metadata/scheduledevents?api-version=2020-07-01"

def check_eviction():
    while True:
        try:
            response = requests.get(
                METADATA_URL,
                headers={"Metadata": "true"},
                timeout=5
            )
            events = response.json().get("Events", [])
            for event in events:
                if event["EventType"] == "Preempt":
                    print("Eviction notice received. Saving checkpoint...")
                    save_checkpoint()  # Your checkpoint logic
                    sys.exit(0)
        except Exception as e:
            pass
        time.sleep(5)

def save_checkpoint():
    # Save current work state to Azure Blob Storage or Azure Queue
    pass

Azure Batch with Spot VMs

Azure Batch is the managed service specifically designed for large-scale parallel computing on Spot VMs. It handles Spot node provisioning, eviction recovery, job queuing, and task retry automatically. For batch workloads, Azure Batch is almost always a better choice than managing Spot VMs directly.

# Create an Azure Batch account
az batch account create \
  --name mybatchaccount \
  --resource-group my-rg \
  --location eastus \
  --storage-account mystorage

# Create a pool with Spot (low-priority) nodes
az batch pool create \
  --id spot-batch-pool \
  --vm-size Standard_D4s_v5 \
  --target-low-priority-nodes 10 \
  --target-dedicated-nodes 0 \
  --image publisher=Canonical offer=UbuntuServer sku=20_04-lts version=latest \
  --node-agent-sku-id "batch.node.ubuntu 20.04"