Azure Batch Overview: Large-Scale Parallel Compute Jobs

The Batch model: pools, jobs, and tasks

Azure Batch has three core concepts:

• Pool. A set of compute nodes (VMs) that run tasks. You choose the VM size, OS image, number of nodes, and scaling policy. Nodes can autoscale based on task queue depth.
• Job. A container for a set of related tasks. A job runs on a specific pool. It has a priority, maximum wall time, and job preparation/release tasks (scripts that run before and after all tasks on each node).
• Task. A single unit of work — a command line that runs on one node. Tasks are the actual computation: run a Python script, execute an FFmpeg command, invoke a binary. Each task can have its own dependencies (one task can depend on another completing first) and resource requirements (specific CPU or memory).

The workflow: create a pool → create a job on the pool → submit tasks to the job → Azure Batch assigns tasks to available nodes → tasks run → results are written to Azure Storage → job completes.

Workloads that fit Azure Batch

Azure Batch is purpose-built for workloads that are:

• Embarrassingly parallel: thousands of independent work items that do not need to communicate with each other during processing
• Variable in scale: you need 500 nodes for 4 hours, then zero — provisioning and de-provisioning on demand
• Long-running per task: individual tasks run for minutes to hours, not milliseconds
• Compute-intensive: CPU, GPU, or memory-bound tasks that justify the overhead of VM-based compute

Real-world Azure Batch use cases include:

• Visual effects rendering (each frame of a film is one task)
• Financial risk modelling (Monte Carlo simulations split into independent runs)
• Genomic sequencing pipeline (each sample processed independently)
• Drug discovery molecular simulations
• Large-scale image processing or OCR across millions of files
• Machine learning model training across dataset partitions

Creating a Batch account and pool

# Create a Batch account (links to a storage account for task I/O)
az batch account create \
  --name mybatchaccount \
  --resource-group my-rg \
  --location eastus \
  --storage-account mystorageaccount

# Log in to the Batch account
az batch account login \
  --name mybatchaccount \
  --resource-group my-rg \
  --shared-key-auth

# Create a pool of 5 Ubuntu nodes
az batch pool create \
  --id my-pool \
  --image canonical:0001-com-ubuntu-server-focal:20_04-lts \
  --node-agent-sku-id "batch.node.ubuntu 20.04" \
  --vm-size Standard_D4s_v5 \
  --target-dedicated-nodes 5

The —node-agent-sku-id must match the OS image — Azure Batch installs an agent on each node to receive tasks. Run az batch pool supported-images list to see valid combinations of image and agent SKU.

Submitting a job and tasks

# Create a job on the pool
az batch job create \
  --id my-render-job \
  --pool-id my-pool

# Submit 10 tasks (each renders one frame)
for i in $(seq 1 10); do
  az batch task create \
    --job-id my-render-job \
    --task-id frame-$i \
    --command-line "python3 /scripts/render.py --frame $i --output /mnt/output/frame_$i.png"
done

# Monitor job progress
az batch job show --job-id my-render-job --output table

# List task statuses
az batch task list --job-id my-render-job --output table

Azure Batch distributes the 10 tasks across the 5 pool nodes — 2 tasks per node running in parallel. As tasks complete, the nodes pick up new tasks from the queue until all tasks are done.

Pool autoscaling

Batch pools can autoscale based on the number of pending and running tasks. The autoscale formula uses a simple expression language:

# Example autoscale formula: scale nodes to match pending task count
# Scale up: add nodes based on pending tasks
# Scale down: remove idle nodes after 10 minutes
$pendingTasks = $ActiveTasks.GetSamplePercent(TimeInterval_Minute * 15) >= 70
               ? max(0, $PendingTasks.GetSample(1))
               : max(0, $PendingTasks.GetSample(1) * initialNodeCount);

$TargetDedicatedNodes = min(100, max(5, $pendingTasks));
$NodeDeallocationOption = taskcompletion;

Set the formula when creating or updating the pool:

az batch pool autoscale enable \
  --pool-id my-pool \
  --auto-scale-formula '$TargetDedicatedNodes = min(100, $PendingTasks.GetSample(1));'

The pool grows to match the pending task count (capped at 100) and shrinks when tasks complete. For variable workloads, this avoids paying for idle nodes between job waves.

Azure Batch versus Azure Functions for batch workloads

Both services process work items, but they target different scales and task durations:

Use Batch for large-scale HPC, rendering, and scientific workloads where tasks take minutes to hours and require full OS access or GPU support. Use Functions for shorter event-driven tasks where Consumption plan billing and managed scaling work in your favour.