Apache Spark in Azure: Synapse and Databricks

What Apache Spark is and why it matters

Apache Spark is a distributed computing framework that processes data across a cluster of machines in parallel. It keeps data in memory between operations, making it 10–100x faster than Hadoop MapReduce for iterative algorithms like machine learning. Spark has four main components:

• Spark SQL — query structured data using SQL or the DataFrame API
• Spark Streaming / Structured Streaming — process real-time data streams as mini-batches
• MLlib — a library of distributed machine learning algorithms
• GraphX — graph processing (less commonly used)

Both Synapse Spark and Databricks support all four components. The differences are in cluster management, ecosystem integrations, and developer experience rather than in the underlying Spark capabilities.

Synapse Spark pools: integrated but limited

Synapse Spark pools are Spark clusters managed within a Synapse workspace. They are defined at the pool level (node size, min/max nodes, Spark version) and allocated to sessions on demand. A session starts when you open a notebook and attach it to a pool; the session ends when you close the notebook or the auto-pause timeout fires.

Synapse Spark’s key advantages:

• Native Synapse integration — Spark can read from and write to dedicated SQL pools using the spark.read.synapsesql() connector without any additional configuration. Credentials flow automatically from the workspace identity.
• Shared development environment — SQL analysts and Spark engineers work in the same Synapse Studio with shared access to the same datasets and linked services.
• Lower operational overhead — pool-level configuration is simpler than Databricks cluster configuration. Good for teams that are primarily SQL-oriented and need occasional Spark for heavy ETL.
• Auto-pause — clusters auto-pause after idle minutes with no manual intervention, keeping costs low for intermittent workloads.

# Reading from a Synapse dedicated SQL pool in a Synapse Spark notebook
# The workspace identity handles authentication automatically
df = spark.read.synapsesql("myDedicatedPool.dbo.FactOrders")

# Writing a Spark DataFrame back to a dedicated SQL pool
df_aggregated.write.synapsesql(
    "myDedicatedPool.dbo.AggregatedOrders",
    options={
        "dbTable": "dbo.AggregatedOrders"
    }
)

Azure Databricks: purpose-built for Spark at scale

Azure Databricks is a managed Spark platform built specifically around Spark as the primary compute engine. It has deeper Spark expertise baked in than Synapse, including the Photon engine, optimised Delta Lake management, and a much richer MLOps ecosystem with MLflow.

Databricks architecture:

┌──────────────────────────────────────────────────────────────┐
│                   Azure Databricks Workspace                 │
│                                                              │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────────┐   │
│  │  All-Purpose │  │    Job       │  │  SQL Warehouse   │   │
│  │  Cluster     │  │   Cluster    │  │  (Photon SQL)    │   │
│  │(interactive) │  │ (scheduled)  │  │                  │   │
│  └──────────────┘  └──────────────┘  └──────────────────┘   │
│                                                              │
│  ┌──────────────────────────────────────────────────────┐    │
│  │      Delta Lake  (versioned, ACID, ADLS Gen2)        │    │
│  └──────────────────────────────────────────────────────┘    │
│                                                              │
│  ┌──────────────────────────────────────────────────────┐    │
│  │    MLflow  (experiment tracking, model registry)     │    │
│  └──────────────────────────────────────────────────────┘    │
└──────────────────────────────────────────────────────────────┘

Databricks’s key advantages:

• Photon engine — native C++ vectorised execution for SQL and DataFrames, 2–8x faster than standard Spark JVM execution on analytical queries
• Delta Lake — Databricks’s ACID-compliant, versioned table format with Z-order clustering, liquid clustering, and predictive I/O optimisation
• MLflow — built-in experiment tracking, model registry, and deployment. The de facto standard for ML experiment management across the industry
• Job clusters — clusters that spin up for a job, run it, and terminate automatically. More cost-efficient than keeping all-purpose clusters running for scheduled jobs
• Unity Catalog — a centralised data governance and metastore that spans multiple Databricks workspaces

Side-by-side comparison

PySpark example: running on both platforms

This code runs identically on Synapse Spark and Databricks. It reads Delta tables, joins them, aggregates, and writes the result:

from pyspark.sql import SparkSession
from pyspark.sql.functions import (
    col, sum as spark_sum, count, avg,
    date_trunc, when, percentile_approx
)

spark = SparkSession.builder.getOrCreate()

# On Synapse: path would be abfss://silver@storage.dfs.core.windows.net/...
# On Databricks: path would be /mnt/silver/... or a Unity Catalog table name
orders = spark.read.format("delta").load(
    "abfss://silver@mydatalake.dfs.core.windows.net/orders/"
)
customers = spark.read.format("delta").load(
    "abfss://silver@mydatalake.dfs.core.windows.net/customers/"
)

# Compute customer lifetime value and segment
clv = (
    orders
    .join(customers, on="customer_id", how="inner")
    .groupBy("customer_id", "customer_name", "country")
    .agg(
        count("order_id").alias("order_count"),
        spark_sum("order_total").alias("lifetime_value"),
        avg("order_total").alias("avg_order_value"),
        percentile_approx("order_total", 0.5).alias("median_order_value")
    )
    .withColumn(
        "segment",
        when(col("lifetime_value") >= 10000, "Platinum")
        .when(col("lifetime_value") >= 5000, "Gold")
        .when(col("lifetime_value") >= 1000, "Silver")
        .otherwise("Bronze")
    )
)

# Write back to gold layer
clv.write.format("delta").mode("overwrite").save(
    "abfss://gold@mydatalake.dfs.core.windows.net/customer_lifetime_value/"
)

print(f"CLV computed for {clv.count()} customers")
clv.groupBy("segment").count().orderBy("segment").show()

The only changes needed to run this on Databricks are the storage paths (or using Unity Catalog table names like spark.read.table(“catalog.silver.orders”)).

When to choose each platform

Choose Synapse Spark when:

• Your team is primarily SQL-oriented and uses Spark occasionally for heavy ETL
• You need tight integration with Synapse dedicated SQL pools
• You want a single tool with both SQL warehousing and Spark in one workspace
• Your team is smaller and you want lower operational complexity

Choose Azure Databricks when:

• Your team is data science-heavy and needs MLflow, feature stores, and model registry
• You process very large datasets (10 TB+) where Photon’s performance advantage matters
• You need multi-workspace governance with Unity Catalog
• You are building Delta Lake as the foundation of your entire data platform
• Your team is Spark-first and will not use dedicated SQL pools at all