Running Spark in Azure: Databricks vs Synapse

Setting up a cluster on each platform

The first practical difference is how you create a Spark cluster.

On Synapse, you define a Spark pool at the workspace level. The pool is a configuration template — actual cluster nodes are allocated when you start a session (open a notebook or run a job). You define the pool once and reuse it for all notebooks in the workspace.

# Create a Synapse Spark pool via CLI
az synapse spark pool create \
  --workspace-name ws-synapse-demo \
  --name SparkPoolMedium \
  --resource-group rg-data \
  --node-size Medium \
  --min-node-count 3 \
  --max-node-count 8 \
  --enable-auto-scale true \
  --enable-auto-pause true \
  --delay 15 \
  --spark-version 3.4

On Databricks, you create a cluster directly, and each cluster is a fully configured, independently running environment. You choose the node type (which maps to an Azure VM size), the Databricks Runtime version, and whether it is a fixed-size or auto-scaling cluster.

# Create a Databricks cluster using the Databricks CLI
# First, install: pip install databricks-cli
# Then configure: databricks configure --token

databricks clusters create --json '{
  "cluster_name": "dev-cluster-medium",
  "spark_version": "14.3.x-scala2.12",
  "node_type_id": "Standard_DS3_v2",
  "autoscale": {
    "min_workers": 2,
    "max_workers": 8
  },
  "auto_termination_minutes": 30,
  "spark_conf": {
    "spark.databricks.io.cache.enabled": "true"
  }
}'

The cluster creation itself takes 3–7 minutes on Databricks (VM provisioning from cold) or 1–3 minutes on Synapse (pool warm-up). Databricks clusters are persistent until terminated; Synapse pool clusters auto-pause and restart as needed per session.

Reading data from ADLS Gen2

Both platforms read from ADLS Gen2, but the authentication and path format differs slightly:

# ─────────────────────────────────────
# SYNAPSE SPARK
# ─────────────────────────────────────
# Synapse uses the workspace managed identity automatically.
# No credential configuration needed in the notebook.

from pyspark.sql import SparkSession
spark = SparkSession.builder.getOrCreate()

# Read using abfss:// directly
# Authentication is handled by the workspace managed identity
orders = spark.read.format("delta").load(
    "abfss://silver@mydatalake.dfs.core.windows.net/orders/"
)

print(f"Synapse: loaded {orders.count()} orders")
orders.printSchema()

# ─────────────────────────────────────
# DATABRICKS
# ─────────────────────────────────────
# Option A: Use a service principal stored in Databricks secrets
spark.conf.set(
    "fs.azure.account.auth.type.mydatalake.dfs.core.windows.net",
    "OAuth"
)
spark.conf.set(
    "fs.azure.account.oauth.provider.type.mydatalake.dfs.core.windows.net",
    "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider"
)
spark.conf.set(
    "fs.azure.account.oauth2.client.id.mydatalake.dfs.core.windows.net",
    dbutils.secrets.get(scope="azure", key="sp-client-id")
)
spark.conf.set(
    "fs.azure.account.oauth2.client.secret.mydatalake.dfs.core.windows.net",
    dbutils.secrets.get(scope="azure", key="sp-client-secret")
)
spark.conf.set(
    "fs.azure.account.oauth2.client.endpoint.mydatalake.dfs.core.windows.net",
    f"https://login.microsoftonline.com/{dbutils.secrets.get(scope='azure', key='tenant-id')}/oauth2/token"
)

# Option B: Use Unity Catalog (recommended for Databricks workspaces with Unity Catalog)
# No credential config needed — Unity Catalog manages access
orders = spark.table("catalog.silver.orders")

# Option C: Mount the storage account (legacy but common)
# dbutils.fs.mount(
#     source="abfss://silver@mydatalake.dfs.core.windows.net/",
#     mount_point="/mnt/silver",
#     extra_configs={ ... }
# )
# orders = spark.read.format("delta").load("/mnt/silver/orders/")

Running the same transformation on both

This PySpark code runs identically on both platforms once the storage connection is configured:

from pyspark.sql.functions import (
    col, sum as spark_sum, count, avg, round as spark_round,
    year, month, dense_rank, desc
)
from pyspark.sql.window import Window

# Compute top 10 customers by monthly revenue for 2025
monthly_customer_revenue = (
    orders
    .filter(year(col("order_date")) == 2025)
    .groupBy(
        year(col("order_date")).alias("order_year"),
        month(col("order_date")).alias("order_month"),
        "customer_id"
    )
    .agg(
        count("order_id").alias("order_count"),
        spark_sum("order_total").alias("monthly_revenue"),
        avg("order_total").alias("avg_order_value")
    )
    .withColumn(
        "monthly_rank",
        dense_rank().over(
            Window.partitionBy("order_year", "order_month")
                  .orderBy(desc("monthly_revenue"))
        )
    )
    .filter(col("monthly_rank") <= 10)
    .withColumn("avg_order_value", spark_round(col("avg_order_value"), 2))
    .orderBy("order_year", "order_month", "monthly_rank")
)

monthly_customer_revenue.show(20)

# Write the result — paths differ between platforms:

# On Synapse:
monthly_customer_revenue.write.format("delta").mode("overwrite").save(
    "abfss://gold@mydatalake.dfs.core.windows.net/top_customers_monthly/"
)

# On Databricks with Unity Catalog:
# monthly_customer_revenue.write.format("delta").mode("overwrite").saveAsTable(
#     "catalog.gold.top_customers_monthly"
# )

Performance: Photon vs standard JVM Spark

Databricks’s Photon engine is a native C++ vectorised execution engine that replaces the JVM-based Spark SQL execution for supported operations. It is particularly effective for:

• SQL queries with GROUP BY, aggregations, and large joins
• DataFrame operations that translate to SQL operations (groupBy, join, agg)
• Reading and writing Parquet and Delta files

Photon does not accelerate:

• Python UDFs (User-Defined Functions) — these still run in the JVM
• Pandas UDFs (though Photon acceleration for pandas UDFs is being added in newer runtimes)
• Some streaming operations

These are illustrative numbers — actual performance varies by cluster size, data distribution, and specific query patterns. The Photon advantage is most pronounced for SQL-style operations on large datasets.

Cost model comparison

Synapse Spark charges per vCore-hour. Databricks charges per DBU (Databricks Unit) plus the underlying Azure VM cost. DBUs are an abstract measure of compute consumption — a standard job cluster consumes approximately 0.75 DBUs per vCore-hour on the Jobs Compute pricing tier.

Databricks job cluster pricing (Jobs tier) is significantly cheaper than the All-Purpose tier used for interactive clusters. For production ETL jobs, Databricks on the Jobs tier is often cheaper than Synapse Spark for equivalent compute. For interactive development, the costs are comparable.