Azure Databricks Overview

Why Databricks exists

Open-source Apache Spark is powerful but operationally demanding. Running Spark yourself means managing cluster provisioning, Spark version upgrades, dependency management, security patches, networking configuration, and performance tuning. A skilled data engineer can spend as much time on infrastructure as on actual data work.

Databricks manages all of that infrastructure while adding a layer of performance optimisations (the Photon engine), storage management (Delta Lake), and tooling (the collaborative workspace UI) that are not available in open-source Spark. The result is a platform where data teams focus on their logic rather than their infrastructure.

Databricks architecture on Azure

Azure Databricks runs in a split-plane architecture:

┌─────────────────────────────────────────────────────────────────┐
│          Control Plane (Databricks-managed, in Databricks VNet) │
│                                                                 │
│  ┌───────────────┐  ┌────────────────┐  ┌─────────────────┐    │
│  │  Workspace UI │  │ Cluster Manager│  │  Job Scheduler  │    │
│  └───────────────┘  └────────────────┘  └─────────────────┘    │
│                                                                 │
│  ┌───────────────┐  ┌────────────────┐                          │
│  │   MLflow      │  │ Unity Catalog  │                          │
│  └───────────────┘  └────────────────┘                          │
└───────────────────────────────────┬─────────────────────────────┘
                                    │ Secure connectivity
┌───────────────────────────────────▼─────────────────────────────┐
│        Data Plane (in YOUR Azure subscription / VNet)           │
│                                                                 │
│  ┌────────────────────────────────────────────────────────┐     │
│  │                 Azure VMs (Spark cluster)               │     │
│  │  ┌───────────┐  ┌───────────┐  ┌───────────────────┐  │     │
│  │  │  Driver   │  │ Executor  │  │  Executor (N)...  │  │     │
│  │  └───────────┘  └───────────┘  └───────────────────┘  │     │
│  └────────────────────────────────────────────────────────┘     │
│                                                                 │
│  ┌────────────────────────────────────────────────────────┐     │
│  │           ADLS Gen2 / Delta Lake storage                │     │
│  └────────────────────────────────────────────────────────┘     │
└─────────────────────────────────────────────────────────────────┘

The control plane — containing the UI, cluster manager, job scheduler, MLflow, and catalog — runs in a Databricks-managed Azure subscription. The data plane — the actual VMs running your Spark code — runs in your own Azure subscription and VNet. Your data never leaves your subscription; only cluster management traffic crosses the plane boundary.

Cluster types: all-purpose vs job clusters

Databricks has two cluster types:

All-purpose clusters are designed for interactive development. They remain running until you manually terminate them or until the auto-terminate timeout fires. Multiple users can share the same all-purpose cluster. You use these for writing notebooks, exploring data, and iterating on code.

Job clusters are created specifically for a job run, used for the duration of the job, and terminated automatically when the job finishes. They are more cost-efficient for production scheduled jobs because you pay only for the actual runtime, not for idle time between jobs. Job clusters cannot be shared between jobs running simultaneously.

# Create an Azure Databricks workspace using Azure CLI
az databricks workspace create \
  --resource-group rg-databricks \
  --name dbw-mycompany-prod \
  --location eastus \
  --sku premium \
  --enable-no-public-ip true

# Get the workspace URL (Databricks instance URL)
az databricks workspace show \
  --resource-group rg-databricks \
  --name dbw-mycompany-prod \
  --query workspaceUrl \
  --output tsv

Delta Lake: the storage layer

Delta Lake is an open-source storage format that adds ACID transactions, schema enforcement, time travel, and efficient upserts to Parquet files in ADLS Gen2. It is the default table format in Databricks and is deeply integrated into the platform.

Key Delta Lake capabilities:

• ACID transactions — reads never see partial writes; concurrent writes are serialised safely
• Schema enforcement — Delta rejects writes that violate the table schema, preventing silent data corruption
• Schema evolution — columns can be added safely with backwards compatibility
• Time travel — query a table as it existed at any previous version or timestamp
• MERGE INTO — efficient upserts for CDC (change data capture) patterns
• Z-order clustering — co-locate related data within files to reduce I/O for multi-column filters

from pyspark.sql import SparkSession
from delta.tables import DeltaTable
from pyspark.sql.functions import col, current_timestamp

spark = SparkSession.builder \
    .config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension") \
    .config("spark.sql.catalog.spark_catalog", "org.apache.spark.sql.delta.catalog.DeltaCatalog") \
    .getOrCreate()

# Create a Delta table
orders = spark.createDataFrame([
    (1001, 42, "2025-01-15", 149.99),
    (1002, 17, "2025-01-15", 89.50),
    (1003, 42, "2025-01-16", 229.00),
], ["order_id", "customer_id", "order_date", "total"])

orders.write.format("delta").mode("overwrite").save(
    "abfss://gold@mydatalake.dfs.core.windows.net/orders/"
)

# MERGE: upsert new/changed records efficiently
delta_table = DeltaTable.forPath(
    spark,
    "abfss://gold@mydatalake.dfs.core.windows.net/orders/"
)

new_data = spark.createDataFrame([
    (1002, 17, "2025-01-15", 95.00),  # updated total for order 1002
    (1004, 55, "2025-01-17", 315.00), # new order
], ["order_id", "customer_id", "order_date", "total"])

delta_table.alias("existing").merge(
    new_data.alias("updates"),
    "existing.order_id = updates.order_id"
).whenMatchedUpdateAll() \
 .whenNotMatchedInsertAll() \
 .execute()

# Time travel: query the table as it was before the merge
historical = spark.read.format("delta") \
    .option("versionAsOf", 0) \
    .load("abfss://gold@mydatalake.dfs.core.windows.net/orders/")

historical.show()

MLflow: ML experiment tracking

MLflow is an open-source platform for managing the ML lifecycle: experiment tracking, model packaging, model registry, and deployment. It is pre-installed and deeply integrated in every Databricks workspace — when you call mlflow.autolog(), MLflow automatically logs parameters, metrics, and the trained model for any supported ML framework (Scikit-learn, XGBoost, TensorFlow, PyTorch, and others).

import mlflow
import mlflow.sklearn
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score, roc_auc_score
import pandas as pd
import numpy as np

# Load training data from Delta Lake
df = spark.read.format("delta") \
    .load("abfss://gold@mydatalake.dfs.core.windows.net/churn_features/") \
    .toPandas()

X = df.drop("churned", axis=1)
y = df["churned"]
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Enable autologging — MLflow automatically records params and metrics
mlflow.sklearn.autolog()

with mlflow.start_run(run_name="GBT_churn_v1"):
    model = GradientBoostingClassifier(
        n_estimators=200,
        max_depth=5,
        learning_rate=0.05,
        random_state=42
    )
    model.fit(X_train, y_train)

    y_pred = model.predict(X_test)
    y_prob = model.predict_proba(X_test)[:, 1]

    accuracy = accuracy_score(y_test, y_pred)
    auc = roc_auc_score(y_test, y_prob)

    # Log additional custom metrics beyond autolog
    mlflow.log_metric("test_accuracy", accuracy)
    mlflow.log_metric("test_auc", auc)
    mlflow.log_param("training_rows", len(X_train))

    print(f"Accuracy: {accuracy:.4f} | AUC: {auc:.4f}")
    # Model, parameters, and metrics are all recorded in the MLflow experiment

After training, the model appears in the Databricks MLflow experiment UI with all parameters and metrics logged. You can compare multiple runs side by side, register the best model to the Model Registry, and deploy it as a REST API endpoint — all from the same workspace.

Unity Catalog: cross-workspace governance

Unity Catalog is Databricks’s data governance layer. It provides a three-level namespace (catalog → schema → table), centralised access control, data lineage, and audit logging across multiple Databricks workspaces. Before Unity Catalog, each workspace had its own isolated metastore — different workspaces could not share table definitions or access controls.

With Unity Catalog, a single governance layer covers all workspaces in an account. A data steward can grant access to a table in one place and the permission applies across all workspaces and all engines (notebooks, SQL warehouses, jobs) that reference that table.