Azure Data Lake Storage Gen2 Overview

What ADLS Gen2 adds to Blob Storage

Standard Azure Blob Storage is a flat namespace: files (blobs) are stored in containers, and “folders” are just a naming convention using forward slashes in blob names. There are no real directories. Moving 10,000 files from one “folder” to another requires copying and deleting each blob individually — an O(n) operation.

ADLS Gen2 introduces a hierarchical namespace — a real directory tree. Directories are first-class objects. Renaming or moving a directory is an atomic metadata operation regardless of how many files it contains — an O(1) operation. This makes ADLS Gen2 suitable for analytics workloads that move, rename, and reorganize data as part of processing pipelines.

The other major addition is POSIX-compatible ACLs. Standard Blob Storage access control is coarse-grained: you assign an Azure RBAC role to a principal and it applies to all blobs in a container or account. ADLS Gen2 ACLs can be applied to individual files and directories, with separate read, write, and execute permissions for owning user, owning group, named users, named groups, and an “other” category. This mirrors the Unix permission model that data engineering teams are familiar with.

Creating an ADLS Gen2 account

Enable the hierarchical namespace at account creation time with the —enable-hierarchical-namespace true flag. The account is otherwise created like any standard storage account.

# Create an ADLS Gen2 storage account
az storage account create \
  --resource-group myResourceGroup \
  --name myadlsaccount \
  --location eastus \
  --sku Standard_LRS \
  --kind StorageV2 \
  --enable-hierarchical-namespace true \
  --access-tier Hot

# Verify hierarchical namespace is enabled
az storage account show \
  --resource-group myResourceGroup \
  --name myadlsaccount \
  --query isHnsEnabled

Choose the SKU based on your redundancy requirements. Standard_LRS is locally redundant within a single datacenter. For analytics data that is expensive to regenerate, use Standard_ZRS (zone-redundant) or Standard_GRS (geo-redundant). For high-throughput analytics workloads, consider Standard_RAGRS for read access from the secondary region.

Storage hierarchy: containers, directories, and files

ADLS Gen2 uses the same container concept as Blob Storage, but inside a container you have a real directory tree. The typical data lake organization follows a zone pattern:

Create directories and files using the Azure CLI or the Data Lake Storage SDK:

# Get a connection string or use --account-name with identity
ACCOUNT_KEY=$(az storage account keys list \
  --resource-group myResourceGroup \
  --account-name myadlsaccount \
  --query '[0].value' -o tsv)

# Create the raw container
az storage fs create \
  --name raw \
  --account-name myadlsaccount \
  --account-key $ACCOUNT_KEY

# Create a directory path inside the container
az storage fs directory create \
  --file-system raw \
  --name "salesdata/2026/03/19" \
  --account-name myadlsaccount \
  --account-key $ACCOUNT_KEY

# Upload a file to a specific directory
az storage fs file upload \
  --source ./sales-20260319.parquet \
  --path "salesdata/2026/03/19/sales-20260319.parquet" \
  --file-system raw \
  --account-name myadlsaccount \
  --account-key $ACCOUNT_KEY

Setting ACLs with Azure CLI

ADLS Gen2 ACLs use the format user::rwx,group::r-x,other::--- for default POSIX permissions, plus named-user and named-group entries like user:<object-id>:r-x. There are two types of ACL per entry: access ACLs (apply to the current item) and default ACLs (inherited by new child items).

# Set ACL on a directory — give a service principal read+execute on a path
PRINCIPAL_ID="00000000-0000-0000-0000-aabbccddeeff"

az storage fs access set \
  --acl "user:${PRINCIPAL_ID}:r-x" \
  --path "salesdata/2026/03/19" \
  --file-system raw \
  --account-name myadlsaccount \
  --account-key $ACCOUNT_KEY

# Set a default ACL so new files in a directory inherit permissions
az storage fs access set \
  --acl "default:user:${PRINCIPAL_ID}:r-x" \
  --path "salesdata" \
  --file-system raw \
  --account-name myadlsaccount \
  --account-key $ACCOUNT_KEY

# Show the current ACL for a path
az storage fs access show \
  --path "salesdata/2026/03/19" \
  --file-system raw \
  --account-name myadlsaccount \
  --account-key $ACCOUNT_KEY

ACLs in ADLS Gen2 work alongside Azure RBAC, not instead of it. A principal needs both an Azure RBAC role (at minimum Storage Blob Data Reader) and the appropriate ACL entry on the specific path. Azure RBAC provides the “gate into the storage account” and ACLs provide the per-directory authorization within it.

Performance tier and optimization

ADLS Gen2 supports the same access tiers as Blob Storage: Hot, Cool, Cold, and Archive. For active analytics workloads, Hot is appropriate. Data that is queried weekly or monthly can be on Cool. Historical raw data that is retained for compliance but rarely queried should use Archive (with the understanding that retrieval takes hours).

For high-throughput analytics, use the premium performance tier (Premium block blobs) rather than Standard. Premium ADLS Gen2 uses SSD-backed storage with significantly lower latency and higher IOPS — important for interactive query engines like Azure Synapse Analytics and Azure Databricks when they read large Parquet files.

# Create a premium-tier ADLS Gen2 account (SSD-backed)
az storage account create \
  --resource-group myResourceGroup \
  --name myadlspremium \
  --location eastus \
  --sku Premium_LRS \
  --kind BlockBlobStorage \
  --enable-hierarchical-namespace true

Premium storage accounts with hierarchical namespace only support LRS redundancy. If you need geo-redundancy on a premium account, you must implement application-level replication.

Integration with Azure Synapse and Databricks

ADLS Gen2 is the default storage layer for both Azure Synapse Analytics and Azure Databricks. When creating a Synapse workspace, you associate it with an ADLS Gen2 account that becomes the workspace’s primary storage. Notebooks, Spark pools, and SQL pools all read and write to this account.

For Databricks, mount the ADLS Gen2 container using a service principal or managed identity, or use the native ABFS (Azure Blob File System) driver which accesses ADLS Gen2 directly without mounting:

# Databricks: Access ADLS Gen2 with ABFS driver and managed identity
df = spark.read.parquet(
    "abfss://raw@myadlsaccount.dfs.core.windows.net/salesdata/2026/"
)

The abfss:// protocol (ABFS Secure) is the native driver for ADLS Gen2. It supports directory listing, rename, and delete operations at the directory level — operations that are slow or unsupported with the wasbs:// (Azure Blob File System) driver that works with flat-namespace accounts.