Managing Secrets in Kubernetes on AKS

How Kubernetes Secrets work

A Kubernetes Secret is an API object that stores small amounts of sensitive data — passwords, tokens, TLS certificates, connection strings. Pods consume secrets either as environment variables or as files mounted into the container filesystem.

The data field in a Secret stores values as base64 strings. Base64 is encoding, not encryption. Anyone with read access to the Secret object can decode the value in one command. By default, Secrets are stored unencrypted in the etcd database that backs the Kubernetes control plane.

apiVersion: v1
kind: Secret
metadata:
  name: db-credentials
  namespace: production
type: Opaque
data:
  username: cHJvZHVzZXI=      # base64 of "produser"
  password: U3VwM3JTM2NyZXQ=  # base64 of "Sup3rS3cret"

You apply this with kubectl apply -f secret.yaml. Any pod in the same namespace can then reference it. The problem is that this Secret object, including the encoded value, lives in etcd and is readable by anyone with RBAC permission to get or list Secrets in that namespace — including cluster admins and, in a misconfigured cluster, service accounts.

Mounting a secret as environment variables is the most common pattern but has a further drawback: environment variables are visible in process listings and are often captured in debug dumps or crash reports. Mounting as files is marginally better because access is controlled by the filesystem, but the root problem — the secret lives in etcd — remains either way.

# Consuming a Secret as environment variables
env:
  - name: DB_PASSWORD
    valueFrom:
      secretKeyRef:
        name: db-credentials
        key: password

# Consuming a Secret as a mounted file
volumeMounts:
  - name: db-secret-vol
    mountPath: /etc/secrets
    readOnly: true
volumes:
  - name: db-secret-vol
    secret:
      secretName: db-credentials

Azure Key Vault integration with the Secrets Store CSI Driver

The Secrets Store CSI Driver is the production solution for AKS secret management. Instead of storing credentials in Kubernetes, you store them in Azure Key Vault and configure the driver to mount them into pods at runtime. Secrets never land in etcd.

On AKS, the CSI driver is available as a managed add-on, which means Microsoft handles installation and upgrades. Enable it when creating a cluster or add it to an existing cluster:

# Enable on an existing cluster
az aks enable-addons \
  --resource-group myResourceGroup \
  --name myAKSCluster \
  --addons azure-keyvault-secrets-provider

# Verify the DaemonSet is running
kubectl get daemonset \
  -n kube-system \
  -l app=secrets-store-csi-driver

The add-on installs two DaemonSets: the CSI driver itself and the Azure Key Vault provider. Both run on every node. When a pod is scheduled that references a SecretProviderClass, the driver contacts Key Vault and mounts the requested secrets before the pod starts.

Authentication from the driver to Key Vault uses the cluster’s managed identity. The add-on creates a user-assigned managed identity for the Secrets Provider automatically, or you can configure Workload Identity for per-pod identity granularity.

Creating a SecretProviderClass

A SecretProviderClass is a custom resource that tells the CSI driver which Key Vault to connect to, which secrets to fetch, and how to map them. You create one per set of secrets your application needs.

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: app-secrets
  namespace: production
spec:
  provider: azure
  parameters:
    usePodIdentity: "false"
    useVMManagedIdentity: "true"
    userAssignedIdentityID: "<client-id-of-managed-identity>"
    keyvaultName: "myapp-keyvault"
    tenantId: "<your-tenant-id>"
    objects: |
      array:
        - |
          objectName: db-password
          objectType: secret
          objectVersion: ""
        - |
          objectName: api-key
          objectType: secret
          objectVersion: ""
  secretObjects:
    - secretName: app-k8s-secret
      type: Opaque
      data:
        - objectName: db-password
          key: password
        - objectName: api-key
          key: apiKey

The objects array lists which Key Vault secrets to fetch. The secretObjects section is optional — it causes the driver to also create a regular Kubernetes Secret synced from Key Vault. This is useful for workloads that consume secrets as environment variables rather than mounted files.

Get the tenant ID and managed identity client ID with:

# Get tenant ID
az account show --query tenantId -o tsv

# Get the client ID of the secrets provider identity
az aks show \
  --resource-group myResourceGroup \
  --name myAKSCluster \
  --query addonProfiles.azureKeyvaultSecretsProvider.identity.clientId \
  -o tsv

Grant the managed identity access to Key Vault secrets:

az keyvault set-policy \
  --name myapp-keyvault \
  --object-id <managed-identity-object-id> \
  --secret-permissions get list

If you are using the newer Azure RBAC model for Key Vault (recommended), assign the role instead:

az role assignment create \
  --role "Key Vault Secrets User" \
  --assignee <managed-identity-client-id> \
  --scope /subscriptions/<sub-id>/resourceGroups/myRG/providers/Microsoft.KeyVault/vaults/myapp-keyvault

Mounting secrets into pods

Reference the SecretProviderClass in your pod spec using a CSI volume. The secrets appear as files in the mount path. Each secret becomes a file named after the Key Vault object name.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
  namespace: production
spec:
  replicas: 2
  selector:
    matchLabels:
      app: myapp
  template:
    metadata:
      labels:
        app: myapp
    spec:
      containers:
        - name: app
          image: myregistry.azurecr.io/myapp:1.4.0
          volumeMounts:
            - name: secrets-vol
              mountPath: /mnt/secrets
              readOnly: true
          env:
            # Reference the synced Kubernetes Secret for env vars
            - name: DB_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: app-k8s-secret
                  key: password
      volumes:
        - name: secrets-vol
          csi:
            driver: secrets-store.csi.k8s.io
            readOnly: true
            volumeAttributes:
              secretProviderClass: app-secrets

With this setup, /mnt/secrets/db-password contains the raw secret value fetched from Key Vault. The file is available only inside the container and is not stored in etcd. When a secret is rotated in Key Vault, the driver polls for changes (default every two minutes) and updates the mounted file. Applications that read the file on each use will pick up rotations automatically; applications that cache the value at startup will need to be restarted.

Files vs environment variables for secrets

The CSI driver mounts secrets as files. If your application reads configuration from environment variables, use the secretObjects field to sync a Kubernetes Secret, then reference that Secret in your env block.

For maximum security, use file mounts exclusively and write your application to read secrets from files at runtime. This keeps credentials entirely out of etcd and supports rotation without pod restarts.