Kubernetes Secrets on AWS: EKS, Secrets Manager, ESO, and CSI Driver

Simple explanation

A Kubernetes Secret is an API object for storing small sensitive values: passwords, tokens, TLS certificates. You define them in YAML and Kubernetes makes them available to pods as environment variables or mounted files.

The catch: Kubernetes Secrets are base64-encoded, not encrypted. Think of base64 like writing your PIN backwards in a notebook. The notebook is still sitting open on the desk. Anyone who picks it up can read it in seconds. Running echo "c3VwZXJzZWNyZXQ=" | base64 -d takes one second to reverse and reveals the real credential.

This is why teams running production workloads on EKS often move the source of truth to AWS Secrets Manager: a purpose-built secrets store with automatic rotation, CloudTrail audit logging, versioning, and IAM-controlled access. Tools like ESO and the CSI Driver bridge Secrets Manager into the Kubernetes world without requiring your application to change how it reads secrets.

Which option should you use?

How secrets flow on EKS

Here is the end-to-end flow when using AWS Secrets Manager with External Secrets Operator:

1. Secret is stored in AWS Secrets Manager. You create a secret (production/db) with the credential. Secrets Manager handles encryption, versioning, and optional automatic rotation.

2. Workload identity grants access. Your service’s Kubernetes ServiceAccount is annotated with an IAM role via IRSA (IAM Roles for Service Accounts). The IAM role policy allows secretsmanager:GetSecretValue on the specific secret ARN only.

3. ESO syncs the secret. The External Secrets Operator reads the ExternalSecret resource you defined, authenticates to Secrets Manager using the IRSA role, and creates (or updates) a Kubernetes Secret object with the current value.

4. The app reads the value. The pod consumes the Kubernetes Secret as an environment variable or volume-mounted file, exactly the same as if you had created the Secret manually.

5. Rotation propagates automatically. When you rotate the credential in Secrets Manager, ESO detects the change within the refreshInterval you configured and updates the Kubernetes Secret. If you use volume mounts, the file updates in-place without a pod restart.

Native Kubernetes Secrets

A Kubernetes Secret stores data as key-value pairs. Values are base64-encoded in the YAML manifest, but Kubernetes decodes them when injecting them into pods.

apiVersion: v1
kind: Secret
metadata:
  name: db-credentials
  namespace: production
type: Opaque
data:
  username: cHJvZHVzZXI=       # base64 of "produser"
  password: c3VwZXJzZWNyZXQ=   # base64 of "supersecret"

Where native Secrets are acceptable:

• Development or staging environments where you are comfortable with the risk.
• Simple apps where you manage rotation manually and RBAC is narrow.
• Secrets generated by Kubernetes itself: ServiceAccount tokens, TLS certificates issued by cert-manager. These are Kubernetes-native and make sense to stay that way.

Limitations:

• No built-in rotation mechanism.
• No audit trail for who read which secret.
• Base64 encoding provides no confidentiality. Any user or pod with RBAC permission to get the Secret reads the real value.
• If you deploy via kubectl or Helm, raw Secret YAML with real values can end up in shell history, CI logs, or terminal output unless you are careful.

EKS encryption at rest: what it does and does not do

A useful analogy: encryption at rest is like a padlock on a filing cabinet. It stops an intruder from stealing the cabinet and reading its contents. It does nothing to stop an authorized employee who already has a key from opening the drawer during working hours.

In Kubernetes terms, RBAC is the employee with the key. Encryption is the padlock on the cabinet.

Generic Kubernetes behavior: Vanilla Kubernetes clusters store Secrets in etcd without encryption at rest unless you configure envelope encryption with a KMS provider. The data sits base64-encoded but readable from etcd storage directly.

Current EKS defaults: EKS clusters running Kubernetes 1.28 and later have default envelope encryption for Kubernetes API data. Secrets stored in etcd are encrypted using AWS-managed keys without any additional setup on your part. You do not need to manually configure KMS just to get basic encryption at rest on a modern EKS cluster.

Customer-managed KMS: If you want to use your own AWS KMS key for compliance, key rotation control, or cross-account scenarios, you can configure the cluster with a customer-managed key. This is an additional control and a custom-key choice, not a baseline requirement:

aws eks associate-encryption-config \
  --cluster-name my-cluster \
  --encryption-config '[{
    "resources": ["secrets"],
    "provider": {
      "keyArn": "arn:aws:kms:eu-west-1:123456789012:key/your-key-id"
    }
  }]'

What encryption at rest does not solve. Going back to the padlock analogy: encryption does not protect against:

• A pod or user with kubectl exec access reading environment variables from a running container.
• A Kubernetes user or ServiceAccount with RBAC permission to get or list Secrets in the namespace.
• Secret values leaking through application logs or error output.
• Overly broad IAM policies on Secrets Manager granting access to more secrets than intended.

Treat encryption at rest as one layer in a larger posture. RBAC, least-privilege IAM, and cluster security hardening are the higher-priority controls.

AWS Secrets Manager with External Secrets Operator

Think of ESO as a translator that continually checks an authoritative source (Secrets Manager) and keeps a local copy (the Kubernetes Secret) in sync. Your application only ever reads the local copy, but the translator ensures it is always up to date.

External Secrets Operator is an open-source Kubernetes controller that watches ExternalSecret resources and creates matching Kubernetes Secret objects, populated from AWS Secrets Manager. The source of truth stays in Secrets Manager. Your pods consume a standard Kubernetes Secret and nothing in your application changes.

ESO is often the best default for teams that want:

• Automatic rotation without redeploying pods.
• A CloudTrail audit trail for every secret access.
• The same credential shared across multiple clusters or services.
• GitOps-friendly secret management: commit ExternalSecret manifests, not values.

Install ESO via Helm:

helm repo add external-secrets https://charts.external-secrets.io
helm install external-secrets \
  external-secrets/external-secrets \
  -n external-secrets-system \
  --create-namespace

Configure a SecretStore using IRSA for access:

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: aws-secrets-manager
  namespace: production
spec:
  provider:
    aws:
      service: SecretsManager
      region: eu-west-1
      auth:
        jwt:
          serviceAccountRef:
            name: external-secrets-sa

Create an ExternalSecret to pull a specific credential:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: db-credentials
  namespace: production
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secrets-manager
    kind: SecretStore
  target:
    name: db-credentials
    creationPolicy: Owner
  data:
  - secretKey: password
    remoteRef:
      key: production/db
      property: password

ESO creates a Kubernetes Secret named db-credentials in the production namespace. When the credential rotates in Secrets Manager, ESO updates the Kubernetes Secret within the refreshInterval. If your pods use volume mounts, the file updates without a restart.

Secrets Store CSI Driver

Think of the CSI Driver like a USB drive that only exists while it is plugged in. The data never lives on your machine. When the pod starts, the driver fetches the secret directly from Secrets Manager and mounts it as a file. When the pod stops, the file is gone.

The Secrets Store CSI Driver mounts secrets from AWS Secrets Manager directly into pods as files. No Kubernetes Secret object is created. The secret never exists in the Kubernetes API.

This is the right choice when your security policy requires secrets to not be stored as Kubernetes resources, or when compliance requirements demand that secrets remain exclusively in Secrets Manager.

apiVersion: v1
kind: Pod
spec:
  serviceAccountName: my-app-sa   # must have IRSA with GetSecretValue on the specific ARN
  containers:
  - name: app
    image: my-app:1.0.0
    volumeMounts:
    - name: secrets-store
      mountPath: "/mnt/secrets"
      readOnly: true
  volumes:
  - name: secrets-store
    csi:
      driver: secrets-store.csi.k8s.io
      readOnly: true
      volumeAttributes:
        secretProviderClass: "aws-secrets"

The application reads /mnt/secrets/db-password as a file. The secret value is never stored as a Kubernetes object.

Kubernetes Secrets vs AWS Secrets Manager

External Secrets Operator vs Secrets Store CSI Driver

When to use each approach

Database credentials for an EKS application: Use Secrets Manager with ESO. The credential rotates automatically, your DBA team can change it in Secrets Manager without touching Kubernetes, and you have a CloudTrail record of every access.

Third-party API keys: Same recommendation. Store in Secrets Manager and sync with ESO. If the key is compromised, rotate it in one place and every cluster picks up the new value within the refresh interval.

TLS certificates: Use cert-manager to issue and renew certificates as Kubernetes Secrets. If you need to mount an existing certificate from Secrets Manager, the CSI Driver is a clean option.

Runtime-mounted secrets in high-security environments: Use the CSI Driver when your threat model specifically requires the secret to never exist as a Kubernetes object. This is often a compliance requirement rather than a practical security improvement over ESO with tight RBAC.

GitOps teams: Commit ExternalSecret manifests to Git. The manifest contains a reference to the Secrets Manager path, not the value. This fits naturally into Argo CD or Flux workflows. See also secrets in CI/CD pipelines for the pipeline side of this pattern.

Private clusters: On a private EKS cluster, ensure your VPC has a Secrets Manager VPC endpoint so ESO and the CSI Driver can reach the API without traversing the public internet.

Observability integrations: When connecting monitoring tools to your EKS cluster, ensure agents do not log secret values. Environment variable leakage through observability pipelines is a common, underappreciated issue.

Environment variables vs volume-mounted files

Once you have a Kubernetes Secret, you inject it into pods as environment variables or volume-mounted files.

Environment variables:

env:
- name: DB_PASSWORD
  valueFrom:
    secretKeyRef:
      name: db-credentials
      key: password

Volume mount (generally safer):

volumeMounts:
- name: db-secret
  mountPath: "/secrets"
  readOnly: true
volumes:
- name: db-secret
  secret:
    secretName: db-credentials

Best-practice checklist

• Least-privilege IAM: Grant each ServiceAccount access only to the specific Secrets Manager ARNs it needs. No wildcards.
• IRSA on every service: Use IRSA to give pods IAM access to Secrets Manager. Never store AWS credentials inside a Kubernetes Secret.
• Narrow RBAC: Restrict get and list on Secrets to the ServiceAccounts that genuinely need them. Audit this regularly.
• Source of truth in Secrets Manager for production: Reserve native Kubernetes Secrets for development, staging, and Kubernetes-internal secrets such as cert-manager TLS.
• Prefer volume mounts: Mount secrets as files with restrictive permissions rather than env vars.
• Log hygiene: Confirm your application code, logging framework, and monitoring agents do not print secret values on startup or in error output.
• Rotation plan: Define a rotation schedule and test it before going live. Automatic rotation in Secrets Manager with ESO handling the sync is the simplest path.
• CloudTrail for Secrets Manager: Ensure API logging is enabled so you have an audit trail of every secret access.
• VPC endpoint for Secrets Manager: Especially on private clusters, avoid routing secret fetches over the public internet.