Managing Secrets in Kubernetes on GKE: Kubernetes Secrets vs Secret Manager

What is a Kubernetes Secret in plain English?

Imagine your app needs a database password to run. You cannot hardcode it in the container image — that would expose it to anyone who pulls the image. Kubernetes Secrets give you a dedicated place to store sensitive values separately from your application code and inject them into pods at runtime.

A Kubernetes Secret is a small Kubernetes object that holds key-value pairs: a name like db-credentials and values like password=s3cr3t. Kubernetes stores these objects in etcd, the same internal database that backs all cluster state. When a pod starts, Kubernetes delivers the secret value as an environment variable or as a file the container can read.

Here is the critical thing beginners get wrong: the values are base64-encoded, not encrypted. Base64 is a text encoding scheme used to make binary data safe to store in YAML. It is trivially reversible.

How Kubernetes Secrets work: the full flow

Understanding the end-to-end data flow helps you reason about where security controls apply and where they do not:

1. You create a Secret object. Using kubectl create secret or a YAML manifest, you create a Secret in a namespace. The values are base64-encoded in the YAML but decoded back to raw bytes when stored.
2. Kubernetes writes it to etcd. etcd is the distributed key-value store that backs all cluster state. By default, GCP encrypts etcd data at rest with AES-256 and Google-managed keys. With application-layer encryption enabled, an additional Cloud KMS key you control wraps the Secret data before it reaches etcd.
3. Your pod references the Secret. In the pod spec you declare that the pod should receive values from a named Secret: as environment variables or as files in a mounted volume.
4. Kubernetes delivers the value at runtime. When the pod starts, the kubelet on the node retrieves the Secret from the API server and either sets environment variables or writes files to the container’s filesystem. The values are not stored on the node’s disk permanently.
5. RBAC controls who can read Secrets. Only principals with an explicit get or list permission on the secrets resource can read Secret objects via the API. The kubelet fetches only the secrets the pod actually references.
6. Optional extra protection comes from Cloud KMS or Secret Manager. For production workloads, you can either encrypt the etcd data with a Cloud KMS key you control, or move secrets out of etcd entirely using Secret Manager synced via the External Secrets Operator.

What a Kubernetes Secret is

A Kubernetes Secret is an API object that stores key-value pairs of sensitive data. The critical fact: Kubernetes Secrets are base64-encoded, not encrypted. Base64 is a text encoding scheme that makes binary data safe to embed in YAML. It is trivially reversible:

echo "cGFzc3dvcmQ=" | base64 --decode
# outputs: password

The security of a Kubernetes Secret depends entirely on access controls: RBAC, network access to the API server, and encryption of etcd at rest. The base64 encoding itself provides no protection.

The difference between “hidden from casual view” and “actually protected” matters a great deal in production. A Secret value in etcd is accessible to anyone with cluster admin privileges, anyone who can exec into the running container, and anyone who can read the Secret object via the API. RBAC is your primary defence at the Kubernetes layer.

Creating and inspecting Secrets

The most common way to create a Secret is kubectl create secret generic, which creates an opaque (arbitrary key-value) Secret. You can create secrets from literal values, from files, or from a YAML manifest.

# Create a secret from literal values
kubectl create secret generic db-credentials \
  --from-literal=username=myapp \
  --from-literal=password=s3cr3t-p@ssword

# Create a secret from files (file contents become the value)
kubectl create secret generic tls-cert \
  --from-file=tls.crt=./server.crt \
  --from-file=tls.key=./server.key

# View the secret — values appear base64-encoded
kubectl get secret db-credentials -o yaml

# Decode a specific value
kubectl get secret db-credentials \
  -o jsonpath='{.data.password}' | base64 --decode

# Delete a secret
kubectl delete secret db-credentials

You can also define Secrets in YAML manifests (common in automated pipelines), but never commit real base64-encoded secret values to source control. See Secrets in CI/CD pipelines for how to handle this properly in automated workflows.

# db-secret.yaml — DO NOT commit real values to git
apiVersion: v1
kind: Secret
metadata:
  name: db-credentials
  namespace: default
type: Opaque
data:
  username: bXlhcHA=      # base64 of "myapp"
  password: czNjcjN0      # base64 of "s3cr3t"

Consuming Secrets in pods

Pods access Secrets in two ways: as environment variables injected at container start, or as files mounted into a volume. Each approach has different trade-offs worth understanding before choosing.

Environment variables are the simpler pattern. The Secret value is decoded and set as an environment variable when the container starts:

spec:
  containers:
    - name: app
      image: my-app:v1
      env:
        - name: DB_USERNAME
          valueFrom:
            secretKeyRef:
              name: db-credentials
              key: username
        - name: DB_PASSWORD
          valueFrom:
            secretKeyRef:
              name: db-credentials
              key: password

Volume mounts write each key as a file in the specified directory. Kubernetes automatically updates the files when the Secret changes (with a short delay), so applications that re-read files periodically can pick up rotated credentials without a restart:

spec:
  containers:
    - name: app
      image: my-app:v1
      volumeMounts:
        - name: db-secret-vol
          mountPath: /etc/secrets
          readOnly: true
  volumes:
    - name: db-secret-vol
      secret:
        secretName: db-credentials

After mounting, the container sees /etc/secrets/username and /etc/secrets/password as plain-text files.

Comparison at a glance:

For values that rotate regularly, volume mounts are the better default. For deployments that already consume environment variables, use kubectl rollout restart after updating the Secret.

When to use Kubernetes Secrets

Plain Kubernetes Secrets are a reasonable choice in the following situations:

• Internal dev or staging clusters where the risk of exposure is lower and operational simplicity matters more
• Low-sensitivity values such as non-production API keys, feature flags with minor security implications, or internal service configuration
• Simple secret injection where your app just needs a value as an environment variable and Secret Manager integration would add unnecessary complexity
• Teams early in their GKE journey who have not yet set up Workload Identity or Secret Manager

They are generally not enough for:

• Production database credentials or any secret that would cause real damage if exposed
• Regulated environments where audit logging of every secret access is required
• Secrets that need automated rotation with immediate propagation
• Multi-cluster or multi-environment setups where a single source of truth reduces operational risk
• Any scenario where secrets must not live in etcd due to compliance requirements

Kubernetes Secrets vs Secret Manager vs External Secrets Operator

This is the decision most teams face when building production workloads on GKE. Each option solves a different part of the problem.

Kubernetes Secrets

• Where the secret lives: etcd inside your cluster
• Access control: Kubernetes RBAC, roles and role bindings within the cluster
• Rotation model: manual update of the Secret object, then rolling restart for env var consumers
• Operational complexity: low, built into Kubernetes with no extra setup
• Best fit: dev/staging workloads, simple injection needs, teams new to GKE

Google Cloud Secret Manager

• Where the secret lives: fully managed GCP service, outside the cluster entirely
• Access control: IAM, per-secret and per-service-account permissions with full audit logging in Cloud Audit Logs
• Rotation model: create a new secret version; notifications via Pub/Sub for automated rotation
• Operational complexity: medium, requires Workload Identity and application code changes (or ESO)
• Best fit: production workloads, regulated environments, teams that need auditable access

External Secrets Operator (ESO)

• Where the secret lives: Secret Manager (or another external store) as the source of truth; ESO syncs copies into Kubernetes Secrets
• Access control: IAM for Secret Manager access; RBAC for the synced Kubernetes Secret
• Rotation model: update Secret Manager version; ESO syncs automatically at the configured refresh interval
• Operational complexity: higher, requires installing ESO and configuring SecretStore and ExternalSecret resources
• Best fit: teams that want Secret Manager as source of truth but cannot modify app code to call the Secret Manager API directly

Encryption at rest in GKE

GCP encrypts all data at rest by default, including etcd data on GKE clusters, using AES-256 with GCP-managed keys. This protects against physical media theft but leaves the encryption key under Google’s control.

Application-layer encryption adds an additional encryption layer using a key you own in Cloud KMS. With it enabled, Kubernetes encrypts Secret data using your Cloud KMS key before writing to etcd. Even if someone obtains an etcd snapshot, they cannot read Secret values without access to your Cloud KMS key, which is governed by IAM and audit-logged separately.

# 1. Create a Cloud KMS key ring and key
gcloud kms keyrings create gke-secrets-ring \
  --location=europe-west2

gcloud kms keys create gke-secrets-key \
  --location=europe-west2 \
  --keyring=gke-secrets-ring \
  --purpose=encryption

# 2. Grant the GKE service account permission to use the key
PROJECT_NUMBER=$(gcloud projects describe $PROJECT_ID \
  --format='value(projectNumber)')

gcloud kms keys add-iam-policy-binding gke-secrets-key \
  --location=europe-west2 \
  --keyring=gke-secrets-ring \
  --member="serviceAccount:service-${PROJECT_NUMBER}@container-engine-robot.iam.gserviceaccount.com" \
  --role=roles/cloudkms.cryptoKeyEncrypterDecrypter

# 3. Create a cluster with application-layer encryption
KEY_URI="projects/$PROJECT_ID/locations/europe-west2/keyRings/gke-secrets-ring/cryptoKeys/gke-secrets-key"

gcloud container clusters create my-cluster \
  --region=europe-west2 \
  --database-encryption-key=$KEY_URI

Using Secret Manager with GKE

For production workloads, many teams prefer to store secrets outside the cluster entirely in Google Cloud Secret Manager. Secrets never touch etcd, access is governed by IAM, and every read is logged.

Secret Manager provides:

• Versioned secrets with automatic replication
• Fine-grained IAM: grant access per secret, per service account
• Automatic rotation notifications via Pub/Sub (see Rotating Secrets Automatically)
• Full audit logging in Cloud Audit Logs — every access is recorded
• Secrets never stored in etcd or in any Kubernetes object

The dependency: your application must call the Secret Manager API at runtime. This requires the pod’s identity via Workload Identity to have IAM access to the secret. Without Workload Identity, you would need to manage static service account keys, which creates a different secret management problem.

# Create a secret in Secret Manager
echo -n "my-db-password" | gcloud secrets create db-password \
  --data-file=- \
  --replication-policy=automatic

# Add a new version
echo -n "my-new-db-password" | gcloud secrets versions add db-password \
  --data-file=-

# Grant a Kubernetes service account access via Workload Identity
gcloud secrets add-iam-policy-binding db-password \
  --member="serviceAccount:my-project.svc.id.goog[default/my-ksa]" \
  --role=roles/secretmanager.secretAccessor

In practice, retrieve secrets from application code using a Google Cloud client library (Python, Go, Java, Node.js) rather than curl. All major runtimes have official libraries. The retrieval happens at application startup or on demand, not via Kubernetes mechanisms.

External Secrets Operator

The External Secrets Operator (ESO) is a Kubernetes controller that reads secrets from external stores (Secret Manager, AWS Secrets Manager, HashiCorp Vault, and others) and creates or updates Kubernetes Secret objects automatically.

The motivation: many applications are written to consume secrets as environment variables or mounted files. Refactoring them to call the Secret Manager API directly requires code changes. ESO lets you keep Secret Manager as the source of truth while your pods continue consuming secrets the normal Kubernetes way, with no application code changes required.

ESO syncs values on a configurable interval. When you rotate a secret in Secret Manager, ESO picks up the new version at the next refresh and updates the Kubernetes Secret. Pods using volume mounts see the update automatically; pods using environment variables still need a rolling restart.

# Install ESO via Helm
helm repo add external-secrets https://charts.external-secrets.io
helm install external-secrets external-secrets/external-secrets \
  --namespace external-secrets \
  --create-namespace

# SecretStore — tells ESO where to fetch secrets from
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: gcp-secret-store
  namespace: default
spec:
  provider:
    gcpsm:
      projectID: my-project-id

---
# ExternalSecret — maps a Secret Manager secret to a Kubernetes Secret
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: db-credentials
  namespace: default
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: gcp-secret-store
    kind: SecretStore
  target:
    name: db-credentials       # the Kubernetes Secret to create/update
    creationPolicy: Owner
  data:
    - secretKey: password      # key in the Kubernetes Secret
      remoteRef:
        key: db-password       # secret name in Secret Manager
        version: latest

With this configuration, ESO creates a Kubernetes Secret called db-credentials and keeps it synchronised with the db-password secret in Secret Manager, refreshing every hour.

Restricting access to Secrets with RBAC

Regardless of where secrets are stored, restrict which Kubernetes principals can read Secret objects. The built-in view ClusterRole does not grant access to Secrets, but edit and admin do. Binding either cluster-wide gives the subject read access to all Secrets in all namespaces. See Securing GKE Clusters for how RBAC fits into a broader cluster security model.

Use resourceNames to restrict access to specific named Secrets rather than all Secrets in a namespace. This follows the principle of least privilege: a workload that needs one credential should not be able to read every Secret in the namespace.

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: read-db-secret
  namespace: default
rules:
  - apiGroups: [""]
    resources: ["secrets"]
    resourceNames: ["db-credentials"]
    verbs: ["get"]

# Bind the role to a service account
kubectl create rolebinding app-read-db-secret \
  --role=read-db-secret \
  --serviceaccount=default:my-app-sa \
  --namespace=default

# Verify access
kubectl auth can-i get secrets \
  --namespace=default \
  --as=system:serviceaccount:default:my-app-sa

Rotating secrets

Secret rotation (replacing a secret value with a new one) limits the exposure window if a secret leaks. The mechanics depend on how the secret is consumed:

• Volume-mounted secrets: Kubernetes automatically updates the files inside the mounted volume within about a minute when the Secret is updated. Applications that re-read files pick up new values without a restart.
• Environment variable secrets: Environment variables are fixed at container start and never update. The pod must be restarted to pick up a new value:

# Rolling restart to pick up updated secret env vars (no downtime)
kubectl rollout restart deployment/my-app

# Monitor the rollout
kubectl rollout status deployment/my-app

• Secret Manager with ESO: Create a new secret version in Secret Manager. ESO picks it up at the next refresh interval and updates the Kubernetes Secret. Pods using volume mounts see the updated file automatically; pods using environment variables still need a rolling restart.

For automated rotation, Secret Manager supports rotation notifications via Pub/Sub. See Rotating Secrets Automatically for how to trigger rotation and update secret versions without manual steps.

Best practices

• Prefer Secret Manager for production-sensitive secrets. Keeps secrets out of etcd, enables IAM-based access control, and provides a full audit trail.
• Use Workload Identity instead of static service account keys. Service account key files are themselves a secret management problem. Workload Identity eliminates them.
• Restrict RBAC narrowly. Use namespace-scoped Roles with resourceNames rather than cluster-wide roles. Never grant edit or admin more broadly than necessary.
• Avoid storing real secret values in repos or manifests. Use placeholder values and inject real values at deploy time, or use ESO to pull from Secret Manager.
• Rotate secrets regularly. Automate rotation with Secret Manager and Pub/Sub where possible. Use kubectl rollout restart to propagate rotated env var secrets without downtime.
• Prefer volume mounts when live refresh matters. Volume-mounted secrets update automatically; environment variables do not.
• Enable application-layer KMS encryption. For production clusters, adding a Cloud KMS layer means etcd backups are unreadable without your key.
• Audit secret access. Enable Cloud Audit Logs and set up alerting for unexpected secret reads, especially in production namespaces.

Frequently asked questions

Are Kubernetes Secrets encrypted by default?

No. Kubernetes Secrets are base64-encoded, which is encoding not encryption. Anyone with read access to the Secret object can decode the value instantly. In GKE you can enable application-layer encryption to encrypt Secret data in etcd using a Cloud KMS key you control. This adds genuine encryption on top of GCP’s default AES-256 at-rest encryption, but it still does not prevent a cluster admin with API access from reading Secret objects directly.

What is the difference between Kubernetes Secrets and Secret Manager?

Kubernetes Secrets live inside your cluster’s etcd and are accessed via RBAC. Secret Manager is a fully managed GCP service that stores secrets outside the cluster entirely, with versioning, IAM-based access control, and full audit logging. For production credentials, Secret Manager is usually the better choice because secrets are not in etcd, access is governed by IAM, and every read is logged in Cloud Audit Logs.

Should I store production secrets in Kubernetes?

It depends on sensitivity. Plain Kubernetes Secrets are fine for dev/staging or low-sensitivity values. For production credentials like database passwords or API keys, prefer Secret Manager. The key test: if exposure would cause a real incident, move it to Secret Manager.

Do pods automatically pick up updated secrets?

Volume-mounted secrets update automatically in the running container within about a minute. Environment variables are fixed at container start. Pods must be restarted (use kubectl rollout restart deployment/my-app) to pick up updated values.

What is the safest way to manage secrets on GKE?

Store secrets in Secret Manager, use Workload Identity to grant pods IAM access, and use the External Secrets Operator to sync values into Kubernetes Secrets automatically. Add narrow RBAC, application-layer KMS encryption on etcd, and Cloud Audit Log alerts for unexpected secret access.