How to Secure Kubernetes Secrets — K8s Security Guide

Kubernetes Secrets let you store and manage sensitive information — passwords, API tokens, TLS certificates, SSH keys — and inject them into pods at runtime. They sound secure by name, but out of the box they have significant security limitations that many teams don’t realize until it’s too late.

The Core Problem: Base64 is Not Encryption

Kubernetes Secrets are stored in etcd encoded in base64. Base64 is an encoding, not encryption — anyone who can read etcd can decode every secret in your cluster instantly.

# Any cluster admin can decode a secret in seconds
kubectl get secret my-secret -o jsonpath='{.data.password}' | base64 --decode

This means that if an attacker gains access to your etcd cluster, your Kubernetes API server backup, or any admin-level kubeconfig, they have all your secrets in plaintext.

Understanding the Default Risk

What K8s Secrets protect against:

• Applications in other pods (a pod can only access secrets explicitly mounted to it)
• Accidental exposure in pod specs (secrets appear as environment variable names, not values)

What K8s Secrets do NOT protect against:

• etcd reads (stored unencrypted by default)
• Kubernetes backups (often include etcd)
• Cluster admin access
• Audit log exposure
• Container environment variable dumps (kubectl exec -- env)

1. Enable Encryption at Rest

Configure the kube-apiserver to encrypt secrets before writing them to etcd:

# /etc/kubernetes/enc/enc.yaml
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
    providers:
      - aescbc:
          keys:
            - name: key1
              secret: <base64-encoded-32-byte-key>
      - identity: {}

Apply with: kube-apiserver --encryption-provider-config=/etc/kubernetes/enc/enc.yaml

After enabling, encrypt existing secrets:

kubectl get secrets --all-namespaces -o json | kubectl replace -f -

2. Restrict etcd Access

etcd should only be accessible from the kube-apiserver, never directly from application nodes or developer machines:

# etcd should use TLS with client certificate authentication
ETCD_CLIENT_CERT_AUTH=true
ETCD_TRUSTED_CA_FILE=/etc/kubernetes/pki/etcd/ca.crt
ETCD_CERT_FILE=/etc/kubernetes/pki/etcd/server.crt
ETCD_KEY_FILE=/etc/kubernetes/pki/etcd/server.key

Firewall all other access to the etcd ports (2379, 2380).

3. Implement RBAC for Secrets

Not every namespace, service account, or user should be able to read secrets:

# Role: read specific secrets in a namespace
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: production
  name: secret-reader
rules:
  - apiGroups: [""]
    resources: ["secrets"]
    resourceNames: ["db-credentials", "api-key"] # Specific secrets only
    verbs: ["get"]

# Never grant broad secrets access:
# - resources: ["secrets"]
#   verbs: ["get", "list", "watch"]  # "list" lets you see all secret names

Audit existing RBAC policies regularly:

kubectl auth can-i get secrets --as=system:serviceaccount:default:myapp -n production

4. Use External Secrets Managers

For production workloads, consider managing secrets outside Kubernetes entirely and syncing them in at runtime using an operator:

External Secrets Operator syncs from AWS Secrets Manager, HashiCorp Vault, GCP Secret Manager, or Azure Key Vault:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: db-credentials
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secrets-store
    kind: ClusterSecretStore
  target:
    name: db-credentials
  data:
    - secretKey: password
      remoteRef:
        key: prod/myapp/database
        property: password

This keeps the actual secret value in your secrets manager and enables centralized rotation, auditing, and access control.

5. Avoid Environment Variables for Sensitive Secrets

Environment variables are convenient but leak easily — they appear in process lists, log files, crash dumps, and kubectl exec -- env output.

# LESS SECURE: Secret as environment variable
env:
  - name: DB_PASSWORD
    valueFrom:
      secretKeyRef:
        name: db-credentials
        key: password

# MORE SECURE: Secret as mounted file, read once at startup
volumeMounts:
  - name: secrets
    mountPath: /run/secrets
    readOnly: true
volumes:
  - name: secrets
    secret:
      secretName: db-credentials

Read secrets from files at application startup rather than from environment variables to reduce exposure surface.

6. Enable Audit Logging for Secret Access

Configure the API server to log all secret access events:

# audit-policy.yaml
apiVersion: audit.k8s.io/v1
kind: Policy
rules:
  - level: Metadata
    resources:
      - group: ""
        resources: ["secrets"]

Monitor for:

• Unexpected secret reads outside business hours
• Service accounts reading secrets they shouldn’t need
• Bulk secret reads (potential data exfiltration)

7. Rotate Secrets Regularly

Secrets that never rotate compound the damage of any exposure. Build rotation into your secret lifecycle:

1. Generate new credential
2. Update secret in Kubernetes (or secrets manager)
3. Rolling restart of pods that use the secret
4. Verify new credential works
5. Invalidate old credential

Automate this with tools like external-secrets (which can trigger rotation) or cloud provider key rotation services.

Summary

Control	Default K8s	Hardened K8s
Secrets at rest	Base64 encoded	AES-CBC or KMS encrypted
etcd access	Certificates only	Certificates + firewall
RBAC	Permissive	Least-privilege per service account
Secret storage	In-cluster etcd	External secrets manager
Delivery to pods	Env vars	Mounted files preferred
Auditing	Basic	Full secret access audit log

Kubernetes secrets can be secured properly — but it requires explicit configuration. The defaults are not sufficient for production environments handling sensitive credentials.