Production-Grade Secrets Management in Kubernetes: Tools, Patterns, and Real-World Configurations

Today's Kubernetes clusters are the backbone of cloud-native deployments, but storing secrets—like database credentials, API tokens, and certificates—remains a critical risk. With high-profile breaches and new compliance mandates (like PCI DSS v4.0 and GDPR) in 2024–2025, production teams must adopt secrets management strategies that scale, audit, and rotate credentials without developer friction.

What Is Kubernetes Secrets Management (and Why Is It Hard)?

A Kubernetes secret is an object that stores sensitive data (e.g., passwords, OAuth tokens) in base64-encoded form. By default, these secrets are stored unencrypted in etcd, which is only as secure as your etcd access controls. This basic setup is insufficient for production, as it exposes organizations to insider threats, audit failures, and potential data exfiltration. True secrets management means:

• Encrypting secrets at rest and in transit
• Centralizing access control and rotation policies
• Auditing all reads/changes for compliance
• Integrating with external secret stores like HashiCorp Vault or AWS Secrets Manager

Here's a real-world example: using Kubernetes External Secrets Operator (ESO, v0.9.x) to sync secrets from HashiCorp Vault into your cluster with RBAC and automatic rotation:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: db-credentials
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: vault-backend
    kind: ClusterSecretStore
  target:
    name: db-credentials
    creationPolicy: Owner
  data:
    - secretKey: username
      remoteRef:
        key: secret/data/prod/db
        property: username
    - secretKey: password
      remoteRef:
        key: secret/data/prod/db
        property: password

Key insight: Kubernetes' native secrets are not enough—production clusters need robust, auditable, and automated secrets management patterns.

Step 1: Enabling Encryption at Rest in Kubernetes

Why Encryption at Rest Matters

Without encryption at rest, anyone with access to the etcd datastore can read all secrets in plain text. Since etcd is a common attack vector, enabling encryption is foundational.

How to Enable Encryption at Rest

1. Edit or create the encryption configuration file (e.g., /etc/kubernetes/encryption-config.yaml):

   apiVersion: apiserver.config.k8s.io/v1
   kind: EncryptionConfiguration
   resources:
     - resources:
         - secrets
       providers:
         - aescbc:
             keys:
               - name: key1
                 secret: <base64-encoded-256-bit-key>
         - identity: {}
   

2. Update the API server manifest to reference the file:

   - --encryption-provider-config=/etc/kubernetes/encryption-config.yaml
   

3. Rotate encryption keys regularly (at least every 90 days in regulated environments).

Validating Encryption

Run kubectl get secrets -o yaml and confirm values are not plain base64-encoded text from your config. Also, test backup/restore to verify key integrity.

Key insight: Encryption at rest with AES-CBC is a Kubernetes best practice and a regulatory requirement for most enterprises.

Step 2: Integrating External Secret Stores for Dynamic Secrets

Why External Stores?

Kubernetes secrets don’t natively support dynamic rotation, strong RBAC, or cross-cluster use cases. External secret managers like HashiCorp Vault (v1.14.x+), AWS Secrets Manager, and Azure Key Vault provide:

• Dynamic secrets (e.g., auto-rotated DB credentials)
• Strong authentication (e.g., Kubernetes ServiceAccount JWTs)
• Audit trails and centralized access policies

Connecting With External Secrets Operator (ESO)

1. Deploy ESO:

   helm repo add external-secrets https://charts.external-secrets.io
   helm install external-secrets external-secrets/external-secrets --namespace external-secrets --version 0.9.10
   

2. Configure a SecretStore: (example for Vault)

   apiVersion: external-secrets.io/v1beta1
   kind: ClusterSecretStore
   metadata:
     name: vault-backend
   spec:
     provider:
       vault:
         server: "https://vault.example.com:8200"
         path: ""
         version: "v2"
         auth:
           kubernetes:
             mountPath: /v1/auth/kubernetes
             role: k8s-app
             serviceAccountRef:
               name: external-secrets-sa
   

3. Use ExternalSecret resources as shown earlier to sync data.

Auditing and Rotation

Vault and AWS Secrets Manager support detailed access logs and rotation policies. Configure periodic rotation, e.g., every 30 days for DB passwords, and monitor access using native audit logs.

Key insight: Integrating Kubernetes with a dedicated secret manager centralizes control, enables just-in-time credentials, and satisfies enterprise audit requirements.

Step 3: Sealing Secrets for GitOps Workflows

The Challenge: Secrets in Git

Storing plain secrets in Git repositories is a common anti-pattern. Encrypting secrets so they can be safely versioned and decrypted only in the target cluster is a game-changer for GitOps.

Using Sealed Secrets (Bitnami, v0.25.x)

1. Install the Sealed Secrets controller:

   kubectl apply -f https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.25.0/controller.yaml
   

2. Install the kubeseal CLI:

   brew install kubeseal
   # or download from GitHub releases
   

3. Seal a secret:

   kubectl create secret generic db-creds --from-literal=username=produser --from-literal=password=Prod-P@ssw0rd -o yaml --dry-run=client |
     kubeseal --controller-namespace=kube-system --format=yaml > db-creds-sealed.yaml
   

4. Commit db-creds-sealed.yaml to Git. The controller will decrypt it in-cluster.

Key Considerations

• Key rotation: Regularly rotate the controller's TLS keys.
• Namespace scoping: Sealed secrets are only decryptable in the target namespace.

Key insight: Sealed Secrets enables GitOps workflows for sensitive data, balancing security and developer velocity.

Step 4: Automating Secret Rotation and Audit at Scale

Automating with Vault and ESO

• Dynamic secrets: Use Vault's database secret engine to auto-generate DB credentials per pod or deployment.
• Automated rotation: Set max_ttl and rotation_period in Vault, and let ESO refresh secrets in Kubernetes without human intervention.
• Monitoring: Enable Vault's audit logs and use Prometheus/Grafana for ESO controller health.

Example: Auto-Rotating Postgres Passwords

# Vault DB secrets engine config (HCL)
database secrets engine
path "database/roles/app-role" {
  db_name = "postgres-prod"
  creation_statements = [
    "CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; GRANT SELECT ON ALL TABLES IN SCHEMA public TO \"{{name}}\";"
  ]
  default_ttl = "1h"
  max_ttl     = "24h"
}

Key insight: Automated rotation and monitoring are essential for scaling secrets hygiene across hundreds of microservices.

Step 5: RBAC and Least Privilege Secrets Access

Principle of Least Privilege

Never grant cluster-wide access to secrets unless absolutely necessary. Use Kubernetes ServiceAccounts and RBAC policies to scope access to just the namespace, label, or resource needed.

Example: Namespace-Scoped Secret Access

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: production
  name: read-db-secrets
rules:
- apiGroups: [""]
  resources: ["secrets"]
  resourceNames: ["db-credentials"]
  verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: bind-db-secrets
  namespace: production
subjects:
- kind: ServiceAccount
  name: app-sa
roleRef:
  kind: Role
  name: read-db-secrets
  apiGroup: rbac.authorization.k8s.io

Auditing Access

Use kubectl auth can-i and audit logs to confirm only intended pods can access sensitive secrets.

Key insight: RBAC and ServiceAccounts provide fine-grained, auditable control over secrets in Kubernetes.

Secrets Management Tools: Comparison and Trade-offs

Tool	Open Source	Rotation	Audit Trails	Dynamic Secrets	GitOps Friendly	Best for
HashiCorp Vault (v1.14)	Yes	Yes	Yes	Yes	Indirect	Large, regulated orgs
AWS Secrets Manager	No	Yes	Yes	Partial	Indirect	AWS-centric workloads
Azure Key Vault	No	Yes	Yes	Partial	Indirect	Azure-centric workloads
Sealed Secrets (v0.25)	Yes	No*	No	No	Yes	GitOps workflows
External Secrets Operator (v0.9)	Yes	Yes (via backend)	Partial	Yes	Yes	Multi-cloud Kubernetes

*Sealed Secrets supports manual re-sealing after credential rotation.

Key insight: No single tool fits all use cases—choose based on audit, rotation, and workflow requirements.

Frequently Asked Questions

Q: What is the best way to store secrets in Kubernetes for compliance? A: Use an external secret manager (like HashiCorp Vault or AWS Secrets Manager) integrated via an operator (e.g., External Secrets Operator), combined with encryption at rest in etcd and strict RBAC. This combination meets most regulatory requirements.

Q: How often should secrets be rotated in production? A: Industry best practice is to rotate sensitive credentials (like API keys and DB passwords) every 30–90 days, or immediately upon compromise. Dynamic secrets managers can rotate on every use or pod deployment.

Q: Can I safely store secrets in Git with Sealed Secrets? A: Yes, Sealed Secrets encrypts secrets so only the controller in your cluster can decrypt them, making it safe to store sealed secrets in version control. Rotate your sealing keys periodically for best security.

Key Takeaways

• Native Kubernetes secrets alone are insufficient for production—enable encryption at rest and enforce RBAC.
• Integrate an external secrets manager for dynamic secrets, audit, and centralized control.
• Use Sealed Secrets or similar tools to safely store encrypted secrets in GitOps workflows.
• Automate secret rotation and monitoring to reduce manual effort and risk.
• Scope access to secrets tightly with namespace-level RBAC and ServiceAccounts.
• Choose secrets management tools based on your audit, rotation, and GitOps needs.