Managing Kubernetes Secrets and ConfigMaps: Best Practices

Table of Contents

1. Understanding Secrets and ConfigMaps
   • What Are Secrets?
   • What Are ConfigMaps?
   • Key Differences Between Secrets and ConfigMaps
2. Best Practices for Secrets Management
   • Avoid Committing Secrets to Version Control
   • Use Encryption (Not Just Encoding)
   • Limit Access with RBAC
   • Rotate Secrets Regularly
   • Leverage External Secret Managers
   • Avoid Exposing Secrets in Logs or Outputs
   • Enforce Immutable Secrets
   • Scope Secrets to Specific Namespaces
   • Validate Secrets Before Deployment
   • Use Secret Injection Tools
3. Best Practices for ConfigMaps Management
   • Never Store Sensitive Data in ConfigMaps
   • Use Immutable ConfigMaps
   • Manage ConfigMaps as Code
   • Choose Between Environment Variables and Volumes
   • Limit ConfigMap Size
   • Version ConfigMaps
   • Avoid Overwriting Files with SubPath
   • Enable Dynamic Config Updates
4. Tools for Enhanced Secrets and ConfigMaps Management
   • External Secret Managers
   • Kubernetes-Native Tools
   • Monitoring and Compliance Tools
5. Conclusion
6. References

Understanding Secrets and ConfigMaps

Before diving into best practices, let’s clarify what Secrets and ConfigMaps are, their use cases, and how they differ.

What Are Secrets?

Secrets are Kubernetes resources designed to store sensitive data such as passwords, API tokens, TLS certificates, or SSH keys. They mitigate the risk of exposing sensitive data by centralizing storage and restricting access.

Key Features:

• Data is stored in key: value pairs.
• Secrets are base64-encoded (not encrypted) by default to avoid accidental exposure in logs or CLI outputs.
• Access is controlled via Kubernetes RBAC (Role-Based Access Control).
• Can be mounted as files in pods or injected as environment variables.

Example Secret:

apiVersion: v1
kind: Secret
metadata:
  name: db-credentials
  namespace: app-team
type: Opaque  # Default type for arbitrary key-value pairs
data:
  db-password: cGFzc3dvcmQxMjM=  # Base64-encoded "password123"
  db-user: YWRtaW4=              # Base64-encoded "admin"

What Are ConfigMaps?

ConfigMaps manage non-sensitive configuration data, such as environment variables, command-line arguments, or configuration files (e.g., app.properties). They decouple configuration from application code, enabling dynamic updates without rebuilding container images.

Key Features:

• Data is stored in plaintext key: value pairs (no encoding).
• Like Secrets, they can be mounted as files or injected as environment variables.
• Ideal for storing feature flags, service URLs, or log levels.

Example ConfigMap:

apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config
  namespace: app-team
data:
  LOG_LEVEL: "info"
  API_ENDPOINT: "https://api.example.com/v1"
  app.properties: |
    server.port=8080
    database.connection.pool.size=10

Key Differences Between Secrets and ConfigMaps

Best Practices for Secrets Management

Secrets protect your most critical data—missteps here can lead to data leaks or unauthorized access. Follow these practices to secure them effectively.

1. Never Commit Secrets to Version Control

Risk: Secrets stored in Git or other version control systems (VCS) are exposed to anyone with repository access, including past employees or attackers.
Solution:

• Use .gitignore to exclude Secret YAML files.
• Store secrets externally (e.g., in a vault) and inject them into the cluster at deploy time.
• Tools like Sealed Secrets encrypt secrets for safe storage in VCS.

2. Encrypt Secrets at Rest and in Transit

Kubernetes stores Secrets in etcd (the cluster’s database). By default:

• Secrets are base64-encoded (not encrypted), so anyone with etcd access can decode them.
• etcd data is unencrypted at rest unless configured otherwise.

Solutions:

• Enable etcd Encryption at Rest: Use Kubernetes’ built-in encryption to encrypt Secrets in etcd.

  # Example encryption configuration (stored in /etc/kubernetes/encryption-config.yaml)
  apiVersion: apiserver.config.k8s.io/v1
  kind: EncryptionConfiguration
  metadata:
    name: encryption-config
  resources:
    - resources:
      - secrets
      providers:
      - aescbc:
          keys:
          - name: key1
            secret: <32-byte-AES-key>  # Generate with `head -c 32 /dev/urandom | base64`
      - identity: {}  # Fallback if encryption fails

• Enable TLS for etcd: Encrypt data in transit between the API server and etcd using TLS.
• Avoid plaintext transmission: Use TLS for all pod-to-secret interactions (e.g., when fetching secrets from external managers).

3. Limit Access with RBAC

By default, Secrets are accessible to anyone with get/list permissions in a namespace. Restrict access using RBAC:

Best Practices:

• Create granular Roles/ClusterRoles with minimal permissions (e.g., get on Secrets only for specific service accounts).
• Avoid using cluster-admin or overly broad roles for applications.
• Example Role for a pod service account:

  apiVersion: rbac.authorization.k8s.io/v1
  kind: Role
  metadata:
    namespace: app-team
    name: secret-reader
  rules:
  - apiGroups: [""]
    resources: ["secrets"]
    resourceNames: ["db-credentials"]  # Restrict to specific secrets
    verbs: ["get", "list"]

4. Rotate Secrets Regularly

Stale secrets are a prime target for attackers. Automate rotation to minimize exposure:

• How to Implement:
  • Use external secret managers (e.g., Vault) with built-in rotation policies.
  • For Kubernetes-native rotation, use tools like Vault Agent Injector or custom operators to update Secrets and restart pods.
• Example Workflow:
  1. Generate a new secret version in the vault.
  2. Update the Kubernetes Secret with the new value.
  3. Restart pods to load the new secret (or use live reloading with tools like Envoy or application-level watches).

5. Leverage External Secret Managers

Kubernetes Secrets are cluster-scoped and lack advanced features like dynamic rotation, auditing, or integration with cloud providers. Use external tools for enterprise-grade security:

6. Avoid Exposing Secrets in Logs or Outputs

Secrets injected as environment variables may leak into logs if applications log all environment variables.

Solutions:

• Mount Secrets as files (e.g., /etc/secrets/db-password) instead of environment variables.
• Use tools like Envoy or Istio to redact secrets from logs.
• Configure applications to avoid logging sensitive fields (e.g., use logredactor libraries).

7. Enforce Immutable Secrets

Prevent accidental or malicious modification of Secrets by marking them as immutable:

apiVersion: v1
kind: Secret
metadata:
  name: db-credentials
immutable: true  # Prevents updates to the Secret
data:
  db-password: cGFzc3dvcmQxMjM=

Why: Immutable Secrets reduce the risk of unauthorized tampering and simplify auditing (no change history to track).

8. Scope Secrets to Specific Namespaces

Isolate secrets by namespace to limit blast radius if a namespace is compromised:

• Create Secrets in the same namespace as the pods that use them.
• Avoid cluster-scoped secrets unless absolutely necessary (use ClusterSecrets sparingly).

9. Validate Secrets Before Deployment

Use tools to scan Secrets for vulnerabilities before deployment:

• Tools:
  • kube-score: Checks for insecure Secret practices (e.g., missing immutability).
  • Trivy: Scans YAML files for hardcoded secrets.
  • OPA Gatekeeper: Enforces policies (e.g., “Secrets must have immutability enabled”).

10. Use Secret Injection Tools

Avoid manually managing Secrets in YAML. Use tools to inject secrets dynamically:

• Sealed Secrets: Encrypts Secrets into “SealedSecrets” that can be safely stored in Git. The Sealed Secrets controller decrypts them in the cluster.
• Vault Agent Injector: Injects secrets directly into pods from Vault without exposing them as Kubernetes Secrets.

Best Practices for ConfigMaps Management

ConfigMaps handle non-sensitive data, but poor management can still cause outages or configuration drift. Follow these practices to keep configurations reliable and maintainable.

1. Never Store Sensitive Data in ConfigMaps

ConfigMaps are not secured—any user with namespace access can read them. Always use Secrets for sensitive data (e.g., passwords, tokens).

2. Use Immutable ConfigMaps

Like Secrets, mark ConfigMaps as immutable to prevent accidental changes and improve performance (Kubernetes caches immutable ConfigMaps):

apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config
immutable: true  # Prevents updates
data:
  LOG_LEVEL: "info"

3. Manage ConfigMaps as Code

Treat ConfigMaps as code to enable versioning, review, and rollbacks:

• Store ConfigMaps in Git (since they’re non-sensitive) using tools like GitOps pipelines (ArgoCD, Flux).
• Use Kustomize or Helm to template ConfigMaps for different environments (dev, staging, prod).

Example Kustomize Patch for Dev Environment:

# kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- base/configmap.yaml  # Base ConfigMap
patches:
- target:
    kind: ConfigMap
    name: app-config
  patch: |-
    - op: replace
      path: /data/LOG_LEVEL
      value: "debug"  # Override for dev

4. Choose Between Environment Variables and Volumes

ConfigMaps can be injected as environment variables or mounted as files—choose based on your use case:

5. Limit ConfigMap Size

etcd has a default size limit of ~1MB per object. Avoid oversized ConfigMaps:

• Split large configurations into multiple ConfigMaps (e.g., app-db-config and app-log-config).
• Use external configuration tools (e.g., Spring Cloud Config) for very large datasets.

6. Version ConfigMaps

Version ConfigMaps to simplify rollbacks if a bad configuration causes outages:

• Append version numbers to names (e.g., app-config-v1, app-config-v2).
• Use Git tags to track ConfigMap versions in VCS.

7. Avoid Overwriting Files with SubPath

When mounting ConfigMaps as volumes, use subPath to avoid overwriting entire directories:

# Pod spec snippet
volumeMounts:
- name: config-volume
  mountPath: /etc/app/config.properties
  subPath: config.properties  # Only mounts "config.properties" instead of overwriting /etc/app
volumes:
- name: config-volume
  configMap:
    name: app-config

8. Enable Dynamic Config Updates

Avoid pod restarts for ConfigMap changes by using tools that reload configurations dynamically:

• Tools:
  • ConfigMap Reload: Triggers a reload (e.g., for Nginx or Prometheus) when ConfigMaps change.
  • Spring Cloud Kubernetes: For Spring apps, auto-reloads ConfigMaps.
  • Envoy xDS: For service meshes, pushes config updates to Envoy proxies.

Tools for Enhanced Secrets and ConfigMaps Management

Adopting tools can automate and strengthen your secrets and ConfigMaps workflow. Here are key tools to consider:

External Secret Managers

• HashiCorp Vault: Manages secrets with encryption, rotation, and access control. Integrates with Kubernetes via the Vault Operator.
• AWS Secrets Manager: Cloud-native secret storage with auto-rotation and IAM integration. Use the AWS Secrets Operator to inject secrets into pods.
• Sealed Secrets: Encrypts Secrets into “SealedSecrets” that can be stored in Git. Only the cluster’s Sealed Secrets controller can decrypt them.

Kubernetes-Native Tools

• Kustomize: Templates and patches ConfigMaps/Secrets for environment-specific configurations.
• Helm: Uses charts to package and deploy ConfigMaps/Secrets with versioning.
• ConfigMap Reload: Automatically reloads applications when ConfigMaps change.

Monitoring and Compliance Tools

• Prometheus + Grafana: Monitor ConfigMap/Secret changes with metrics (e.g., kube_configmap_created).
• Falco: Detects suspicious activity (e.g., unexpected Secret access) via runtime security rules.
• Trivy: Scans YAML files for hardcoded secrets or misconfigurations.

Conclusion

Effective management of Kubernetes Secrets and ConfigMaps is critical for security, reliability, and compliance. By following these best practices—encrypting secrets, limiting access, managing configs as code, and leveraging tools—you can reduce risks and ensure your applications run smoothly.

Remember: Secrets and ConfigMaps are foundational to Kubernetes applications. Invest time in securing and automating their management to build robust, production-ready systems.

References

• Kubernetes Secrets Documentation
• Kubernetes ConfigMaps Documentation
• HashiCorp Vault Kubernetes Integration
• Sealed Secrets
• Kustomize Documentation
• etcd Encryption at Rest
• ConfigMap Reload