← All writing

Kubernetes ConfigMaps & Secrets Done Right: Beyond Base64 Traps

· Sourabh G Kulkarni

Why Base64 isn't encryption, volume mounts vs env vars, zero-restart reloads, and External Secrets Operator (ESO) integration with Vault and AWS Secrets Manager.

Kubernetes ConfigMaps & Secrets Done Right: Beyond Base64 Traps

Every Kubernetes engineer learns early that running echo -n "password" | base64 does not encrypt anything. Base64 is an encoding format designed for safe byte transmission over ASCII channels, not a security boundary. If anyone has read access to your cluster API or namespace, your raw secrets are plain text.

Beyond the Base64 trap, configuration management in Kubernetes breaks production systems in two subtle ways: immutable environment variables requiring full pod restarts and uncontrolled secret sprawl across Git repos.

Here is how production engineering teams configure applications cleanly, securely, and without downtime.

flowchart LR
    subgraph External["External Secret Stores (KMS / IAM Guarded)"]
        Vault["HashiCorp Vault"]
        AWS["AWS Secrets Manager"]
    end

    subgraph Cluster["Kubernetes Cluster"]
        ESO["External Secrets Operator (ESO)"]
        Secret["Native k8s Secret"]
        
        subgraph Pod["Application Pod"]
            Vol["Mounted Volume (/etc/config)"]
            App["App Process (Live Reload)"]
        end
    end

    Vault -->|Fetch via IAM/OIDC| ESO
    AWS -->|Fetch via IAM/OIDC| ESO
    ESO -->|Auto-Sync & Reconcile| Secret
    Secret -->|Projected Volume| Vol
    Vol -.->|inotify / polling| App

1. Volume Mounts vs. Environment Variables

Most beginners inject configurations and secrets into containers via env: or envFrom:.

# ❌ The Anti-Pattern: Static Environment Variables
env:
  - name: DB_PASSWORD
    valueFrom:
      secretKeyRef:
        name: db-credentials
        key: password

Why Env Vars Become a Liability in Production

  1. Crash & Crash Dumps: If an application crashes and generates a core dump or stack trace, environment variables are frequently dumped into stdout/stderr logs.
  2. Subprocess Leakage: Child processes inherited from the main runtime automatically inherit the parent’s environment.
  3. No Dynamic Reloading: Environment variables are baked into the container process table at launch (/proc/1/environ). Updating the Secret in Kubernetes will never update the running app without terminating and recreating the pod.

The Production Standard: Projected Volume Mounts

Mount configurations and secrets as read-only files:

# ✅ The Production Standard: Volume Mount
spec:
  containers:
    - name: api-server
      image: registry.sourabhkulkarni.in/api:v1.2.0
      volumeMounts:
        - name: app-config
          mountPath: /etc/app/config
          readOnly: true
  volumes:
    - name: app-config
      configMap:
        name: api-config
        defaultMode: 0440

When you update the ConfigMap in Kubernetes, the kubelet automatically updates the mounted files inside the container via atomic symlink swaps (usually within 30–60 seconds, governed by the kubelet sync loop). Your app can watch the file path using inotify or library watchers (like Go's fsnotify or Spring Cloud Kubernetes) and refresh config without dropping a single active TCP connection.


2. External Secrets Operator (ESO): Keeping Git Repos Clean

Storing Secrets directly in Git (even in private repositories) is an operational risk. The modern cloud-native pattern uses External Secrets Operator (ESO) to synchronize secrets from HashiCorp Vault, AWS Secrets Manager, Azure Key Vault, or GCP Secret Manager directly into native Kubernetes Secrets on demand.

Step 1: The SecretStore Definition

Define the link between your cluster and your cloud KMS:

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: aws-secrets-manager
  namespace: production
spec:
  provider:
    aws:
      service: SecretsManager
      region: us-east-1
      auth:
        jwt:
          serviceAccountRef:
            name: eso-irsa-serviceaccount

Step 2: The ExternalSecret Mapping

Tell Kubernetes which cloud secret to fetch and how frequently to refresh it:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: database-credentials-sync
  namespace: production
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secrets-manager
    kind: SecretStore
  target:
    name: db-credentials # The native k8s secret created automatically
    creationPolicy: Owner
  data:
    - secretKey: password
      remoteRef:
        key: prod/rds/postgres
        property: db_password

3. The Production Checklist

PatternBad PracticeProduction Standard
Storage FormatPlain k8s Secret committed to GitExternal Secrets Operator (ESO) syncing from KMS / Vault
Injection Methodenv / envFromProjected read-only volume (mountPath)
Config RotationManual pod restarts / rolling restartFile-system watch (inotify) or Reloader operator
PermissionsDefault 0644Strict defaultMode: 0400 or 0440 with non-root user

By treating configurations as living files and delegating secret storage to IAM-authenticated cloud vaults, you turn Kubernetes configuration from a brittle restart trigger into a resilient, zero-downtime architecture.