Configuration is where many Kubernetes tutorials look easy and production starts to feel less tidy.

The container image should contain the application and its runtime dependencies. It should not contain the database password for production, the URL of the staging payment provider, or a feature flag that changes every week. Kubernetes gives us ConfigMaps and Secrets to move those values out of the image and into cluster objects.

That separation is useful, but it is not magic. A ConfigMap can still be wrong. A Secret can still leak. An environment variable can still be read by the wrong process. The point is not to put every value into Kubernetes and relax. The point is to make configuration explicit, inspectable, and replaceable without rebuilding the application image.

The basic separation

I use this simple rule at the beginning:

Use a ConfigMap for non-sensitive configuration: log level, public base URLs, feature switches, queue names, timeout values, or application settings that would not hurt if seen by someone with normal read access to the namespace.

Use a Secret for sensitive values: passwords, tokens, private keys, API credentials, database connection strings with credentials, signing keys, and anything that would require rotation if it appeared in a screenshot.

That rule is not perfect. Some values are sensitive because of context. An internal hostname might reveal architecture. A feature flag might expose a future launch. But the rule prevents the worst beginner habit: putting passwords into ConfigMaps because the YAML is convenient.

Here is a small ConfigMap:

apiVersion: v1
kind: ConfigMap
metadata:
  name: checkout-config
data:
  LOG_LEVEL: info
  PAYMENT_PROVIDER_URL: "https://payments.example.internal"
  FEATURE_NEW_TAX_RULES: "false"

And a Secret:

apiVersion: v1
kind: Secret
metadata:
  name: checkout-secret
type: Opaque
stringData:
  DATABASE_PASSWORD: "replace-me-in-real-life"
  PAYMENT_API_TOKEN: "replace-me-too"

stringData lets you write plain strings in the manifest and have Kubernetes store them in the Secret’s data field as base64. It is convenient for examples and some workflows, but be careful: a committed manifest containing stringData still contains the secret in plain text in Git.

Base64 is not encryption

This is important enough to say plainly. Kubernetes Secret values are base64-encoded by default. Base64 is not encryption. It is a transport encoding.

If you run:

kubectl get secret checkout-secret -o yaml

You may see something like:

data:
  DATABASE_PASSWORD: cmVwbGFjZS1tZS1pbi1yZWFsLWxpZmU=

Anyone who can read that Secret can decode it:

echo 'cmVwbGFjZS1tZS1pbi1yZWFsLWxpZmU=' | base64 --decode

Secrets are still useful because Kubernetes treats them as a separate object type, RBAC can restrict access to them, kubelet handles them differently than ConfigMaps in some paths, and clusters can encrypt Secret data at rest in etcd. But none of that helps if every developer has broad get secrets permission, if Secrets are printed in logs, or if raw Secret manifests are committed to a public repository.

The right mental model is: a Kubernetes Secret is a sensitive configuration object, not a password manager by itself.

Environment variables: simple and sharp

The simplest way to pass configuration to a container is with environment variables:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: checkout
spec:
  replicas: 2
  selector:
    matchLabels:
      app: checkout
  template:
    metadata:
      labels:
        app: checkout
    spec:
      containers:
        - name: checkout
          image: ghcr.io/example/checkout:1.0.0
          env:
            - name: LOG_LEVEL
              valueFrom:
                configMapKeyRef:
                  name: checkout-config
                  key: LOG_LEVEL
            - name: DATABASE_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: checkout-secret
                  key: DATABASE_PASSWORD

This is easy to understand. The application reads LOG_LEVEL and DATABASE_PASSWORD from its process environment.

You can also import all keys:

envFrom:
  - configMapRef:
      name: checkout-config
  - secretRef:
      name: checkout-secret

I prefer explicit env entries for important applications. envFrom is useful, but it makes the environment less visible in the Deployment. It can also create accidental name collisions. If a ConfigMap and Secret both define the same key, you need to understand which value wins based on order and validation behavior. Explicit wiring is boring, and boring is often good for production configuration.

There is one crucial operational detail: environment variables are captured when the container starts. If you update the ConfigMap or Secret, already-running containers do not automatically get new environment variable values. You usually need to restart Pods, often by rolling the Deployment:

kubectl rollout restart deployment/checkout
kubectl rollout status deployment/checkout

This is not a Kubernetes bug. A process environment is created at process start. Kubernetes cannot reach into a running process and change how the application already read its config.

Mounted files: better for structured configuration

ConfigMaps and Secrets can also be mounted as files. This is often better for larger config, certificates, config files, or applications that already expect files.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: checkout
spec:
  replicas: 2
  selector:
    matchLabels:
      app: checkout
  template:
    metadata:
      labels:
        app: checkout
    spec:
      containers:
        - name: checkout
          image: ghcr.io/example/checkout:1.0.0
          volumeMounts:
            - name: app-config
              mountPath: /etc/checkout
              readOnly: true
      volumes:
        - name: app-config
          configMap:
            name: checkout-config-files

With this ConfigMap:

apiVersion: v1
kind: ConfigMap
metadata:
  name: checkout-config-files
data:
  application.yaml: |
    logLevel: info
    paymentProviderUrl: https://payments.example.internal
    features:
      newTaxRules: false

Inside the container, the file appears at:

/etc/checkout/application.yaml

Mounted ConfigMaps and Secrets can be updated on disk after the object changes, although not instantly. kubelet periodically refreshes the mounted data. The application still needs to reread the file or watch for changes. Many applications only read config at startup, so a rollout restart may still be necessary.

There is also a common trap with subPath. If you mount a single key using subPath, updates to the ConfigMap or Secret are not reflected in the mounted file the same way. This surprises people. If live updates matter, avoid subPath for this pattern and test the behavior in the cluster you use.

Creating and inspecting objects with kubectl

For learning, kubectl create configmap is convenient:

kubectl create configmap checkout-config \
  --from-literal=LOG_LEVEL=info \
  --from-literal=FEATURE_NEW_TAX_RULES=false

From a file:

kubectl create configmap checkout-config-files \
  --from-file=application.yaml

For Secrets:

kubectl create secret generic checkout-secret \
  --from-literal=DATABASE_PASSWORD='not-for-shell-history-in-real-life' \
  --from-literal=PAYMENT_API_TOKEN='also-rotate-this'

Those commands are useful in a lab. In real environments, be careful with shell history, terminal recordings, CI logs, and copy-paste trails. A Secret created with --from-literal can still leak before it reaches Kubernetes.

To inspect:

kubectl get configmap checkout-config -o yaml
kubectl describe configmap checkout-config
kubectl get secret checkout-secret
kubectl describe secret checkout-secret

describe secret shows keys and metadata, not decoded values. That is deliberate. get secret -o yaml shows base64-encoded data if your RBAC allows it.

To check what a running Pod received:

kubectl exec -it <pod-name> -- printenv | sort
kubectl exec -it <pod-name> -- ls -la /etc/checkout
kubectl exec -it <pod-name> -- cat /etc/checkout/application.yaml

Do not run printenv casually on production Pods if it might print secrets into shared logs or screenshots. It is a debugging tool, not a habit.

Optional references and failed starts

By default, if a referenced ConfigMap or Secret does not exist, the Pod will not start. That is usually good. A missing database password should be obvious.

You may see optional: true:

env:
  - name: OPTIONAL_BANNER_TEXT
    valueFrom:
      configMapKeyRef:
        name: checkout-config
        key: BANNER_TEXT
        optional: true

Use this carefully. Optional config can be reasonable for a non-critical feature. It is dangerous for values the application truly needs. Beginner-friendly systems fail loudly when required configuration is missing. Quiet defaults can turn a deployment problem into a business problem.

Rollouts and configuration changes

A common surprise: changing a ConfigMap does not automatically create a new Deployment rollout. The Pod template did not change, so the Deployment controller has no reason to replace Pods.

Some teams solve this by adding a checksum annotation to the Pod template, often generated by Helm or another deployment tool. When the ConfigMap or Secret changes, the checksum changes, the Pod template changes, and Kubernetes rolls the Deployment.

The plain idea looks like this:

spec:
  template:
    metadata:
      annotations:
        checksum/config: "generated-hash-of-config"

Do not hand-maintain that hash. Let tooling generate it if your deployment workflow needs this pattern.

For manual learning, a rollout restart is enough:

kubectl rollout restart deployment/checkout
kubectl rollout status deployment/checkout

Then verify:

kubectl get pods -l app=checkout
kubectl describe pod <new-pod-name>

Security habits that matter

Secrets deserve boring discipline.

Do not commit raw Secret manifests with real values. Use a secret management workflow: external secret operators, sealed secrets, SOPS, cloud secret managers, Vault, or whatever your organization supports. The exact tool matters less than having a repeatable path that avoids plain secrets in Git.

Limit RBAC. A user who can get secrets in a namespace can usually read the values. A user who can create Pods in a namespace may be able to mount Secrets into a Pod and read them indirectly. RBAC design needs to consider both direct and indirect access.

Enable encryption at rest for Secrets in the cluster’s backing store if you operate the control plane. Managed Kubernetes platforms often provide a setting or integration for this, but do not assume it is enabled exactly the way your policy expects.

Avoid logging secrets. This includes startup logs that print full configuration, exception traces that include connection strings, debug endpoints that dump environment variables, and support scripts that collect too much.

Rotate secrets. Rotation is not only for incidents. If a credential cannot be rotated without panic, the system is not operationally healthy yet.

What belongs where

Here is the mental sorting I use:

Application binary and dependencies belong in the image.

Environment-specific non-sensitive values belong in ConfigMaps.

Sensitive values belong in Secrets or, more often in mature setups, in an external secret system that syncs into Kubernetes.

Large structured files can be mounted from ConfigMaps or Secrets.

Values that change application behavior should be visible in the deployment process. A feature flag hidden in an unreviewed ConfigMap change can be just as risky as a code deploy.

Configuration is part of the release. Treat it with the same seriousness as code: review it, test it, roll it out, and know how to roll it back.

A small checklist

When a Pod does not have the config you expect, check in this order:

  1. Does the ConfigMap or Secret exist in the same namespace as the Pod?
  2. Does the key name match exactly, including case?
  3. Is the Deployment referencing the right object name?
  4. Is the value passed through env, envFrom, or a mounted volume?
  5. Was the Pod restarted after an environment-variable change?
  6. Is the application reading the value you think it reads?
  7. Are RBAC, admission policies, or external secret controllers changing the behavior?

Most configuration incidents are not dramatic. They are small mismatches that survive because nobody checked the actual Pod. Kubernetes gives you the tools to inspect the contract. Use them before guessing.