Why One of Two Kubernetes Pods Missed a Secret That Already Existed

The Kubernetes Secret contained a new key. A Deployment ran two pods from the same image. One process had the matching environment variable and the other did not.

I found the mismatch during a post-deploy inspection, before it caused a user-visible problem. The cluster view looked consistent. kubectl get secret showed the key, both pod specifications referenced that Secret, and one Deployment owned both containers. Inspecting each process produced the useful evidence:

for pod in $(kubectl get pods -l app=web -o name)
do
  printf '%s: ' "$pod"
  kubectl exec "$pod" -c app -- printenv \
    | cut -d= -f1 \
    | grep -qx FEATURE_CREDENTIAL \
    && echo present || echo missing
done

One pod printed present. The other printed missing.

One Secret Followed Two Update Paths

The application used Azure Key Vault with the Secrets Store CSI Driver. Each pod mounted a CSI volume that referenced a SecretProviderClass. The provider fetched values from the vault and mirrored selected values into a native Kubernetes Secret. The container consumed that Secret through envFrom.

Figure 1. A synced Secret and a container environment update at different points.

Those paths update at different times. The CSI Driver synchronizes the native Secret only after a pod mounts the provider class. Kubernetes initializes environment variables when it creates the container. A later Secret update does not change the environment of a running process. The container must restart.

At inspection time, the native Secret represented the most recent completed synchronization. Each process still held the data available when its container started. Reading the Secret answered what Kubernetes stored at that moment. It could not answer what each process had read earlier.

Helm 4.2.3 Left an Ordering Window

The chart placed the Deployment and SecretProviderClass in one release. Template filenames did not control their installation order. Helm 4.2.3 sorts manifests by Kubernetes kind. Deployment appears in its built-in install order, while SecretProviderClass is an unknown kind and comes after known kinds.

Rendering the chart with the exact Helm version placed the Deployment before the provider class. The Helm source confirms the same behavior.

The order and the post-deploy state support this explanation: Helm applied the Deployment, Kubernetes began replacing the pods, one replacement mounted the earlier provider class, Helm applied the updated provider class, and the next replacement synchronized the new key. I did not preserve pod timestamps, mount events or CSI logs, so that exact five-step sequence remains an inference.

Figure 2. The verified resource order and observed pod states support the inferred sequence.

A Plain-manifest Deployment Exposed the Same Dependency

A second application used kubectl apply. Its workflow applied the provider class after the rollout and health check. That temporary order came from an earlier identity migration and remained after the migration ended.

When another key was added, the native Secret eventually contained it while all three running pods lacked the environment variable. Restarting the workloads recreated the containers, and all three then received the key.

I moved the provider-class apply before the rollout in that workflow. Existing pods kept their current environments, while replacement pods mounted the current provider definition. The final per-pod check remained necessary because synchronization still begins with a pod mount.

kubectl apply --dry-run=server -f deploy/secret-provider.yaml
kubectl apply -f deploy/secret-provider.yaml

The Deployment Gate Verifies Every Process

The Helm release kept both resources, so its ordering window remained. I added a post-rollout gate that reads expected key names from the SecretProviderClass, inspects the environment-variable names in every running non-Job pod, and never prints secret values.

When a pod misses a key, the gate restarts the Deployment once and checks again. A second mismatch fails the deployment.

Figure 3. The generalized gate checks each process and fails after a second mismatch.

The shipped check still has one weakness. If kubectl exec cannot read a pod, it logs a warning and skips that pod. An unreadable environment therefore passes. The reusable version should select the workload explicitly and treat inspection failure as a failed check:

if ! have=$(kubectl exec "$pod" -c app -- printenv \
  | cut -d= -f1 | sort -u)
then
  echo "Could not inspect $pod" >&2
  failed=1
  continue
fi

The deployment identity needs namespace-scoped pods/exec permission for this check. Teams that cannot grant it can validate required variable names during application startup and keep an incomplete pod out of service.

A Successful Health Check Did Not Inspect This Configuration

The original health endpoint did not exercise the optional integration that used the new key. It returned 200 while one process lacked the corresponding variable, so Kubernetes had no reason to replace that pod.

The provider class existed, the native Secret contained the key, and the health endpoint returned 200. Each signal described part of the system. The per-pod check answered the deployment question that mattered: What configuration did every running process receive?