Kubernetes Key Management Streamlined by HashiCorp Vault Plug-In
IBM HashiCorp has released a plug-in that allows Kubernetes clusters to use HashiCorp’s Vault Enterprise as an external key management service (KMS), eliminating the need to store unencrypted passwords or bog Kubernetes servers down with encryption/decryption tasks.
The technology resulted directly from a Kubernetes update to how it manages encryption keys.
Because containerized workloads are ephemeral and constantly rescheduled across nodes, Kubernetes manages secrets natively—delivering credentials for databases, third-party APIs, private image registries, and other microservices directly to pods at runtime. The etcd database holds all the secrets, encoded in base64 by default.
Kubernetes users could already use external key management services, but because there was no official plug-in, they had to roll their own wrappers or use third-party modules (K8s cloud services did offer their own key services).
Not helping matters was Kubernetes’ original default KMS, which required every single secret to be encrypted or decrypted. That resulted in a lot of work for ephemeral environments.
In addition to saving compute cycles and network bandwidth, this new plug-in offers a clearer separation of responsibilities, which should please the security team. Kubernetes handles the encryption and decryption, using the DEKs. Vault manages the master keys, giving security teams a coherent architecture that is easier to audit.
Kubernetes Key Management Mark 2
The 2023 release of Kubernetes 1.29 replaced this heavyweight model with the lighter KMSv2, which uses Kubernetes-generated Data Encryption Key (DEK) seeds to cut network traffic and further safeguard operations.
With KMSv2, the Kubernetes API server encrypts the data locally in memory using a DEK. It sends this DEK to Vault to be wrapped in a master Key Encryption Key (KEK), which is then returned to the Kubernetes server. Kubernetes places the wrapped DEK alongside the now-encrypted data in etcd, then purges the original, plaintext DEK from its memory.
In the legacy KMS mode, Kubernetes called the external key vault every time the data needed to be decrypted. This could be a painful bottleneck when a cluster needed to be rebooted, as thousands of decryption requests would simultaneously pummel the key server.
With KMSv2, the API server requests Vault to unwrap a single encrypted seed key at startup. It caches that seed in memory to derive individual data encryption keys and handle decryption locally. Vault is then called primarily during cluster boots and periodic key rotations. It no longer has to act as a real-time gatekeeper for every individual decryption request, and the network breathes a sigh of relief.
Vault of Secrets
The new HashiCorp plug-in is built on this sleeker model, and uses Vault Enterprise as a base of operations.
Vault safely stores static secrets and encrypted information such as API/SSH keys and TLS certificates, as well as a set of fine-grained policies for granting access to established users and apps.
In addition to secret storage, Vault provides key management, generating cryptographic keys internally to encrypt and decrypt data on behalf of applications via API.
“Instead of exposing sensitive root cryptographic material to the Kubernetes control plane, Vault becomes the centralized authority responsible for protecting the keys that ultimately secure Kubernetes data,” wrote HashiCorp engineers Rich DuBose and Steve Almy in a blog post announcing the plug-in.
Platform and Architecture Manager Mark Silvester noted some limitations to keep in mind in an InfoQ post. In its beta incarnation anyway, it only works on Vault Enterprise, not the open source Vault. It probably won’t work on most managed control planes, because modifications will need to be made in the Kubernetes EncryptionConfig and the kube-apiserver manifest, which rule out most fully managed control planes.
Lastly, Silvester advised thinking hard about putting the KMS provider directly in the path of decrypting cluster data.
