Kubernetes Is Strengthening its Position as the AI Platform of Choice

Kubernetes’ next AI push targets GPU efficiency, massive clusters, and safer agents.

SALT LAKE CITY — I’m often surprised how many people don’t know just how important Kubernetes is to AI. After all, in the Cloud Native Computing Foundation (CNCF) January 2026 Annual Cloud Native Survey, 66% of organizations hosting generative AI models use Kubernetes for their inference workloads. Throw in everyone who runs other kinds of AI workloads on Kubernetes, and it’s clear a majority of AI users depend on Kubernetes for their workloads. So, at KubeCon North America 2026, both the CNCF and vendors presented the next wave of Kubernetes improvements, designed to keep AI productive while making the most of their cloud infrastructure with smarter scheduling, larger control planes, faster container startup, and tighter controls over autonomous agents.

Kubernetes 1.38, scheduled for December 16, is the next milestone. So you can expect, as an early Christmas present, gang scheduling and additional accelerator-allocation capabilities advancing toward stable status. But the conference’s message extended beyond one release. Other important improvements are also arriving through Karpenter. This open-source, high-performance Kubernetes node lifecycle manager automatically provisions and terminates compute resources in response to unscheduled pod requests and other emerging CNCF projects.

As Mike Stefaniak, AWS’s Head of Containers, said in his keynote, “AI workloads can spin up 10s of 1000s of GPU instances per cluster, and things break at that scale.” To address this, fixes discovered in enormous production deployments should go upstream rather than remain inside a provider’s private fork.

Making the Control Plane Keep Up

Stefaniak described bottlenecks AWS encountered while pushing beyond Kubernetes’ conventional 5,000-node design envelope toward 100,000 nodes. In a million-Pod benchmark, an API-server watch-cache lock left writers waiting more than half a second. A lazy-snapshot implementation reduced that wait to two milliseconds, he said.

Other changes removed serialization in Horizontal Pod Autoscaler processing and replaced repeated scheduler volume-reference calculations with incremental updates. Stefaniak reported scheduling rates above 300 Pods per second under sustained backlog in AWS’ largest clusters. Those fixes, he said, have already shipped in Kubernetes 1.37.

For 1.38, AWS co-authored a proposal to compress API objects before storing them in etcd, Kubernetes’ backing data store. “It’s accepted for 1.38 alpha, and the first code was actually merged just last week,” Stefaniak said. The immediate target is storage pressure in very large clusters.

Scheduling Whole AI Jobs

Another major shift is treating distributed AI workloads as coordinated groups rather than unrelated Pods. This will be done with Kubernetes’ gang-scheduling enhancement. At the same time, the Workload and PodGroup APIs provide native scheduling primitives for these groups.

That matters because a misfiring training job can stall out on GPUs while not making any progress when only some workers fit. Gang scheduling holds the group until its required minimum membership can be placed. It coordinates admission. It doesn’t promise that every container starts simultaneously, but what matters is it gets work done without wasting time.

Looking ahead, the workload-aware scheduling roadmap also targets beta maturity for topology-aware placement and CompositePodGroups. Together, these address communication-sensitive placement and workloads containing multiple coordinated groups. Such workloads include distributed training and disaggregated inference.

Moving on, Dynamic Resource Allocation (DRA) complements that work by describing accelerator capabilities and capacity more precisely. This will provide general-availability promotions for partitionable devices, consumable capacity, fractional capacity ranges and device binding conditions. These features extend smart allocation well beyond simply counting whole GPUs.

Getting GPUs Online Faster

That’s not to say GPUs are being ignored. Far from it! Stefaniak said Karpenter now models devices on instance types it can provision. This addresses the mismatch between DRA’s device inventories and just-in-time (JIT) node creation. That means, he explained, “GPU capacity can scale from zero and be optimally placed,” he said. This is Karpenter work, not a Kubernetes 1.38 feature.

Finally, container startup is another bottleneck. These days, AI images can reach tens of gigabytes, and parallel downloading can merely move the delay to unpacking layers. My mind boggles at the idea of multi-GB containers, but here we are. To address this, Stefaniak said an upstreamed parallel-unpack implementation, available in containerd 2.2, reduced a 10GB image’s registry-to-node time from two minutes to consistently under 45 seconds in AWS testing.

Beyond Raw Performance

Efficiency increasingly includes agents, not just models. At the conference press briefing, Jonathan Bryce, the CNCF Executive Director, framed the challenge directly: “This is how do we take the same footprint, the same power consumption, the same set of hardware, and get more out of it.”

Bryce highlighted OpenShell’s default-deny approach to agent permissions and Agent Substrate’s handling of idle agents. He cited a demonstration moving from one agent per node to 65 by sleeping and working agents around their work. This is work in progress, but it sounds very promising.

Portability is another unfinished layer. Dilpreet Bindra, VMware Senior Director of Engineering, said, “Our goal is to define and verify how workloads should behave across any environment, reducing the fragmentation and ensuring portability.” That sounds like a win to me.

The direction is clear. AI is forcing Kubernetes to coordinate workloads, hardware, and security together. What ships in December remains subject to the release process; making these pieces work consistently in production will take longer. Still, when it delivers, it will be well worth it, as with Kubernetes, AI jobs will get more production out of the same platforms for less money.