AI Didn’t Break Kubernetes Optimization. GPUs Did.
The industry is buzzing about how AI agents have driven up Kubernetes costs. But most agents run like any other service, and the usual tools like right-sizing, bin packing and autoscaling still work on them. For teams hosting their own model inference, the real change is more specific and far more expensive: GPUs.
By default, Kubernetes allocates GPUs whole. You ask for one, you get all of it, even if your workload only needs a fraction. A lot of inference work is exactly like that, holding an entire GPU while using a small fraction of its compute. MIG can partition supported GPUs into smaller instances, while Dynamic Resource Allocation offers more flexible ways to manage those resources. But adoption is still uneven, and teams routinely provision for a peak that never arrives. GPUs are the most expensive hardware in the stack, and much of the time they’re barely used. Traditional monitoring may not flag any of this as a problem. The pods are healthy, the dashboards look good and the cost lands in the same node group as everything else.
The Sharing Tax
The obvious move is to let jobs share a GPU. Time-slicing, configured through the NVIDIA device plugin or GPU Operator, lets them take turns on one GPU, each running as if it owned it. Serving engines like vLLM improve compute utilization through continuous batching, though they address a different problem and come with trade-offs of their own.
Sharing has two costs people underestimate. The first is compute overhead: context switches between jobs eat into actual work. The second is more serious. Time-slicing gives you no memory isolation, so every job sharing the GPU competes for the same memory pool. One job that spikes can exhaust memory needed by other workloads, triggering out-of-memory errors and potentially disrupting multiple jobs sharing the GPU. In practice, that can show up as an out-of-memory error in a job that did nothing wrong. The batch job that spiked gets the blame, the inference server that failed alongside it gets restarted and nothing in the pod’s own metrics fully explains why.
When Both Signals Lie
Engines like vLLM grab a big chunk of GPU memory at startup no matter how much traffic they’re seeing, so a server doing nothing and a server at full load can look the same on memory. Compute utilization can also be misleading, since continuous batching makes it difficult to distinguish useful work from inefficiencies using node-level metrics alone. A cluster can be badly overprovisioned without its conventional metrics making that obvious, since some of the numbers that actually reflect load aren’t on the node. They’re inside the serving engine, and many autoscalers aren’t wired to them. One of the most useful numbers to watch is queue depth: how many requests are waiting to be served. It sits in the serving engine’s metrics, not on the node, and often tracks demand more directly than allocated memory does.
What to Do About It
Start by not trusting node-level memory or compute as your read on how hard a GPU is working. Pull queue depth and other serving metrics from inside the serving engine and use them to guide autoscaling. Keep spiky, memory-hungry jobs off shared GPUs, since time-slicing doesn’t protect them from each other. For workloads that genuinely need to share a card, hardware partitioning (MIG on supported NVIDIA hardware) gives you the memory isolation that time-slicing doesn’t. It reduces flexibility, since partition sizes are predefined, but it turns a noisy neighbor risk into a capacity planning decision. And chase the easy money first: the workloads sitting on a whole GPU while barely using its compute. Treat GPUs like any other expensive shared resource and watch them closely.
All the attention in AI infrastructure has gone to the models and almost none to how the hardware running them gets watched. On a growing number of Kubernetes bills, the GPU is both the largest line item and the least instrumented. The waste shows up once you look in the right places, which for GPU inference often means inside the serving engine, not just on the node.
