Stop Treating GPUs Like Web Pods

The first time I pulled a nvidia-smi loop across an inference fleet, I had to read it twice. Card after card, sitting at 12%, 18%, maybe 30% utilization. Not idle — ‘busy’, in the sense that every one of them had a serving pod pinned to it. Each pod had asked for a whole accelerator, gotten one and then spent most of its life waiting for the next request to tokenize. The cluster looked fully scheduled. The dashboards were green and the GPU line on the cloud bill kept climbing, because we were paying for silicon by the card and using it by the sliver.

That gap — between ‘scheduled’ and ‘utilized’ — is where most cloud-native AI budgets quietly leak. It isn’t a tuning problem; it’s a category error. We took the patterns that made Kubernetes great for stateless web services and pointed them at the most expensive, least fungible compute in the building.

Kubernetes Was Built for the Wrong Unit of Compute

The whole Kubernetes contract assumes your workload is fungible: Small, stateless, cheap to kill, fast to replace. A GPU inference pod is none of those things. The accelerator is expensive and indivisible by default. The pod carries multi-gigabyte state — model weights — that takes tens of seconds to hydrate into VRAM before it can serve a single token.

Look at how you ask for one. The device-plugin model advertises GPUs as the extended resource nvidia.com/gpu, an integer count. That’s the entire vocabulary. It does not model memory, compute fraction or device attributes. So, nvidia.com/gpu: 1 means ‘give this pod one whole accelerator, exclusively, as an opaque count’, and the scheduler is blind to everything that actually matters: How much VRAM the model needs? Whether a smaller slice would do? Which fault domain the card is in?

That single line is the most expensive default on most AI platforms. It pins a card that rents for a few dollars an hour to a pod that touches a fraction of it, and the scheduler can’t tell the difference because, as far as it knows, the GPU is a number.

Share the Silicon

The fix starts by refusing to treat the GPU as a unit of one. There are three established ways to share a physical card, and they are genuinely different tools, not tiers of the same thing.

‘Time-slicing’ is pure software. The driver round-robins process contexts onto the GPU in millisecond quanta. It works on essentially every NVIDIA GPU — T4, V100, L40S, the lot — and you turn it on with a ConfigMap. The catch is real: No memory isolation, no fault isolation, no compute guarantee. One greedy tenant can starve the others. It’s right for dev, notebooks and bursty low-criticality inference, and wrong for anything you’d page someone over.

Multi-process service (MPS) multiplexes process contexts so kernels run concurrently on different SMs instead of taking turns. Higher throughput, lower latency than time-slicing — but still no memory isolation, and weaker fault isolation, so a crashing client can take neighbors with it.

Multi-instance GPU (MIG) is hardware partitioning: Up to seven isolated instances per card, each with its own memory, compute and fault domain. It requires Ampere or newer — A100, H100 and friends; a T4 or V100 cannot do it. Profiles are static and planned in advance (1g.5gb, 2g.10gb, 3g.20gb), so it’s a capacity-planning decision, not an autoscaling knob. Use it when tenants are untrusted or you need predictable QoS.

The point isn’t which one wins. It’s that once the node exposes slices, the scheduler finally has something better than a count to reason about:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: chat-small
spec:
  replicas: 4
  template:
    spec:
      containers:
        - name: server
          image: my-registry/inference:latest
          resources:
            limits:
              # one 5GB MIG slice, not a whole card
              nvidia.com/mig-1g.5gb: "1"
              memory: 12Gi
            requests:
              nvidia.com/mig-1g.5gb: "1"

Four replicas, four slices, one physical card. The nvidia.com/mig-1g.5gb name comes from running the GPU Operator in mixed MIG strategy; swap the resource name for nvidia.com/gpu: ‘1’ and you’re back to one pod owning everything.

The longer-term answer is dynamic resource allocation (DRA). Core DRA graduated to GA in Kubernetes v1.34, released in September 2025 — the first time the scheduler has a native way to ask for “a device with these attributes” instead of an integer. Be precise about what’s GA, though: The framework is stable, but the fine-grained sharing knobs you’d actually want for fractional GPU serving — consumable capacity, extended-resource mapping — were still alpha or beta in that release. DRA is the right direction. It is not yet a finished replacement for the device plugin.

Scale on a Signal That’s on the Critical Path

Sharing the card fixes packing. It does nothing for ‘when’ you add replicas — and here cloud-native muscle memory fails again. The default HPA scales on CPU, and in a GPU inference pod CPU is a decorrelated proxy. The accelerator does the heavy lifting; the CPU is doing tokenization, HTTP and batch assembly. So CPU can sit at 30% while the GPU is pinned at 100%, the KV cache is full and requests are backing up in the engine’s queue. HPA reads ‘healthy’, does nothing, and users wait.

Scale on something causal instead. Queue depth is a leading indicator — backlog rises before users are starved. GPU utilization, pulled from the DCGM exporter into Prometheus, is a good coincident guardrail. KEDA is the standard mechanism because those signals live outside the pod’s cgroup where vanilla HPA can’t reach, and because it can scale to zero, which raw HPA cannot. Use backlog to decide how many replicas; use GPU percent as the ceiling. Treat any ‘5%’ GPU thresholds you copy from tutorials as demo values — real serving wants 70–80%.

The Cold-Start Tax Nobody Budgets For

Now the honest part, because the two fixes above have a sharp edge. The moment you embrace sharing and scale-to-zero, you sign up for cold starts — and an LLM cold start is not a container restart. Multi-gigabyte weights have to load into VRAM. Loading a large model from local NVMe can run tens of seconds before you add CUDA graph capture on top. Scale a serving pod from zero and the first user feels all of it.

The instinct is to reach for lazy-pull image snapshotters. For most ML serving, that’s the wrong reach. Lazy loading wins when a container starts but only touches a small fraction of its bytes. Inference is the opposite: It reads nearly the whole image — CUDA libraries and weights — within seconds of boot, so on-demand fetching just relocates the stall from pull to first inference. AWS effectively conceded the point by shipping a ‘parallel’ pull mode for EKS aimed at AI/ML: A faster full download, not lazy loading.

The structural fix is to stop baking weights into the image at all. Keep a lean runtime image, stream the weights from object storage or a node-local cache at startup and the multi-gigabyte blob gets fetched once per node instead of re-pulled with every scale-out. Pair that with a warm baseline of minReplicas: 1 for latency-sensitive paths, and reserve scale-to-zero for workloads whose idle gaps are long enough — roughly an hour or more — to outweigh the re-wake penalty.

None of this is free. Sharing trades isolation for density; MIG profiles are static and disruptive to reconfigure; KEDA adds a polling loop and a cooldown window; scale-to-zero adds latency exactly when traffic returns. The win is real, but it’s an engineering trade, not a switch you flip.

The Takeaway

Stop scheduling your most expensive compute with the contract you wrote for stateless web pods. A GPU is not an integer, weight-loading is not a cold restart and CPU percent is not a saturation signal. Divide the card, scale on backlog, keep the weights out of the image — and the same fleet that idled at 20% starts paying for itself. The accelerators were never the waste. The defaults were.