Kubernetes Wasn’t Built for GPUs. Make It Behave
TL;DR — Key Takeaways
- Kubernetes traditionally treats GPUs as whole integer resources, so a pod requesting
nvidia.com/gpu: 1can occupy an entire accelerator even when it uses only a fraction of its VRAM or compute. - Simply adding autoscaling can multiply the waste rather than solve it.
- NVIDIA offers three main sharing approaches: time-slicing for low-criticality workloads, MPS for higher throughput among trusted workloads, and MIG for hardware-isolated GPU slices.
- LLM serving pods should be treated as stateful because model weights can take tens of seconds or more to reload into GPU memory.
- Node-local model caching, appropriate readiness delays and sensible
minReplicassettings can reduce cold-start pain. - Kubernetes Dynamic Resource Allocation (DRA) provides a more sophisticated way for the scheduler to understand device characteristics instead of simply counting GPUs.
- The key lesson: Stop automatically assigning an entire expensive GPU to workloads that only need a slice.
The cost dashboard is what tipped me off — a cluster of eight-GPU nodes, every card showing Allocated: 1/1, finance asking why the bill looked like a small data center. So, I pulled nvidia-smi across the fleet. Utilization hovered around 10%. Memory: A 7B model in BF16 sitting in 80 GB of VRAM, using maybe a sixth of it.
Every pod had grabbed a whole card. Not because it needed one — because that’s the only thing it knew how to ask for. The deployment said nvidia.com/gpu: 1, the scheduler found a free integer and an accelerator that bills like a luxury car got pinned to a workload that idles between requests.
This is not a tuning problem. It’s a vocabulary problem. Kubernetes was built to pack fungible, stateless and fast-to-replace units of compute. An LLM serving pod is none of those things, and the default GPU contract pretends otherwise.
The Scheduler Counts Cards, and That’s All it Does
Here’s the mechanism, because the mechanism is the whole story. The NVIDIA device plugin advertises GPUs as an extended resource — nvidia.com/gpu. It advertises them as an integer count — not gigabytes of VRAM, not a fraction of streaming multiprocessors, not ‘this one shares an NVLink domain with that one’ — a count.
Therefore, when your pod requests nvidia.com/gpu: 1, the scheduler does exactly what it was designed to do with extended resources: It finds a node with a free unit and decrements the counter. It has no idea your model needs 14 GB and the card has 80. It has no idea two of your pods could coexist on the same silicon without ever touching each other’s memory. It can’t reason about any of that, because the resource model gave it one number.
That’s why ‘just turn on autoscaling’ doesn’t fix it. The HorizontalPodAutoscaler can spin up more whole-GPU pods. Karpenter can spin up more whole-GPU nodes. You scale the ‘waste’ linearly. The bin you’re packing into is the wrong shape.
Three Ways to Share One Card, and How They Actually Differ
The card was always shareable. The platform just wasn’t asking. NVIDIA’s GPU Operator gives you three mechanisms, and the difference between them is not academic — pick wrong and you either leave throughput on the table or expose tenants to each other.
Time-slicing is a pure software. The driver round-robin context-switches processes onto the GPU in millisecond quanta — they take turns. It works on essentially every NVIDIA GPU: T4, V100, L40S, A100, H100. You configure it with a ConfigMap (replicas: N) and suddenly one card advertises as four. The catch, in capital letters: No memory isolation, no fault isolation, no proportional-compute guarantee. One pod’s OOM can take the card down for everyone. Great for notebooks, dev, bursty low-criticality inference. Not where I’d put a paying customer’s tenant.
Multi-process service (MPS) is a step up in throughput. Instead of taking turns, the CUDA MPS server multiplexes process contexts so kernels run ‘concurrently’ on different SMs. Lower latency, higher utilization. However, there’s ‘still no memory isolation’, fault isolation is weaker (a crashing client can hurt its neighbors) and it’s the least battle-tested of the three in Kubernetes. Reach for it when you control all the tenants and you want the throughput.
Multi-instance GPU (MIG) is the only one that partitions in ‘hardware’. It carves a card into up to seven isolated instances, each with its own dedicated memory, SMs, and fault domain. This is real isolation — a tenant can crash inside its slice and the rest of the card never notices. The price: It’s Ampere-generation and newer only (A100/H100/H200/B-series — your T4 and V100 cannot do it), and the profiles are static. You plan 1g.5gb, 2g.10gb, 3g.20gb in advance, and reconfiguring drains the GPU. It is an isolation tool, not an autoscaling knob. Use it when tenants are untrusted or you need predictable QoS.
With the GPU Operator in mig.strategy=mixed, MIG instances show up as fully-qualified resources — nvidia.com/mig-1g.5gb, nvidia.com/mig-2g.10gb — so a pod asks for a ‘slice’ by name instead of pretending it needs the whole card.
The Pod That Asks for a Slice and Warms its Own State
Here’s a KServe InferenceService that does both reframes at once: It shares the silicon and it treats weight-loading as state hydration instead of pretending a restart is cheap.
apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
name: llama3-8b
annotations:
serving.kserve.io/enable-node-scaling: "true" # allow GPU node scale-to-zero
spec:
predictor:
minReplicas: 0 # scale-to-zero when idle (deliberate: you accept the re-wake cold start)
maxReplicas: 4
scaleTarget: 10 # target concurrent requests per replica (Knative KPA)
model:
modelFormat: { name: huggingface }
runtime: kserve-vllmserver
storageUri: "pvc://model-cache/llama-3-8b" # node-local cache to cut weight-load cold start
resources:
limits:
nvidia.com/gpu: "1" # whole GPU; swap for nvidia.com/mig-1g.5gb to take a MIG slice
memory: 24Gi
requests:
nvidia.com/gpu: "1"
readinessProbe: # weights take 40s+; don’t get marked Ready too early
initialDelaySeconds: 60
periodSeconds: 10To take a MIG slice instead of a whole card, replace both nvidia.com/gpu: “1” lines with nvidia.com/mig-1g.5gb: “1” — the node has to run the GPU Operator with mig.strategy=mixed. This substitution is the difference between pinning an 80 GB H100 to a 7B model and letting six other tenants share the same physical card.
The other half is storageUri: ‘pvc://model-cache/…’ and initialDelaySeconds: 60. Those exist because of the tax I keep coming back to.
Weight-Loading is the Tax Nobody Budgets For
Scale-to-zero looks like free money on a slide. The reality is that a serving pod is ‘stateful’, and the state is multiple gigabytes of weights that have to hydrate into VRAM before the pod can serve a single token.
The numbers are real and they hurt. Typical cold-start penalties run 60–120 seconds; a 140 GB BF16 model loading from local NVMe on an H100 is roughly 40–50 seconds just for the load, plus another 25–30 seconds of CUDA-graph capture. Over a minute to warm a pod from ‘local disk’ — before you’ve even pulled an image over the network. (Treat those figures as point examples, not benchmarks; they swing wildly with model size, storage tier and runtime.)
Therefore, when you set minReplicas: 0, you are signing up for that latency every time traffic resumes. That’s a deliberate trade, and it only pays off when idle gaps are long — roughly an hour or more — relative to re-wake time. For anything with steady traffic, minReplicas: 1 keeps a warm baseline and you eat the cost of one card to dodge the cold-start cliff on every request.
The mitigations are all about treating the load as hydration, not a restart. KServe’s native model caching parks weights on a node-local volume so the next pod skips the download entirely. A readiness probe with a real initialDelaySeconds stops Knative from routing traffic to a pod that’s still loading. You keep your scale-to-zero honest by verifying your KServe/Knative version actually tolerates long model loads — early versions broke InferenceService creation when init took too long.
One more landmine, because it bit me: Karpenter doesn’t understand GPU occupancy. Its consolidation reasons about CPU and memory bin-packing and will happily recycle a node it thinks is underutilized while a GPU on it is mid-inference — there’s an open issue requesting GPU-aware consolidation. Pair WhenEmptyOrUnderutilized with do-not-disrupt annotations on stateful serving pods, or your ‘optimization’ evicts live work.
DRA is the Real Fix, and It’s Arriving in Stages
Everything above is working around a resource model that only knows how to count. Dynamic resource allocation is the actual repair: A native vocabulary for ‘a device with these attributes’ instead of an integer. The scheduler can finally reason about what a GPU is.
Get the status right, because there’s conflated information floating around. ‘Core’ DRA graduated to GA in Kubernetes v1.34, released September 1, 2025 — stable API group resource.k8s.io/v1, on by default (If you’ve seen ‘1.36’, that’s wrong; the primary source says 1.34). Red Hat shipped it to GA in OpenShift 4.21 in early 2026.
But — and this matters before you bet a platform on it — the ‘sharing-oriented’ sub-features are still maturing. Consumable capacity (the flexible device-sharing model) and extended-resource mapping were alpha in 1.34. The framework is GA; the fine-grained GPU-fractioning knobs you’d want for dense LLM serving are not. Therefore, DRA is where you’re going, not necessarily where you ship next quarter. Check the exact feature-gate status against the docs for the specific minor version your cluster runs.
Takeaways
nvidia.com/gpu: 1 is a confession that your scheduler can’t see the GPU. It counts cards. It doesn’t know VRAM, MIG geometry or fault domains. Every workaround starts with admitting that.
Match the sharing mechanism to the trust boundary. Time-slicing for dev and bursty low-criticality work; MPS for trusted multitenant throughput; MIG when tenants are untrusted or QoS must be guaranteed — and only on Ampere or newer.
A serving pod is stateful. The state is gigabytes of weights and a 40 seconds to 2 minutes hydration. Budget for it: Node-local model caches, real readiness delays, minReplicas: 1 unless idle gaps are genuinely long.
Scale-to-zero and Karpenter consolidation are GPU-blind by default. They’ll evict live inference to save a node. Fence stateful serving with do-not-disrupt annotations.
DRA is the destination; plan the staging. Core is GA in v1.34, the sharing knobs are still alpha/beta. Adopt the framework, don’t assume the fine-grained features are ready.
Stop asking Kubernetes for one whole GPU and hoping autoscaling fixes the bill. It won’t — you’ll just scale the waste. Ask for a slice, treat the weights as state worth keeping warm and make the scheduler see the silicon it’s been counting blind. The card was always shareable. You just have to stop pretending it isn’t.
Frequently Asked Questions
Why does Kubernetes waste GPU capacity?
The traditional NVIDIA device plugin exposes GPUs to Kubernetes as integer resources. The scheduler knows whether a GPU is available, but not how much VRAM or compute capacity a workload actually requires.
Why can scale-to-zero be problematic for LLM inference?
Restarting an inference workload may require gigabytes of model weights to be loaded back into VRAM before requests can be served. That can create significant cold-start latency when traffic returns.
What is Kubernetes Dynamic Resource Allocation?
DRA gives Kubernetes a richer model for allocating devices based on their attributes rather than treating accelerators simply as integer counts. The article describes it as the longer-term direction for smarter GPU allocation.
