Running Kubernetes at Its Extremes

TL;DR — Key Takeaways

– 5,000 nodes isn’t a brick wall. It’s where the guarantees stop. Push Kubernetes beyond its supported scale and API performance, etcd capacity and networking become serious bottlenecks.

– Small hardware, big headaches. Shrinking Kubernetes to palm-sized devices exposes the CPU, memory and storage demands hidden beneath the control plane.

– Scale isn’t just about size. It’s about trade-offs. Whether running thousands of nodes or a single tiny device, Kubernetes forces tough decisions about throughput, availability and resource efficiency.

Kubernetes has strict thresholds that it supports. The project maintainers set these to ensure SLAs for API responsiveness and availability can be met. But the real world doesn’t care about limits when there is a job to be done. Pushing beyond these limits is where we can learn about real-world requirements and nice-to-have suggestions. At KubeCon + CloudNativeCon North America 2026, I will demo two clusters at opposite ends of cluster scale. The first will be scaled up larger than recommended to see what breaks and how we can work around the limitations. The second will be small enough to fit in the palm of a hand, to show what scaling Kubernetes down requires.

The Large End

Here’s what the project actually supports. The Kubernetes documentation puts the limits at 5,000 nodes, 110 pods per node, 150,000 pods and 300,000 containers per cluster. SIG Scalability uses those numbers as the conditions for its performance targets, like API calls that change a single object finishing within one second at the 99th percentile. Go past them, and those targets are no longer promised, but nothing stops you from trying.

All cluster state is stored in etcd. Every node and pod adds objects and updates, and all of it goes through the API server first. As the cluster grows, etcd has more work to do, and you eventually hit the storage limit, which is 2 GB by default. When that happens, etcd raises an alarm and only accepts reads and deletes until you free up space. Compaction and defragmentation help, but are likely bandages on a much bigger problem.

The API server takes the brunt of the load. Every open watch and every big list adds to it. There’s only so much you can do to scale an API. Read caches, gRPC, and reusing connections per client help, but there are still a lot of requests. The project is constantly optimizing, and since Kubernetes 1.34, nearly all reads can be served from its cache. Scaling isn’t something you’ll ever be done with, and the project keeps pushing boundaries with every release.

One of the problems with scaling the Kubernetes API is the constant background traffic from the nodes and workloads. Each kubelet renews a Lease object every 10 seconds by default, and it sends a full status update at least every five minutes even when nothing has changed. At 5,000 nodes, lease renewals alone come to about 500 updates per second.

Some limits have nothing to do with the control plane. When the controller manager hands out pod ranges, each node gets a /24 IPv4 range by default, so your cluster-wide pod range decides how many nodes you can add. A /16 only holds 256 of those node ranges. Add-ons that run on every node through DaemonSets grow with the cluster too, and their default resource limits are based on small and medium clusters, so expect to raise them.

To keep a big cluster running, optimize throughput. Throughput for an end-to-end request from a kubelet, kubectl, or controller needs to be as low as possible at a scale that fits within budget and your availability requirements. Test lower availability with more throughput. In a distributed system with automatic failover, that is sometimes the correct trade-off. Remember “big” doesn’t only mean number of nodes. There are multiple dimensions for large clusters that you need to consider when scaling.

The Small End

Going small is about how little hardware a control plane can get by on. If you build with kubeadm, the docs ask for at least 2 GB of RAM per machine and two CPUs for control plane machines, and they warn that less memory leaves little room for your apps. If you’ve ever run a Kubernetes control plane node with those specs, you’ll know it’s not practical. A standard control plane runs etcd, the API server, the scheduler and the controller manager, and on small hardware they all share the same memory and CPU before a single workload starts.

With fewer resources, you get fewer pods per node. SIG Scalability puts the per-node threshold at 110 pods or 10 pods per CPU core, whichever is lower. On a two-core device, that’s 20 pods. In practice, that’s one pod if it runs the JVM.

Then there’s storage. etcd writes every change to disk and waits for that write to finish before confirming it, so it’s only as fast as the storage underneath. The etcd docs recommend SSDs and suggest checking that 99th-percentile write-ahead log sync time stays under 10 milliseconds and backend commit time under 25 milliseconds.

Run a cluster on a single device, and etcd has a single leader and a single member. This leaves etcd with no fault tolerance. If the device goes down, your workloads and cluster might be down with it. If you have the luxury of dedicated worker nodes running pods, they keep running, but you can’t schedule new pods or change anything until it’s back.

At Both Ends

Neither cluster is meant for production. But pushing both ends shows which assumptions the defaults make about hardware and scale, and where they stop holding.