Why IDPs are the Only Way to Scale Kubernetes Beyond Experts
Kubernetes has become the default control plane for modern infrastructure, but scaling it across large engineering organizations has exposed a fundamental mismatch between how the platform was designed and how many companies try to use it.
While Kubernetes excels at orchestrating infrastructure, it was never intended to be the primary interface for most developers to interact with production systems.
Yet in many environments, application teams are still expected to work directly with Kubernetes primitives—managing YAML manifests, networking policies, storage classes and role-based access controls alongside their actual job of writing code.
As adoption expands, that model becomes increasingly difficult to sustain. Platform teams struggle to maintain consistent environments, while developers face growing cognitive load navigating the operational complexity of the underlying infrastructure.
Kubernetes’ Role in Modern Platforms
Ant Newman, director at Spectro Cloud, says this dynamic reflects a broader misunderstanding of Kubernetes’ role within modern platforms.
“Kubernetes was designed as an infrastructure orchestration layer, not a developer tool,” he explains. “Yet many organizations still treat it as both, expecting application developers to wrangle networking policies, RBAC configurations, storage classes, and deployment manifests alongside their actual job of writing application code.”
The result is a familiar cycle in organizations attempting to scale Kubernetes usage. Developers often wait for platform teams to provision clusters or configure environments, slowing delivery.
In response, they attempt to self-serve—taking on operational tasks that fall outside their core expertise. Meanwhile, platform teams become de facto gatekeepers for cluster access and configuration, responsible for enforcing guardrails while responding to a growing backlog of requests.
According to Newman, the friction intensifies as infrastructure footprints expand. Many organizations now operate dozens of clusters across multiple environments, making manual coordination increasingly unsustainable.
“When you’re operating at that density, the ticket-based model—where platform teams are effectively a help desk for cluster access—completely breaks down,” he says. “The cost of human coordination grows exponentially while the infrastructure grows linearly.”
Internal Developer Platforms Emerge
Internal developer platforms (IDPs) are emerging to resolve this tension by introducing abstraction layers between developers and the underlying Kubernetes infrastructure.
By providing curated “golden paths,” service catalogs and standardized APIs, these platforms allow application teams to consume infrastructure capabilities without interacting directly with cluster-level complexity—reducing operational friction while enabling Kubernetes environments to scale beyond a small group of specialists.
Newman explains the most effective IDPs work in two layers: The developer-facing layer handles application intent, and an infrastructure layer enables platform teams to create and manage full-stack Kubernetes clusters that are tailored to different tenant needs.
“A data science team needs a fundamentally different cluster configuration than a team running stateless microservices. An edge deployment has completely different constraints than a cloud one,” he says.
He cautions that this is where the industry’s approach often falls short.
“If your IDP only abstracts the application deployment piece but leaves platform engineers manually wiring together clusters from disparate tools, you’ve just moved the problem,” Newman says.
True abstraction means platform teams can declaratively define and manage the entire stack, from the OS layer through Kubernetes, add-ons, security policies, and application services, and then offer that as a curated, self-service experience to developers.
“That’s what gives developers both guardrails and genuine flexibility,” he says.
Golden Paths and Infrastructure
Newman explains that golden paths are only as good as the infrastructure they sit on top of.
“The industry loves talking about golden paths as the answer to developer experience, but a golden path that leads to an inconsistently configured or poorly managed cluster is just a paved road to production incidents,” he says.
From Newman’s perspective, the real power of golden paths is when they encode the full-stack decisions that platform teams have already validated for interoperability and alignment to company policies. When infrastructure decisions are baked in and consistent, the golden path delivers on its promise.
“Organizations still deal with inconsistencies between dev, staging, and production environments,” he says.
Service catalogs and self-service APIs can reduce that, but only if the underlying cluster environments are themselves standardized.
“This is where a lot of platform engineering efforts stall: teams build beautiful developer portals but still have snowflake clusters underneath,” Newman explains.
Tracking Measurable Outcomes
To demonstrate the impact of IDPs on Kubernetes scalability and sustainability, Newman says the DORA metrics (deployment frequency, lead time, change failure rate, mean time to recovery) are the industry standard.
“But I’d argue the most revealing metric is one people rarely track: the ratio of infrastructure operations work to feature development work per engineering team,” he says. “If your developers are spending 30% of their time on infrastructure toil, no amount of improved deployment frequency compensates for that lost productivity.”
The outcomes he sees organizations achieve when they get this right are significant: Onboarding time for new developers drops from weeks to days because the cluster environment is pre-built, consistent, and self-service.
“Our research showed that 51% of developers are frequently unproductive waiting for clusters to spin up,” Newman says. “Eliminating that wait time alone is transformative.”
