Why Kubernetes Costs Spiral Out of Control—And How to Prevent It

As even small and mid-sized organizations migrate to Kubernetes, many are surprised to find themselves with whistle-inducing public cloud bills. In the CNCF’s most recent FinOps for Kubernetes report, 68% of respondents report that Kubernetes costs were on the rise—while only 13% had showback measures in place to identify the specific teams and services driving cloud costs.

There is a cruel irony here. Kubernetes is a resource-aware system. It’s extraordinarily well-suited to keeping users up-to-date on utilization—and keeping that utilization within a set of defined parameters.

But just because a technology is good for a particular job doesn’t mean it will be used effectively. First, you need to recognize that there is a problem in the first place—and then you need to know how to use the technology to solve it.

Identify the Problem—Before it’s a Problem

Kubernetes’ cost isn’t so much a singular challenge as a landscape riddled with potholes (and the occasional full-on sinkhole).

One major issue is that a Kubernetes cluster is typically a shared resource. Without showback measures—and complementary chargeback programs to actually assign costs to the appropriate team—it can be unclear who is responsible for what on the cluster. This phenomenon is sometimes described as the “Kubernetes black hole,” and it’s common enough that it has given rise to a term I mentioned earlier: FinOps, or the strategic practice of cloud cost management.

Another challenge: Kubernetes is built to scale. There are plenty of guardrails available to keep things from getting out of hand, but you need to know how to use them—and it’s all too easy for a misconfigured cluster to go to town on your cloud budget.

So how can you get some visibility into the black hole and keep your cluster from scaling beyond your needs and means?

Take Advantage of the Tooling

The Kubernetes ecosystem is full of open source tooling to help you manage costs. The big name is OpenCost, the open source core of the popular Kubecost project. OpenCost runs directly on a Kubernetes cluster and provides cost allocation, showback and chargeback functionality. Better still, it is navigable via a simple web interface. As a CNCF sandbox project, OpenCost is well-supported and positioned as a community standard for cost management.

OpenCost’s big sibling, Kubecost, is also available as a self-managed and as-a-service offering, with a self-managed free tier that provides cost data across unlimited clusters.

Know Your Needs (and Your Limits)

Some Kubernetes spend issues are functions of larger public cloud dynamics. In Flexera’s 2022 State of the Cloud report, 83% of executives say that managing cloud spend was a heavy challenge—and estimated that they wasted 32% of that spend on average.

Kubernetes’ resource awareness and ability to autoscale against demand can reduce cloud waste … but if you’re not careful, it can also be an accelerant. A recent CNCF article cites an incident in which one developer’s configuration error resulted in a $50,000 overspend within 24 hours.

Standing up your cloud infrastructure requires intimate expertise in the following.

  • Appropriate types and quantities of instances to use for nodes
  • Storage for persistent volumes
  • Policies around autoscaling and affinity
  • Kubernetes monitoring stack

It’s a lot! And these are just the fundamentals. As we’ve seen, mistakes can be costly. In the world of open source, there’s always the temptation to DIY, but this is a territory where you really want to ensure that all your technical t’s are crossed and i’s are dotted. You probably wouldn’t try to stand up your own power station, and for many organizations, this is a matter of equal infrastructural importance and complexity.

Assess the skills you have in-house. Do you truly have the resources to set up and maintain this infrastructure in such a way that you can be confident about the costs and reliability moving forward?

If the answer is no, that’s okay! This is a difficult and widespread problem, but it doesn’t have to be a blocker for leveraging Kubernetes. You may simply have to compete to bring those skilled DevOps practitioners on board. Or you may find that it is more efficient to take advantage of DevOps-as-a-service offerings that combine outside cloud expertise and automation to give you the resources you need, as you need them.

Don’t Suffer From a Solved Problem

Runaway Kubernetes costs are a thorny problem, and the fundamental drivers behind them aren’t going away—but they are resolvable, and better still, you can address them with the very powers that Kubernetes brings to bear.

Eric Gregory

Eric Gregory is a senior technical writer at Mirantis who likes to share experiences with Kubernetes, containers, open source, and DevOps. His past experience includes teaching, as well as writing and curating content for DZone.

Eric Gregory has 1 posts and counting. See all posts by Eric Gregory