Buoyant Unveils Open Source Control Plane for AI Workloads Running on Kubernetes
TL;DR — Key Takeaways
- Buoyant Brings AI Inference Under Control: Open-source Kinference adds a Kubernetes control plane for managing distributed vLLM workloads, giving IT teams greater control over performance, costs and compliance.
- Stop Wasting Expensive GPUs: Kinference uses GPU cache data, queueing information and intelligent workload management to improve resource utilization and reduce AI infrastructure costs.
- Self-Hosted AI Gets a Hybrid Escape Route: Organizations can run sensitive AI workloads on infrastructure they control while selectively routing inference requests to cloud providers when local GPU capacity is stretched.
Buoyant today made available an open-source control plane for managing artificial intelligence (AI) workloads running on top of a virtual large language model (vLLM) inference engine running on a Kubernetes cluster.
Originally developed by researchers from the Sky Computing Lab at the University of California, Berkeley, vLLM includes both an inference server to manage network traffic and an inference engine to maximize memory throughput using a PagedAttention algorithm.
Buoyant CEO William Morgan said Kinference adds a control plane that makes it simpler to manage AI workloads running in a distributed Kubernetes environment, which enables IT teams to scale out those workloads more cost-effectively. That’s critical because more organizations are looking to deploy AI workloads in self-hosted IT environments that they directly control, he added.
That shift reduces costs by eliminating dependencies on cloud services that charge to access models based on tokens in favor of an alternative that allocates a specific amount of IT infrastructure resources to specific workloads. Additionally, IT teams gain more control over the quality of service (QoS) that can be provided to each of those workloads, added Morgan.
In general, many IT teams are looking to host AI models in self-hosted IT environments because of security concerns and compliance requirements. Much of the data exposed to AI models is highly sensitive, so many organizations are reluctant to expose it to a model managed by a third party.
IT teams are also concerned about latency issues because the use cases involving AI workloads also tend to be highly iterative, noted Morgan. Kinference provides a control plane through which IT teams can define boundaries around performance, cost, and compliance, he added.
Those boundaries also make it possible to improve the utilization rates of the graphical processing units (GPUs) that have been embedded in a Kubernetes cluster to run AI workloads, said Morgan. While there is a general shortage of GPUs, one of the great paradoxes of AI is that the utilization rate of GPUs is typically in the single digits. The Kinference control plane creates an opportunity to maximize GPU utilization across multiple AI workloads running on the cluster by leveraging cache data on the GPU, queueing status, and other controls, noted Morgan.
Kinference can optionally send inference requests to cloud providers under conditions defined by an IT team, including GPU pool saturation, usage by specific tenants, or characteristics of the inference request itself.
Finally, Kinference can track and attribute GPU usage per tenant, group, or team to help IT teams optimize infrastructure costs.
It’s not clear how many AI workloads are running on Kubernetes clusters, but as their number increases, IT teams are now being tasked with running a new class of applications alongside existing workloads. There may even come a day soon when the number of AI workloads running on those machines far exceeds other classes of workloads. The challenge, as always, is finding a way to optimally manage all those workloads across what remains a limited amount of IT infrastructure.
Frequently Asked Questions
What is Buoyant Kinference, and how does it work with Kubernetes?
Kinference is an open-source control plane that manages AI inference workloads running on vLLM in distributed Kubernetes environments. It helps IT teams optimize resource allocation, manage performance and enforce operational policies.
How does Kinference improve GPU utilization and reduce AI costs?
Kinference uses GPU cache information, queue status and workload controls to distribute inference requests more efficiently. It also attributes GPU consumption to individual teams or tenants, helping organizations identify inefficiencies and control spending.
Can Kinference run AI workloads across on-premises and cloud infrastructure?
Yes. Kinference primarily supports self-hosted inference but can route selected requests to cloud providers based on conditions such as GPU saturation, tenant requirements or request characteristics.
