Your ML Platform Isn’t Kubernetes-Native Just Because It Runs on Kubernetes

The real transformation begins when the ML lifecycle becomes declarative, observable, and recoverable.

As Vara Bonthu described in a recent CNCF article, the great migration of AI workloads to Kubernetes is well underway. Training jobs run in containers, inference services run as Deployments, and GPUs are scheduled across clusters.

These developments address difficult infrastructure problems such as scheduling, isolation, scaling, and container recovery, but moving an ML workload onto Kubernetes only changes where it runs, not how its lifecycle is managed. Training, evaluation, registration, deployment, and rollbacks often remain scattered across independent service APIs, databases, and custom scripts. Kubernetes may be running the compute, but custom glue is still running the lifecycle.

That distinction matters because one of Kubernetes’ most transferable ideas is its operating model. Declare the state you want, continuously observe the state you have, and reconcile the difference. For machine learning, applying that model can transform much more than infrastructure.

ML Is Not One Job

Production ML is not a simple checklist of independent steps or a collection of ad hoc runs. The process must be repeatable as datasets change, while preserving the lineage needed to understand each result. Each stage creates artifacts, metadata, and status that affects the next: a model connects to its training run, code version, parameters, and input dataset, while a deployment refers to a specific model version alongside rollout policies, health signals, and traffic splits.

Because these relationships must survive infrastructure failures, process restarts, and organizational change, machine learning operates less like a sequence of independent batch jobs at the platform level and more like a continuously managed state machine. Managing that state requires an operating model built for continuous reconciliation.

Meet the Reconciliation Loop

Reconciliation is simple in concept: declare the desired state, continuously observe reality, and act to reconcile any difference.

Imagine you want version 42 of a model to receive 10% of production traffic. You declare that intent. A controller detects that version 42 currently receives 0%, initiates a canary rollout, evaluates health signals, and updates status until reality matches your goal. If the rollout stalls, explicit status conditions explain whether it is waiting for capacity, failing a policy check, or encountering an unhealthy model state.

The loop works in five simple steps:

1. Declare intent: Model version 42 should receive 10% of traffic and remain healthy.
2. Observe reality: It currently receives 0%.
3. Act: Create or update runtime resources.
4. Report status: Record progress, conditions, and failures.
5. Repeat: Reconcile continuously until reality matches intent or intervention is required.

Across multiple clusters, the control plane can roll out changes gradually and stop if problems appear, limiting how far a failure can spread. Custom resources and controllers allow platform teams to apply this pattern directly to domain-specific concepts like training runs, evaluations, and rollouts.

Turn the Lifecycle Into Explicit State

When lifecycle concepts become explicit Kubernetes resources, their relationships form a clear, observable object graph. A training run produces a model resource that records lineage and evaluation results, while a deployment points to that model and reports real-time health and traffic allocation.

Structuring the platform this way unlocks several benefits for Day 2 operations at scale:

• Recovery by design: Controllers repeatedly compare desired and observed states, treating retries and transient failures as normal operations rather than edge-case errors.
• Shared status interface: Consistent status conditions clearly indicate whether a resource is pending, progressing, ready, degraded, or blocked across all stages.
• Policy at the intent boundary: Admission controls can enforce fast, deterministic requirements such as resource limits and data restrictions, while controllers can coordinate longer-running evaluation and approval gates.
• Stable contracts for automation: Web interfaces, CLIs, controllers, and AI agents build on a consistent set of versioned, typed APIs.
• Contextual observability: Metrics and logs attach directly to the responsible lifecycle object (e.g., “failed during evaluation for model candidate X” rather than “pod failed”).

A Control Plane Should Connect Tools, Not Replace Them

Reimplementing every specialized tool inside a single platform creates an inflexible monolith. Instead, a well-designed control plane divides responsibilities cleanly:

• The control plane defines lifecycle intent, relationships, policy, and status.
• Specialized execution engines perform training, data processing, inference, and other heavy lifting.
• Adapters translate lifecycle intent into the runtime-specific resources those engines understand.
• Results flow back into a consistent lifecycle model.

This approach gives users a unified operating model without pretending every workload is identical, and it lets platform teams swap execution engines without redesigning the user experience.

Not Everything Needs to Be a Custom Resource

Kubernetes APIs should not become a dumping ground for internal details. A good lifecycle resource represents durable intent that users or automation need to observe and manage, such as a model deployment, but not every temporary function call. The goal is not to place every artifact or operational detail in Kubernetes. Large models, logs, traces, and detailed evaluation results belong in purpose-built stores, while lifecycle resources should contain durable intent, references, and compact status.

Before introducing a new resource, platform builders should confirm that it meets these criteria:

• Represents meaningful user intent.
• Has a lifecycle independent of a single process.
• Requires user or automated status tracking.
• Benefits from continuous reconciliation.
• Relies on a stable enough API to serve as a long-term contract.

Lessons From Operating a Large-Scale ML Platform

My experience with Uber’s ML platform Michelangelo AI, whose modular control plane has been released as an open-source project, made this distinction concrete.

The most durable abstraction is not a particular training engine, serving runtime, or workflow implementation. Those technologies evolve, and workloads vary. The truly durable layer is the lifecycle contract connecting them. Once training runs, models, evaluations, and deployments are unified through consistent resources and reconciliation patterns, specialized runtimes can keep doing what they do best.

That lesson is even more relevant as platforms expand into generative and agentic AI. Models are now joined by prompts, tools, policies, benchmark suites, and traces. The artifacts are changing, but the operational questions remain the same:

• What did we ask the system to do?
• Which versions and inputs did it use?
• What happened?
• Did it satisfy our evaluation and policy requirements?
• Can we reproduce, promote, or roll it back?

These are lifecycle questions, and they all benefit from explicit, observable state.

The Next Opportunity for Open Source

Kubernetes has given the industry a common foundation for running AI workloads. The next opportunity is developing consistent patterns for managing what happens around them.

That doesn’t mean defining one universal MLOps API, as workloads vary too widely for a single rigid abstraction. However, the community can collaborate on shared fundamentals:

• Lifecycle relationships and lineage
• Status and condition conventions
• Portable workload definitions across runtimes, clusters, clouds, and accelerator types
• Identity and policy boundaries
• Evaluation and promotion workflows
• Artifact and resource observability for compute, accelerators, and storage

The great migration gets AI workloads onto Kubernetes. The next step is taking full advantage of the operating model that brought everyone there: declaring, observing, reconciling, and recovering the lifecycle itself.

KubeCon + CloudNativeCon is taking place in Salt Lake City from November 9 to 12. Register Now!