The main open source Kubernetes solutions for enterprise AI are Kubeflow for the end-to-end ML platform, KServe and vLLM-based projects such as llm-d for model serving, Ray (with the KubeRay operator) for distributed compute, Kueue and Volcano for GPU-aware scheduling, KitOps for packaging models as OCI artifacts, and Kyverno or OPA for policy. No single project covers everything, so enterprises assemble a stack. This guide maps each layer, what the leading projects do, and how they fit together.
What does an enterprise AI stack on Kubernetes need?
Enterprise teams usually need six layers.
- Development and pipelines. Notebooks, training jobs and repeatable workflows.
- Scheduling. Fair sharing of expensive GPUs across teams with quotas and queues.
- Distributed compute. Scaling training, fine-tuning and batch inference across nodes.
- Serving. Production inference endpoints with autoscaling.
- Packaging and versioning. A consistent, portable unit for models and everything they depend on.
- Policy and security. Rules about what can run, enforced by the cluster.
The CNCF maturity levels (Sandbox, Incubating, Graduated) are a useful signal of governance and adoption, so they are noted below where they apply.
Which open source projects cover the ML platform layer?
Kubeflow
Kubeflow is the most complete open source AI platform for Kubernetes, covering notebooks, pipelines, training operators, hyperparameter tuning and a model registry. It joined the CNCF in 2023 and graduated in July 2026. It is modular, so many teams adopt only the pieces they need.
MLflow
MLflow is a widely used open source experiment tracker and model registry. It is not Kubernetes-specific, but it commonly runs on Kubernetes alongside the other projects here.
Which projects handle GPU scheduling and distributed compute?
Kueue
Kueue is a Kubernetes-native job queueing system from the Kubernetes community. It manages quotas and decides when jobs should start, which matters when many teams share a limited GPU pool.
Volcano
Volcano is a CNCF Incubating batch scheduler for AI, HPC and big data workloads, with features like gang scheduling for distributed training.
Ray and KubeRay
Ray is a distributed compute framework for Python and AI workloads, now hosted by the PyTorch Foundation. KubeRay is the operator that runs Ray clusters on Kubernetes.
Which open source tools serve models on Kubernetes?
KServe
KServe is a CNCF Incubating project that provides a standard way to deploy predictive and generative models on Kubernetes, with pluggable runtimes and autoscaling. It also underpins commercial platforms such as Red Hat OpenShift AI's single-model serving.
vLLM and llm-d
vLLM is a popular high-throughput inference engine for LLMs, often used as a runtime inside KServe or directly. llm-d, a CNCF Sandbox project, builds Kubernetes-native distributed LLM inference on top of vLLM.
How should you package and version models on Kubernetes?
This is the layer most stacks skip, and it causes problems later. Models end up in object storage, code in Git, datasets somewhere else and configuration in a wiki. Nothing ties a production model to the exact data and code that produced it.
KitOps
KitOps is an open source CNCF project that packages a model, its datasets, code, prompts and configuration into a single versioned artifact called a ModelKit. ModelKits are OCI artifacts, so they live in the registries you already run (Harbor, Artifactory, cloud registries) and move through the CI/CD tools you already use.
What this gives an enterprise team:
- One unit to version, review and roll back. The model and its dependencies are tagged together.
- Selective unpacking. Pull only the model, only the dataset or only the code, which keeps large artifacts practical.
- Works with your stack. ModelKits can feed serving with KServe, and the PyKitOps SDK lets data scientists push and pull from Python.
- Standard supply chain tooling. Because ModelKits are OCI artifacts, they can be signed and verified with tools like Sigstore cosign. Signing is a separate step after you pack and push.
Which open source tools enforce policy for AI on Kubernetes?
Kyverno
Kyverno is a Kubernetes-native policy engine that graduated in the CNCF in March 2026. It can block workloads that do not meet rules and verify Sigstore signatures on images and OCI artifacts.
Open Policy Agent and Gatekeeper
OPA is a CNCF Graduated general-purpose policy engine, and Gatekeeper brings it to Kubernetes admission control.
How do these projects fit together?
A typical open source enterprise AI stack looks like this.
- Data scientists experiment in Kubeflow notebooks and track runs in MLflow.
- Training and fine-tuning jobs run through Kubeflow Trainer or Ray, queued by Kueue or Volcano.
- The resulting model, data, code and config are packed into a ModelKit and pushed to an OCI registry.
- CI signs approved versions and checks them against policy.
- Kyverno or Gatekeeper admits only approved artifacts.
- KServe, or vLLM with llm-d for large LLMs, serves the model.
How should an enterprise choose between these projects?
Start from the problem, not the project list.
- GPU contention between teams? Add Kueue or Volcano before buying more hardware.
- Mostly LLM inference? Look at vLLM-based serving, with llm-d when a single replica is not enough.
- Mixed classic ML and LLM models? KServe gives one serving interface for both.
- Large-scale training or batch jobs in Python? Ray on KubeRay is a common fit.
- Audits, rollbacks or "which model is in production?" questions? Fix packaging first with KitOps, so every later layer handles a versioned, traceable artifact.
- Security review blocking production? Put admission policy in place with Kyverno or Gatekeeper, and plan for AI-specific scanning and runtime controls that general policy engines do not provide.
Adopt incrementally. Most of these projects are independent, so you can add one layer at a time without a platform rewrite.
What does open source leave to you?
Open source covers the building blocks well. Enterprises still need to operate them, integrate identity and logging, and answer security questions that general tools were not designed for, such as scanning model files for malicious payloads or governing what LLM agents do at runtime. Some teams build that themselves. Others add a commercial layer. Jozu, the company that created KitOps, offers Jozu Hub for scanning and approving ModelKits and Agent Guard for runtime policy on agents, and other vendors offer their own approaches.
FAQ
What is the best open source Kubernetes platform for AI? Kubeflow is the most complete open source AI platform for Kubernetes and is a CNCF Graduated project. Most enterprises combine it with KServe for serving and other projects for scheduling and packaging.
Is KServe part of Kubeflow? KServe began within the Kubeflow community and is now its own CNCF Incubating project. Kubeflow integrates with it for serving.
How do I schedule GPUs fairly across teams on Kubernetes? Kueue and Volcano both provide queueing and quota management for batch and AI workloads.
What is a ModelKit? A ModelKit is the KitOps packaging format. It bundles a model with its datasets, code and configuration into one OCI artifact.
Can I store AI models in a container registry? Yes. KitOps ModelKits are OCI artifacts and can be stored in OCI-compliant registries such as Harbor or Artifactory.
CTA: Package your first model as a ModelKit. [Get started with KitOps]