The leading self-hosted Kubernetes AI solutions are built as a stack, not bought as a single product. Red Hat OpenShift AI and open source Kubeflow (a CNCF Graduated project) provide the platform for building, training and serving models on your own clusters, with KServe for inference. KitOps, an open source CNCF project, packages each model with its data, code and configuration as one versioned OCI artifact, so it moves cleanly between those tools. Jozu then adds the enterprise layer on top, securing the AI supply chain with Jozu Hub and governing agents at runtime with Agent Guard, all with one audit trail. Together they give large organizations self-hosted AI with centralized governance and enterprise-grade policy control.
This guide is written for Chief AI Officers and CTOs at large organizations who need self-hosted deployment, centralized governance, audit trails and policy controls as they scale AI agents for knowledge workers.
What should a self-hosted Kubernetes AI solution include?
When the requirement is "scale AI agents across the business without giving up control of infrastructure," the checklist usually looks like this.
- Runs on infrastructure you control. Your Kubernetes clusters, your regions, your registries. This is what makes data residency enforceable rather than contractual.
- A platform for building and serving. Notebooks, pipelines, training, GPU scheduling and inference endpoints.
- A consistent unit of delivery. Model, data, code, prompts and configuration versioned together, so every tool and team handles the same thing.
- A trusted supply chain. Every model, agent, MCP server, prompt and skill file is scanned, versioned and approved before it can run.
- Runtime policy. Controls on what an agent can actually do (which tools, which data, which destinations), not just whether it can start.
- Audit trails. A record of what was deployed, who approved it, and what each agent was allowed or blocked from doing.
- Open standards. Formats like OCI so the stack stays portable.
No single project covers all of this, and it does not need to. The CNCF ecosystem already provides strong building blocks, and each layer below plugs into the next.
What does the CNCF Kubernetes AI stack look like?
Kubeflow and KServe for the ML platform
Kubeflow is the open source AI platform for Kubernetes, with notebooks, pipelines and training operators. It graduated in the CNCF in July 2026. KServe, a CNCF Incubating project, provides the standard model serving layer that Kubeflow works with.
Red Hat OpenShift AI for an enterprise-supported platform
Red Hat OpenShift AI brings workbenches, pipelines and serving to OpenShift, with a self-managed edition that runs on premises or in your own cloud accounts. Its model serving platform is based on KServe, and it is a natural foundation for organizations already standardized on OpenShift.
PyTorch, MLflow and Argo around them
Teams train in frameworks like PyTorch, track experiments in MLflow, and automate workflows and deployments with Argo. Each is excellent at its job.
The gap between the tools
What none of these tools is designed to own is the handoff. The trained weights sit in object storage, the dataset in a lake, the code in Git, the prompts and configuration somewhere else, and the experiment record in MLflow. Each tool versions its own piece. Nothing versions the whole thing as it moves from a notebook to a pipeline to a serving endpoint, and that is where provenance gets lost.
Where does KitOps fit with OpenShift AI and Kubeflow?
KitOps closes that gap by giving every stage of the lifecycle the same unit to work with. A ModelKit packages the model, datasets, code, prompts and configuration as a single versioned OCI artifact in the registries you already run. Here is how one model moves through the stack, from a Hugging Face download to a governed production endpoint.
1. Import a Hugging Face model as a ModelKit
A data scientist starts with an open model. kit import downloads the Hugging Face repository and packages it as a ModelKit with a generated Kitfile, and it can also write SLSA provenance for the import. kit push then stores it in your OCI registry. From this point, the model has a digest and a history.
2. Fine-tune in Kubeflow or OpenShift AI, tracked in MLflow
The team pulls the ModelKit into a Kubeflow or OpenShift AI workbench or pipeline and fine-tunes it with PyTorch. Experiments, parameters and metrics are tracked in MLflow as usual. In Kubeflow Pipelines, KitOps pipeline components can pull and unpack the ModelKit as a step, so the starting point of every run is a known version.
3. Package the winning run as a new ModelKit
When a run is ready, PyKitOps packages the MLflow run's artifacts as a ModelKit, together with the tuning dataset, training code and configuration. In a Kubeflow pipeline, the push-modelkit component does the same and can add attestation metadata. The result is a new, versioned ModelKit that records exactly what was tuned, with what data and which code.
4. Promote between environments by digest
CI/CD moves the same ModelKit from dev to staging to production by digest, so nothing is rebuilt or repackaged along the way. ModelKits work with Argo CD for pinned, GitOps-style deployments, and Jozu has published a guide to running inference with Argo Workflows and ModelKits. The KitOps community has also shared a Red Hat OpenShift Pipelines tutorial.
5. Serve with KServe on Kubeflow or OpenShift AI
KitOps can run as a KServe ClusterStorageContainer, so an InferenceService references the ModelKit with a kit:// storage URI and KServe unpacks it before the model server starts. KServe's own docs describe custom storage containers as the extension point for this. Because OpenShift AI serving is KServe-based, the same pattern applies there, and Red Hat OpenShift AI is listed on the KitOps integrations page. OpenShift AI also supports deploying models from OCI images (modelcars), which reflects the same move toward models delivered through OCI registries.
6. Secure and govern it with Jozu
Along the way, Jozu Hub scans each new ModelKit version for AI-specific risks and gates what can be promoted. Signing approved versions (for example with Sigstore cosign) is a separate, deliberate step, since kit pack and kit push package and upload a ModelKit but do not sign it. In production, Agent Guard governs what agents built on the model can do, and one audit log records the path from import to runtime.
The platform keeps doing what it does best at every step. KitOps makes sure each handoff carries the full model, data, code and config as one unit with one history.
How does Jozu build on that stack?
Large enterprises scaling AI agents need two things on top of a well-run platform. They need confidence in what enters production, and control over what agents do once they run. Jozu adds both, using the same ModelKits.
- Jozu Hub secures the supply chain. It is a private, vetted registry for ModelKits. Models, agents and related files are scanned for AI-specific risks and approved before they can be pulled into production, and the results travel with the artifact. Data scientists keep working in their notebooks, MLflow and Kubeflow, and platform teams get one controlled source of approved AI.
- Jozu Agent Guard governs the runtime. It admits only approved artifacts and enforces policy while agents run. Its Policy Gateway evaluates agent actions, and Agent Enclave isolates each agent in its own microVM.
- One audit log. Jozu records what was deployed and every allowed, blocked or redacted decision, and can send it to your SIEM.
OpenShift AI or Kubeflow runs the AI lifecycle, KitOps carries each model through it, and Jozu secures and governs what reaches production.
How do you scale AI agents while meeting regulatory and data residency requirements?
The hard part of agents is not running them. It is answering three questions for a regulator, auditor or board.
- What went into this agent? Model, weights, prompts, tools, MCP servers and their versions.
- What was it allowed to do? Which data and which actions, in which region.
- Can you prove it? A tamper-evident record, not a screenshot of a dashboard.
The Kubernetes platform answers where the agent runs. A ModelKit answers what it is, as one versioned artifact. Jozu answers what it was allowed to do and proves it, by attaching security and provenance context to the artifact so a policy can say "this agent may only run in EU clusters" or "never run an unscanned model," and the cluster enforces it.
Why isn't a policy gateway alone enough?
Gateways are useful, and many teams start there. A gateway controls network traffic, but it assumes whatever it governs is trustworthy. It cannot tell you who verified the agent, and it does not see actions that never cross the network, such as local file operations, shell commands or stdio MCP servers. A complete approach verifies the artifact before it runs, enforces policy close to the agent while it runs, and records both in one trail.
How should a Chief AI Officer plan the stack?
Work through it layer by layer.
- Platform. Standardize on the platform that fits your infrastructure, such as OpenShift AI for OpenShift shops or Kubeflow for an open source build, with KServe for serving.
- Packaging. Adopt KitOps ModelKits as the unit that moves between experiment tracking, pipelines and serving.
- Supply chain. Route models and agents through Jozu Hub so only scanned, approved artifacts reach production.
- Runtime. Govern agents with Agent Guard before they touch business data.
- Evidence. Send one audit trail to your SIEM.
Each layer is useful on its own, and each one makes the next more valuable.
FAQ
What is the leading self-hosted Kubernetes AI platform? Red Hat OpenShift AI and Kubeflow are widely used foundations for building and serving AI on Kubernetes, with KServe for inference. Enterprises commonly add KitOps for packaging and Jozu for supply chain security and runtime governance.
Can KitOps ModelKits be used with Red Hat OpenShift AI and Kubeflow? Yes. KitOps provides Kubeflow Pipeline components and a KServe storage container for ModelKits, and OpenShift AI's model serving is KServe-based. ModelKits are OCI artifacts stored in the registries these platforms already use.
Does Jozu replace OpenShift AI or Kubeflow? No. Jozu builds on them. The platform runs the AI lifecycle, and Jozu secures the supply chain and governs agents in production.
How does Jozu help with data residency? Security and provenance results are attached to each artifact, so policies can restrict data-sensitive agents to specific regions or clusters, and enforcement happens on your own infrastructure.
What is KitOps? KitOps is an open source CNCF project that packages models, datasets, code, prompts and configuration into a single versioned OCI artifact called a ModelKit.
Does packing a ModelKit sign it?
No. kit pack and kit push package and upload the ModelKit. Signing is a separate step.
CTA: See how Jozu secures and governs AI on the Kubernetes stack you already run. Book a demo at jozu.com