AI model delivery usually loses its audit trail in three places. It breaks at handoffs, when a model moves from data science to ML engineering to platform and security and each team keeps its own records. It breaks through version drift, when the model, dataset, prompts, configuration and code that were tested together end up as different versions in production. And it breaks when deployment context goes missing, so nobody can say what was deployed where, by whom, and under which policy. The practical fix is to deliver every model as one versioned, immutable artifact, attach provenance and approvals to it, gate deployment on policy, and record every step in one audit log. That is what Jozu Hub and KitOps ModelKits are built to do.
Why is auditing AI model delivery harder than auditing software delivery?
Software delivery has a mature chain of custody. Code lives in Git, builds produce a container image with a digest, the image is scanned and signed, and the cluster records what it runs. An auditor can follow one identifier from commit to pod.
A model has no equivalent by default. It is weights in object storage, a dataset in a lake, a notebook or training script in Git, prompts in a config repo, hyperparameters in an experiment tracker and serving settings in a Helm chart. Each piece has its own versioning (or none), and none of them point reliably to the others. You can audit each system and still be unable to answer the basic question, which is "what exactly is this model, and how did it get here?"
Where does auditability break at team handoffs?
A model typically passes through three or four teams before production, and each handoff loses information.
- Data science to ML engineering. The handoff is often a path to a weights file plus a message describing which run it came from. The experiment tracker knows the run, but the link between that run and the file that was handed over is informal.
- ML engineering to platform. ML engineers repackage the model for serving, often into a container with its own build. The model inside is now one layer removed from its training record, and any changes made during packaging are rarely captured.
- Platform to security review. Security reviews what it is shown, which may be a container image, a model card or a spreadsheet. If what was reviewed is not cryptographically tied to what is deployed, the approval is a statement about intent, not about the artifact.
The common failure is that each team passes along a description of the model rather than the model itself, with its history attached. Records multiply and drift apart, and reconciling them during an audit or incident is slow manual work.
How does version drift break the audit trail?
A model's behavior depends on more than its weights. The same weights with a different system prompt, tokenizer config, preprocessing code or retrieval dataset is effectively a different system. Drift happens when these pieces are versioned separately and promoted separately.
Common examples include the following.
- The weights in production match staging, but the prompt template was edited directly in the production config.
- Retraining produced new weights, but the evaluation report on file refers to the previous dataset version.
- Preprocessing code was patched in one environment and not the others.
- A hotfix rolled back the model image but not the serving config that was changed alongside it.
Each change may be harmless on its own. The audit problem is that you can no longer state which combination was tested, which was approved and which is running. If the answer differs by environment, your validation evidence does not describe your production system.
What deployment context goes missing?
Even when the artifact is well defined, the context around its deployment is often scattered across CI logs, chat threads and ticket comments. An auditor, regulator or incident responder typically needs the following.
- Which exact version (by digest, not by a mutable tag like "latest") is running in each environment and cluster.
- Who approved it, and who triggered the deployment.
- Which policy it was checked against, and the result of each check.
- What scans ran on it, with what findings, before it was allowed through.
- When it changed, and what it replaced.
If this context lives in five systems with different retention rules, it effectively does not exist when you need it.
How do you make model delivery auditable?
The fix follows the pattern that already works for containers. Make the model a single artifact, make its history travel with it, and make the cluster refuse anything without the right history.
Package the whole project as one versioned artifact
KitOps packages weights, datasets, code, prompts and configuration into a ModelKit, an OCI artifact with its own digest. When the prompt changes, the ModelKit version changes. That removes most version drift by design, because the pieces that were tested together can only be promoted together. ModelKits live in OCI registries, so they fit the registries, CI/CD tools and access controls platform teams already run.
Hand off the artifact, not a description of it
Data scientists push a ModelKit (from the CLI or with the PyKitOps SDK in Python), and ML engineering, platform and security all pull the same thing. Every team works from one identifier, and each team's actions are recorded against it.
Attach provenance and scan results to the artifact
Jozu Hub scans each ModelKit version for AI-specific risks and keeps the results with the artifact, so the security context follows it wherever it goes. Signing approved versions (for example with Sigstore cosign) ties the approval to the exact digest. Note that kit pack and kit push do not sign anything. Signing is a separate, deliberate step, which is the point of an approval.
Gate deployment on policy
Promotion and deployment should check the artifact automatically. Typical rules include requiring a passing scan, an approved license, a valid signature from the release identity, or a region restriction for models that handle regulated data. Because the checks run against the artifact and its attached results, the same rule can be applied in CI and at cluster admission, and Jozu Agent Guard can admit only approved artifacts.
Record every step in one audit log
Jozu keeps an audit log of what was pushed, scanned, approved and deployed, and for agents, what was allowed, blocked or redacted at runtime. It can be forwarded to your SIEM so model delivery sits alongside the rest of your security evidence. Because every version is immutable, rollback is a redeploy of a known digest, and the log shows exactly what changed.
Model delivery auditability checklist
Use this to test your current process. Pick one model in production and try to answer each item in under an hour.
- I can name the exact digest running in each environment.
- Weights, dataset version, prompts, code and config for that digest are retrievable as one unit.
- Staging and production run the same combination, or I can show precisely how they differ.
- I can show which scans ran on this version and their results.
- I can show who approved it and that the approval is bound to this digest (for example by signature).
- I can show which policy admitted it to the cluster.
- I can show who deployed it, when, and what it replaced.
- I can roll back to the previous version without rebuilding anything.
- All of the above lives in records my security team can query, ideally in the SIEM.
Every unchecked box is a place where your audit trail depends on someone's memory.
How does this fit with the tools teams already use?
It complements them. Data scientists keep using notebooks, MLflow or Weights & Biases for experiments. Serving stays on KServe, Ray or your existing platform. CI/CD stays on GitHub Actions, GitLab CI, Jenkins or OpenShift Pipelines. What changes is the unit that moves between them. Instead of a loose collection of files and links, every stage handles one ModelKit, and Jozu Hub keeps its provenance, scan results, approvals and audit history.
FAQ
What is AI model delivery auditability? It is the ability to show, for any model in production, exactly what it contains, where it came from, who approved it, which policy admitted it and when it changed.
Isn't an experiment tracker like MLflow enough? Experiment trackers record training runs well. They do not usually bind those runs to the exact artifact that was deployed, or record approvals and admission policy in production.
What is a ModelKit? A ModelKit is the KitOps packaging format, a single OCI artifact that bundles a model with its datasets, code, prompts and configuration.
Does packing a ModelKit sign it?
No. kit pack and kit push package and upload the artifact. Signing is a separate step, typically done when a version is approved.
Does Jozu replace our registry or CI/CD? No. ModelKits are OCI artifacts that work with existing registries and pipelines. Jozu adds scanning, policy gates and the audit log around them.
CTA: See how Jozu gives every model one auditable path to production. Book a demo at jozu.com