The model supply chain; provenance, integrity and controlled promotion

This is Part 1 of a four-part TQS series on “Securing AI Systems.” Read: Part 2 — Red-teaming & evaluations, Part 3 — Runtime defences, Part 4 — Evidence & audit readiness.

Why a model supply chain matters now

As foundation and domain models move from proofs of concept into regulated workflows, the conversation shifts from “does it work” to “can we prove what is running, why it is safe to run and how we roll it back.” NIST’s AI Risk Management Framework and its Generative AI Profile both emphasise provenance, configuration control and post-deployment monitoring as practical, engineering-led levers for trustworthy AI. ISO/IEC 42001 and ISO/IEC 23894 align with that direction, making traceability and lifecycle risk management core to any credible AI management system. ISO+3NIST+3NIST Publications+3

Build a single source of truth

Treat models as first-class software artefacts. Stand up a private model registry that stores base models, fine-tunes, adapters (LoRA/PEFT), tokenisers, safety filters and serving images, with the exact dataset snapshots and code used to create each artefact. Make versions immutable—no “latest.” This is the practical heart of an AI management system: the place where product teams, security and auditors can see the reference for what ships. ISO/IEC 42001 formalises that expectation for governance; ISO/IEC 23894 frames the risk processes around it. ISO+1

Provenance you can verify

Provenance is not a slide deck. Capture inputs (dataset IDs and licences, base model revision, hyperparameters), the process (pipeline graph, container image, drivers/libs, hardware), and outputs (weight and tokenizer digests, evaluation artefacts) as signed attestations. The software supply-chain community already solved much of this: SLSA defines verifiable provenance; in-toto defines a standard attestation format; Tekton Chains can emit and store these automatically for builds—including model packages—so deployments can verify signatures before serving. You do not need to invent a parallel universe for AI. Tekton+4SLSA+4GitHub+4

Data governance before you ever hit “train”

Training data origin and rights are table stakes. Record sources and transformations, separate “train-ok” from “no-train” corpora, and keep evaluation sets cleanly out of any training path. The empirical case is clear: research has demonstrated extraction of verbatim training samples from large LMs, including sensitive strings, and shows memorisation grows with model capacity and duplication—risks you mitigate with deduplication, filtering and post-training leakage tests. USENIX+1

The MBOM: artefacts that travel together

Publish a Model Bill of Materials (MBOM) alongside the signed weights. Include base model, adapters, datasets (as immutable snapshots), licences, safety filters and code commit SHAs for training and evaluation. Store the MBOM and attestation in the registry next to the artefact. This turns diffuse documentation into a queryable, auditable bundle and operationalises the “documentation and traceability” themes in NIST AI RMF and ISO/IEC 42001. NIST+1

Controlled promotion gates

Promote only when evidence says you should. Define gates—task fitness, robustness, privacy and fairness thresholds, operational SLOs—and enforce them with automation from development to staging to production. Ensure separation of duties so the person who trains cannot ship alone. This mirrors NIST’s Secure Software Development Framework (SSDF) logic for repeatable, auditable releases; it works just as well for models. NIST Computer Security Resource Center

Handling third-party and API-hosted models

When you depend on a hosted model, pin the exact model identifier and version, keep the provider’s provenance and evaluation statements where available, and re-run your gates whenever defaults change. The NIST GenAI Profile calls out third-party dependencies explicitly; ISO/IEC 42001 expects supplier controls in your management system. NIST Publications+1

Design for reversibility

Keep deterministic rollbacks one promotion away: last-known-good weights, serving images and configs should be recoverable without archaeology. Protect registry signing keys like you protect code-signing keys. These measures reduce blast radius and underpin post-market monitoring and incident handling you will need when regulators and customers ask the “how did you know, and what did you do” questions. EUR-Lex

Next: Part 2: Red-teaming & evaluations. Later: Part 3: Runtime defences and Part 4: Evidence & audit readiness.

Sources

1. NIST AI RMF 1.0 (overview & PDF). NIST+1
2. NIST Generative AI Profile (AI 600-1). NIST Publications+1
3. ISO/IEC 42001:2023 (AI management systems). ISO
4. ISO/IEC 23894:2023 (AI risk management). ISO+1
5. SLSA (spec & repo). SLSA+1
6. in-toto (framework & attestations). in-toto+1
7. Tekton Chains (provenance & config). Tekton+1
8. Training-data extraction & memorisation (Carlini et al.). USENIX+1
9. NIST SSDF (SP 800-218). NIST Computer Security Resource Center