ISO/IEC 23894 AI Audit Evidence at the Request Boundary

An AI audit evidence packet should let a second reviewer retrieve the assessment, treatment and test for one risk without asking the control owner which folder to search. For ISO/IEC 23894, build that chain around a stable system and risk ID, then show how the treatment performed after approval. ISO/IEC 23894:2023 is customizable guidance on AI risk management. The packet below is an illustrative working format, not a list of artifacts mandated by the standard.

TL;DR

• Keep a dated index that points from system ID to risk row, treatment, test, approval and monitoring sample.
• Retain only the evidence needed to prove the control; a redacted decision record can be more useful than a full customer prompt.
• Ask a reviewer outside the team to reproduce one sample retrieval before declaring the packet ready.

A sample artifact index

Consider SUP-17, a support-case summarizer using a hosted model. Its controls mapping records R-01, the risk of case data reaching an unapproved model, and R-02, the risk of an invented commitment in the generated summary. The evidence index below is a sample design. Each item needs a storage location, access owner, creation time, retention rule and version or hash so a later export can be tied to the reviewed state.

• E-01, system record: the application owner keeps the approved SUP-17 inventory and data-flow diagram. Index the approval date, model route, data classes and diagram version. Review it after a new provider or case-data field is introduced.
• E-02, risk assessment: the risk lead stores R-01 and R-02 with scenario, affected party, rating criteria, uncertainty and assessment date. Link the rubric version used on that date; a score copied without its criteria is hard to reproduce.
• E-03, treatment decision: the support director records the chosen controls, due dates, exceptions and residual-risk acceptance. Store the sign-off separately from the technical test so the assessor can see who made the business decision.
• E-04, control configuration and test: the platform engineer keeps the approved route-policy version and the allowed and denied fixtures for R-01. The index includes the run time, expected outcome, observed outcome and corrective ticket if a fixture failed.
• E-05, operating sample: the evidence custodian stores a redacted decision record with its request ID, timestamp, policy version, route and outcome for a sampled R-01 event. Keep full case text out of the audit export unless the privacy owner approves a specific need and retention period.
• E-06, model-quality sample: support operations keeps the R-02 evaluation set, human review notes and an issue record for unsupported promises. Request telemetry gives a call trail; it cannot certify that a customer summary was accurate.

The AI risk register template supplies a compatible risk-row structure. Set retention according to the organization's legal and contractual schedule, recording the policy ID in this index rather than inventing a universal number of days. If the policy allows an evidence item to expire, record the expiry and the approved replacement source before it disappears.

Reproduce one retrieval

Give an independent reviewer only SUP-17, R-01 and a review date. First, they locate E-02 and confirm the scenario and assessed scope. Next, they open E-03 to see who approved the route restriction and when. They use the policy version in E-04 to retrieve the allowed and denied fixtures. Finally, they retrieve E-05 by request ID and verify the timestamp, route and outcome against that version. Record the retrieval time, redactions and any missing link in a review log.

The same procedure must expose gaps. If the test used policy version 12 but the operating sample says version 13, the reviewer asks for the intervening change and its retest. If E-05 is missing because the event store was unavailable, mark the packet incomplete and open an incident or corrective action. I would rather show one documented gap than fill it with an unrelated screenshot.

The index is the map. When the approval lives in a ticket system, the route policy in a configuration repository and the decision sample in an event store, the reviewer still needs one stable identifier that joins all three without exposing a customer's full case file.

The operational checklist helps owners schedule that review after a model or data change. The NIST AI RMF Playbook offers voluntary actions that can inform governance and monitoring, but its suggested actions do not substitute for the actual E-04 and E-05 samples.

DeepInspect

For HTTP AI traffic routed through DeepInspect, a policy decision record can contribute E-05's route, policy and outcome evidence when the integration supplies the relevant caller context. The organization remains responsible for the risk assessment, model-quality tests, retention decision and approval record. A gateway sample should be joined to those other artifacts by a stable system and risk ID rather than presented as the complete audit packet.

Book a technical deep dive at deepinspect.ai.