ISO/IEC 25059: Turn AI Quality Characteristics into Acceptance Evidence

ISO/IEC 25059:2023 gives AI teams a quality model. The useful work starts when a team takes one selected characteristic and writes down the requirement and measure, acceptance threshold, accountable owner and approver, and evidence. Without that chain, a quality review ends with a slide full of green cells and no reproducible acceptance decision.

I would reject any ISO 25059 mapping that uses a product feature as proof of system quality. Put the requirement sheet beside the test result on a desk. A reviewer should be able to trace the stated need to the measured result and see who accepted it.

TL;DR

• ISO/IEC 25059 supplies consistent terminology for specifying and measuring AI system quality, then evaluating the result.
• Build each quality row with the characteristic and requirement, measure and threshold, test context and owners, plus the approver and evidence.
• DeepInspect can evidence routed request and response controls, caller identity and policy decisions, plus route outcomes.
• Training-data quality and model accuracy need evidence from their own systems. The same applies to UX and human-process quality, plus lifecycle governance.

ISO 25059 is a quality model

The official IEC publication page for ISO/IEC 25059:2023 describes the standard as an application-specific extension to the Systems and software Quality Requirements and Evaluation family, usually shortened to SQuaRE. It outlines a quality model for AI systems. Its characteristics and sub-characteristics give teams consistent terminology for specifying and measuring quality, then evaluating the result. They also provide a reference against which stated quality requirements can be checked for completeness.

ISO 25059 helps a team decide which quality properties to address. Pass marks remain specific to the intended use and operating context. They also depend on affected stakeholders and the measure, the sample and threshold, plus the acceptance authority. A fraud model and support chatbot need different acceptance decisions. The same applies to a medical classifier and coding assistant.

The standard was published on 28 June 2023 as edition 1.0. This article stays at the level available in the official summary, without assigning unpublished clause numbers or reproducing the licensed characteristic catalogue.

ISO 25010 supplies the broader product-quality frame

ISO 25059 sits within a larger quality-model structure. The official IEC publication page for ISO/IEC 25010:2023 defines a product quality model for ICT and software products. IEC says its nine characteristics, with their sub-characteristics, provide a reference for specifying and measuring product quality, then evaluating the result.

IEC also lists practical uses for that model: defining requirements and checking whether requirements are complete, setting design and test objectives, identifying quality-control and acceptance criteria, and selecting measures. Select the relevant AI quality concern and write a requirement in observable language. Choose a measure, define the acceptance boundary, run the test in a declared context, and preserve the result.

ISO 25059 and ISO/IEC 5338 produce different artifacts. The ISO/IEC 5338 lifecycle checklist asks whether lifecycle processes operate across acquisition and development, transition and operation, plus maintenance and disposal. ISO 25059 gives the quality vocabulary used to state and evaluate what the AI system should be like, so a team may use both standards without merging their evidence files.

The requirement-to-evidence chain

Use one record for each selected quality characteristic or sub-characteristic. The record needs enough detail for a second reviewer to reproduce the acceptance decision.

Record the exact characteristic term and its location from the licensed ISO 25059 copy used by the organization. Add the edition so later reviewers know which vocabulary governed the decision.

Name the AI system and version. Record its intended users and task, operating environment, and relevant population. Quality results lose meaning when their context disappears.

State the quality requirement as an observable outcome. "The assistant must be reliable" is too loose. Name the behavior and operating conditions. Include the allowed failure and affected route or population.

Define the measure through its calculation and unit, sample and exclusions, test method, and data source. A label such as "accuracy" leaves the denominator and error cost unresolved.

Set the acceptance threshold before the test. Record the subgroup floor and severity rule. Include any confidence interval or escalation condition that changes the decision.

Assign the team that runs the measure and separately identify the person authorized to accept residual risk.

Preserve the acceptance evidence: test version and fixtures or dataset reference, result and timestamp, configuration and reviewer, plus the decision and link to corrective action when the result fails.

The same structure can feed an AI risk assessment. Keep quality and risk as distinct records. A quality result describes a measured property. The risk process decides what that result means for exposure and treatment, release, and later review.

Quality-characteristic matrix for an AI system

The rows below are implementation examples. Replace each working label with the exact ISO 25059 characteristic or sub-characteristic selected from the licensed taxonomy. Preserve the chain from requirement to acceptance evidence.

Task-result quality

• Requirement: The AI system must meet the approved task-performance target on a versioned evaluation set that represents the declared use case and affected populations.
• Measure: Use a task-appropriate metric with its formula and dataset version, sample size and exclusions, subgroup breakdown, and uncertainty recorded.
• Acceptance: Set the overall threshold and any subgroup floors before evaluation. Define which error types trigger rejection even when the aggregate result passes.
• Evidence: Keep the evaluation protocol and dataset reference, model and prompt versions, raw outputs and calculated results, plus the reviewer decision and defect links.
• Owner: Model evaluation and the business owner for the intended use.
• DeepInspect allocation: This row sits outside scope. Model evaluation establishes accuracy and task fitness.

Stability under declared variation

The AI system must remain within approved performance and safety bounds when inputs vary across the declared formats and languages. The same requirement applies to declared noise levels and operating conditions.

Compare baseline and varied-condition results using a pre-defined perturbation set and failure taxonomy. Set a maximum permitted degradation and identify severe failures that cause immediate rejection. Preserve the perturbation fixtures and execution configuration, before-and-after results, failed cases and review notes, plus remediation records and the retest. Model evaluation and quality engineering own this work with the use-case owner. DeepInspect has a partial allocation only when the variation is visible in routed HTTP request or response content and a route policy evaluates it. Broader model performance under varied conditions remains outside scope.

Explanation fit for the intended stakeholder

The system must provide the approved explanation content to the named stakeholder at the decision point where that explanation is required.

Test completeness and consistency with a declared protocol. Test comprehensibility and task usefulness with the declared stakeholder group. Define mandatory explanation elements and prohibited omissions. Add comprehension criteria and escalation behavior. Keep the explanation samples and protocol, participant or expert review results, system version and decision, plus corrective action. Product and UX research, domain experts, legal or compliance, and model evaluation own the row. Route evidence can show that an explanation response passed through an approved route. UX research measures comprehension with the named stakeholder group.

Human control and intervention

Authorized operators must be able to pause and override the declared AI action. They must also be able to escalate or stop it within the approved operating procedure.

Run intervention scenarios and record completion and latency, authorization checks, resulting system state, and recovery behavior. Set the allowed intervention time and authorized roles. Record the required confirmation and conditions that force escalation or shutdown. Preserve test scripts and identity records, action logs and tickets, screenshots where useful, and signed exercise results. Product operations and safety, IAM, and the business process owner own this row. DeepInspect can record an identity-bound permit or deny decision for routed LLM traffic. It can also record redact or escalation decisions. Human staffing and procedure quality remain outside scope. The same applies to off-route actions.

Data-related quality

Training and validation data must meet approved criteria for the declared use case. The requirement also covers retrieval and production-input data. Criteria include origin and permission, completeness and relevance, representativeness, and error handling.

Use dataset measures and pipeline measures tied to versioned populations and lineage. Record transformations and missingness, error rates, and review decisions. Define release thresholds and prohibited sources. Add subgroup coverage rules and lineage requirements. Record the conditions that trigger quarantine. Keep data cards and lineage records, source approvals and quality reports, dataset hashes and transformation versions, plus exceptions and remediation results. Data engineering and data governance, privacy, domain owners, and model evaluation own the evidence. DeepInspect is outside scope for training and validation data. It can classify production content deliberately routed through its HTTP boundary. Dataset-quality evidence comes from the data pipeline and its reviewers.

Operational route and policy quality

Authenticated callers may send approved data classes only to authorized model routes under the active policy. The policy must define actions for missing context and prohibited content.

Sample permit, redact, deny and escalation outcomes by caller role and data class, route, model destination, and policy version. Include tests for bypass attempts and missing identity. Require the expected decision for every test fixture. Fail closed when required context is absent and set a remediation rule for policy errors. Preserve routed request and response samples. Keep caller identity and role, route and destination, classification and policy version, decision and enforcement action, plus the timestamp and integrity record. AI platform security and IAM, data governance, and the route owner own this row. DeepInspect is in scope when the application deliberately routes authenticated HTTP LLM traffic through DeepInspect and supplies the required identity context.

Lifecycle and process quality

The organization must control acquisition and development, transition, and operation for the AI system under its adopted lifecycle. It must also control change and maintenance. Retirement must be controlled as well.

Assess process execution against defined owners and entry criteria, exit criteria and approvals, evidence locations, and corrective actions. Require each applicable process to reach its approved operating state and evidence state before release or closure. Keep lifecycle plans and supplier records, requirement baselines and change approvals, validation reports and maintenance tickets, plus disposal results. AI governance and engineering management, procurement, quality assurance, and system owners own this row. DeepInspect is outside scope as a lifecycle-governance owner. Runtime records can support selected operating tests without proving the full lifecycle process.

Acceptance evidence needs context and reproducibility

A measured number alone is weak evidence. Record the exact system version and test configuration. Add the population and time window, calculation, threshold, and reviewer. If the measure changes, version the definition rather than overwriting it. If a threshold changes after a failed test, preserve the old threshold and approval trail.

I prefer a failed row with a named owner and retest date over a green row supported by a dashboard screenshot. The failed row tells the truth and creates work. The screenshot creates an argument about filters and time zones. It also raises the question of which model version was live.

For operational controls, use the AI model monitoring guide to separate model-quality measures from route telemetry. Accuracy and calibration come from evaluation systems. Drift and subgroup behavior come from evaluation and monitoring systems designed for those measurements. Request-boundary evidence answers a narrower set of questions: who called and what route was used, which policy applied, what the decision was, and what request or response evidence the control retained.

Coverage claims need an explicit boundary

A quality program should mark each row as owned or supported. Other rows may be observed or outside scope. "Covered" is too vague for a system-quality review.

DeepInspect supports a row when its HTTP enforcement point performs part of the required control. It observes a row when routed traffic provides useful evidence without establishing the quality property itself. A row sits outside its scope when the evidence comes from training pipelines or offline evaluation. The same applies to UX studies and organizational procedures. Lifecycle governance also requires its own evidence.

This boundary also prevents a common procurement error. A record that shows a prompt was allowed under policy proves the policy decision for that event. Separate evidence must cover training-data representativeness, answer accuracy, explanation comprehension, operator competence and supplier management. The ISO/IEC 42001 Annex A controls guide can help assign governance owners. A management-system control still needs the quality measure and acceptance evidence defined here.

DeepInspect

DeepInspect covers the operational request-boundary slice of an ISO 25059 evidence matrix. The application deliberately routes HTTP traffic between authenticated users or agents and LLM endpoints through the proxy and supplies identity context. DeepInspect can then evaluate the caller and role. It can also evaluate data classification and route, model destination, and active policy before forwarding permitted traffic.

Each routed event can preserve the request or response sample allowed by the organization's evidence policy. It can also preserve caller identity and policy version, decision and enforcement action, destination and timestamp, plus route evidence. Those records support acceptance tests for identity-bound routing and policy behavior as one evidence source inside the larger AI quality program.

Book a technical deep dive at deepinspect.ai.