OSFI E-23 Model Risk: Production Evidence for Routed AI

OSFI Guideline E-23 takes effect on 1 May 2027. It expands model risk management across all federally regulated financial institutions and expressly covers AI and machine learning methods. For a bank or insurer routing production requests to an LLM, the hard part begins after the model enters service because the institution must prove which use occurred under which approved conditions.

I would start with one production event, printed on a single page. It should identify the caller and model version, then show classification, policy and override decisions, and business outcome on the same page. If six teams need an afternoon to assemble it, the monitoring process still lives in diagrams.

TL;DR

• OSFI E-23 applies to all federally regulated financial institutions on 1 May 2027. It uses a risk-based approach across the model lifecycle.
• Routed LLM monitoring should connect each production event to the caller and model version. It should also preserve classification, policy and override decisions, and business outcome.
• A request gateway can supply route-level policy evidence. Model development and independent review remain with the institution, alongside business-outcome assessment and risk acceptance.
• Third-party and black-box models still require governance and validation. Monitoring should be commensurate with their model risk.

E-23 brings AI into enterprise model risk management

The final OSFI Guideline E-23 defines a model broadly enough to include AI and machine learning methods that process input data to generate results. Its scope reaches all federally regulated financial institutions, including foreign bank and insurance branches where the guideline is consistent with their Canadian obligations. OSFI applies the guideline proportionally according to an institution's size and strategy, as well as its risk profile and operational complexity. Interconnectedness also affects that judgment.

The use case is decisive. A hosted LLM used for document summarization may receive a different risk rating from the same model used to rank a fraud queue or draft a customer decision. OSFI's response to stakeholder comments says institutions make that judgment in their organizational context, and models with non-negligible inherent risk enter the model inventory and full lifecycle governance.

The broader Canada AI compliance checklist covers privacy and automated-decision obligations beyond financial supervision. E-23 owns the narrower supervisory question here. It addresses how a regulated institution identifies and governs model risk, then monitors and reports it.

The supervisory crosswalk starts with the model inventory

E-23 calls the model inventory the enterprise system of record for models with non-negligible inherent risk. Appendix 1 lists the minimum information, starting with model ID and name, plus description and risk rating. It also records owner and developer, along with origin. For an inventoried model, the institution maintains the version and production deployment date, identifies reviewers and approvers, and documents dependencies and data sources, approved uses and limitations, and review dates and monitoring status.

For routed LLM use, one inventory entry may need several deployment records beneath it. Each record should name the calling application and the provider route. Add the provider and its returned model identifier, along with the version or dated release where the provider exposes one. State the approved business use. Record the identity source and applicable data classes. Identify the policy and monitoring owners. The same record should show override authority and the outcome system.

The AI model inventory management guide describes the wider inventory artifact. Request telemetry can reconcile actual routes against that approved register. Observed traffic cannot assign the risk rating or approve the use.

Production monitoring needs event-level evidence

OSFI's model-monitoring principle says monitoring should keep models fit for purpose and detect performance issues or breaches. The monitoring standard should define frequency and scope, with evaluation criteria set by risk rating and model type. OSFI also expects quantitative measures and qualitative assessment, including verification that the model remains within its original scope. The review covers changes in model performance and usage, as well as inputs and external dependencies, plus the subject being modelled.

A routed LLM event should preserve these fields:

• Caller: the authenticated person, agent, or service supplied to the request boundary, including any verified on-behalf-of identity.
• Model and version: the provider and route, plus the model identifier and exposed release or version. Add a flag when the provider supplies no stable version.
• Classification: the request and response data classes produced under the institution's approved taxonomy.
• Policy decision: the policy identifier and version, matched rule and decision, and timestamp and reason code.
• Override: the approving authority and stated reason. Preserve its scope and effective time, as well as its expiry and linked exception record.
• Outcome: the downstream business result and review status, joined through a correlation identifier to the routed request.

The AI audit-log schema provides a technical starting point for the request record. E-23 still requires the institution to define which outcomes and thresholds make that evidence meaningful for the model's approved use.

Overrides and outcomes expose monitoring failures

OSFI names excessive overrides as one reason a model may be decommissioned. An override is therefore a monitored event rather than a free-text note buried in a case system. The evidence should preserve who authorized it and which model result or policy decision changed, why the change was permitted, and how long the exception remained active. Reviewers can then distinguish a one-off correction from a pattern that signals poor fit or drift, including use outside scope.

Outcomes need a separate system owner. A gateway can record that a claims application sent a classified request to an approved model and received a permitted response. The claims platform knows what recommendation entered the case file. Later records may show a payment or denial, an analyst correction or customer complaint, or a confirmed fraud result. Correlation IDs connect those records. Request telemetry alone does not contain the business outcome.

I would reject aggregate latency and token counts as model-monitoring evidence. They describe request volume and service speed alongside cost, while E-23 asks whether the model remains fit for its purpose. That judgment needs sampled events tied to actual use and results.

Thresholds turn records into a monitoring control

E-23 expects institutions to define breach thresholds and criteria for material model modifications. It also expects contingency plans for unavailability, deteriorating performance, or failure, with escalation procedures attached. A useful monitoring specification names each indicator and its data source. It identifies the calculation owner and review frequency, then sets the threshold and required action.

For a routed LLM workflow, the indicator set may include use outside the approved caller population and attempts to reach an unapproved model. It may also cover classification-policy conflicts and expired overrides. Add provider version changes and missing identity context. Business owners should contribute outcome indicators suited to the use case, such as analyst reversal or confirmed error categories. The model risk function sets the materiality and escalation treatment.

When a threshold fires, preserve the reviewed event population and decision maker alongside the chosen action. Link the remediation ticket and retest result. The action may restrict a route or add a compensating control, trigger independent review, or change the risk rating. Suspension remains another option. E-23 leaves those choices risk-based. The evidence should show which choice the institution made.

Third-party LLMs retain the same governance burden

OSFI's E-23 release letter and stakeholder response rejects a governance exception for third-party black-box models. Institutions should apply Guideline B-10 principles and ensure validation matches the model risk, with monitoring at the same level. OSFI also declined to reduce expectations for vendors with proprietary techniques or data. The institution needs controls that assess whether the vendor solution produces an appropriate response.

A provider update therefore belongs in the monitoring evidence when it changes an external dependency. Record the first observed use of a new model identifier or version, then compare it with the approved route and apply the institution's change and review criteria. Where a provider exposes only a mutable model alias, preserve that limitation in the inventory. Retain any provider request ID or release notice available for later reconstruction.

OSFI permits an exceptions policy for limited and specific use before validation finishes. That route requires an explicit scope and approval, as well as an expiry and compensating controls. The exception should stay visible in monitoring reports until it closes.

Model development and independent review keep separate owners

E-23 assigns standards for model development, including documentation and methodology. Those standards also cover assumptions and performance tests, plus explainability and monitoring criteria. Its model-review principle requires an independent assessment of conceptual soundness and performance. A request gateway operates after those responsibilities and supplies evidence to them. It cannot inspect a vendor's training data or validate the methodology, conduct the independent review, or issue the recommendation to the model approver.

The distinction is especially important beside the US supervisory history covered in the banking AI model risk guide. Canadian E-23 expressly includes AI and machine learning methods in its model definition and lifecycle expectations. An institution should build its E-23 crosswalk from the Canadian text rather than importing a US scope decision.

Use routed production records as inputs to review. Independent reviewers can test approved scope and inspect overrides. They can also examine performance breaches and select events for outcomes analysis. The model owner remains accountable for remediation. Senior management receives the enterprise view required by E-23.

DeepInspect

DeepInspect can produce the request-boundary part of an E-23 production evidence file when an institution deliberately routes authenticated HTTP LLM traffic through its stateless proxy. Each decision can bind the supplied caller identity to the provider and model route. It can also preserve classification and policy version, the matched rule and enforcement result, and the timestamp before permitted traffic continues.

Those records can reconcile observed routes with the model inventory. They also support event sampling and policy-breach review, including override investigation. The application should attach a correlation identifier so reviewers can join the request decision to the downstream business outcome. Model development and independent review remain with the institution. So do risk rating, outcome assessment, and approval.

Book a technical deep dive at deepinspect.ai.