NYDFS Part 500 AI Audit Evidence for Monitoring and the 72 Hour Notice

An examiner asks a covered entity to show which authorized users sent nonpublic information to an external model during a named week, what policy decided, and how the entity would have detected misuse. Section 500.14(a)(1) of the amended 23 NYCRR 500 requires risk-based controls designed to monitor authorized user activity and detect unauthorized access or use of, or tampering with, nonpublic information by those users. NYDFS Part 500 AI audit evidence answers that from a saved query or it does not answer it at all.

I build the package so the CISO can reuse the same queries at 2am when the 72 hour clock in 500.17(a)(1) has already started.

TL;DR

• Freeze a population of in-scope AI requests for a named window before sampling, including permitted, redacted, denied and failed events.
• Anchor the package on 500.14(a)(1) monitoring, 500.7 access privileges, and the asset inventory fields required by 500.13(a).
• Test integrity by tampering with a copy and running the documented check, then attempting a modification as an application administrator.
• Keep the same record set usable for the 72 hour notice under 500.17(a)(1) and for the April 15 certification file under 500.17(b)(1).

Freeze the population before sampling

Define every in-scope HTTP AI request during a named window in a stated time zone. The population holds successful model calls along with redactions, denials, policy errors, missing-identity failures and blocked responses. A population assembled from successful calls alone removes the events that best demonstrate the control operated.

The manifest carries a stable event identifier, timestamp, originating principal, calling application, model destination, policy version, action, data classification, and record location. Freeze it, then let the examiner select rows or document a repeatable selection method. Screenshots picked by the control owner are a demonstration. A frozen manifest with examiner-selected rows is evidence, and the gap between the two becomes obvious the moment someone asks why the denial count is zero.

Align the population boundary with the asset inventory required by 500.13(a), which must be complete, accurate and documented, tracking for each asset as applicable the owner, location, classification or sensitivity, support expiration date and recovery time objectives, plus the frequency for updating and validating it. An AI route inside the evidence population and missing from the inventory is a finding on its own.

Bind each sample to the access decision

Every selected event needs the identity assertion and its validation result, the model destination and route, the policy rule and version, the enforcement outcome, and the response disposition. Section 500.7(a)(1) limits user access privileges to information systems providing access to nonpublic information to only those necessary to perform the user's job, so the evidence must show that limit applied to a named person on a specific request.

The originating identity is where these packages usually break. An application authenticates a claims adjuster, then calls the provider through a single shared service account, and the provider record shows the relay rather than the adjuster. Joining the upstream authentication event to the outbound request by correlation identifier is what keeps the human recoverable. The AI agent identity guide covers that design problem.

Section 500.7(a)(4) requires review of all user access privileges at least annually with removal or disabling of accounts and access no longer necessary, and (a)(6) requires prompt termination following departures. Include a sample where a departed user's credential was presented to the AI route and denied, with the timestamp relative to the departure record. That single pairing evidences two paragraphs at once.

Prove detection and protect the records

The monitoring duty at 500.14(a)(1) needs four artifacts: the classification rules in force with their version, a staged test showing a nonpublic information pattern detected and blocked or redacted, the alert record it produced, and the investigation note that closed it. Add a permitted request alongside it so the examiner can see the control discriminating rather than blocking everything.

Then prove the records resist the system that wrote them. Alter one decision field on a copied event, execute the documented integrity check, and record the failure output and the reviewer. Attempt a modification as an application administrator against the protected store and record the rejection. Two short tests settle custody more convincingly than a policy paragraph. The tamper-evident AI audit logs guide describes the write path they exercise.

A Class A company under 500.1(d) owes three more artifacts in the same folder. Section 500.14(b)(2) calls for a solution that centralizes logging and security event alerting, (b)(1) calls for endpoint detection and response monitoring anomalous activity including lateral movement, and 500.7(c)(1) calls for privileged access monitoring with a privileged access management solution. Where any of those is absent, the file needs the CISO's written approval of reasonably equivalent or more secure compensating controls.

Keep the same records usable for reporting

Section 500.17(a)(1) requires notice to the superintendent electronically, in the form on the department's website, as promptly as possible and no later than 72 hours after determining that a cybersecurity incident occurred at the covered entity, its affiliates, or a third-party service provider. Paragraph (a)(2) adds a continuing obligation to update the superintendent with material changes or newly available information.

Save the queries in advance and drill them: all requests for a principal, all traffic to a model destination, all events under a policy version, all denials in a window, and all events carrying a regulated classification. Time the drill and keep the artifact, because the timing evidence is the part that proves the capability existed before the incident rather than after it.

Section 500.17(b)(1) requires an annual electronic submission by April 15, either a written certification that the entity materially complied with Part 500 during the prior calendar year, based upon data and documentation sufficient to accurately determine and demonstrate that material compliance, or a written acknowledgment that it did not fully comply. The AI evidence package becomes one folder inside that documentation index. A certification signed without the index behind it is the exposure.

Retention and the uncovered edges

Section 500.13(b) requires policies and procedures for secure disposal on a periodic basis of nonpublic information identified in 500.1(k)(2) and (3) that is no longer necessary for business operations or other legitimate business purposes, subject to legal retention requirements and feasibility. Prompt and response records containing that information inherit the rule, so the package shows the retention schedule, the disposal job output, and the reasoning for any records held longer for examination purposes. Test retrieval against the oldest available period and compare the returned record with its integrity value. The LLM audit log retention guide covers how that design decision plays out.

Name what the package cannot cover. Direct provider traffic from an unmanaged device, local model execution, browser sessions outside the enforced route, compromised endpoints, and inference embedded in a vendor product where the call is invisible all sit outside a routed record set. So do the penetration testing and vulnerability assessments under 500.5, the risk assessment under 500.9, and third-party service provider policies under 500.11. The NYDFS AI compliance overview covers how those pieces fit together.

DeepInspect

DeepInspect intercepts HTTP AI traffic between authenticated users or agents and LLM endpoints. On routed traffic it evaluates the identity and policy context the application supplies, classifies the request, applies a permit, redact or deny decision, inspects the response, and writes a signed per-decision record. That gives a covered entity a population export, examiner-selectable samples, integrity checks, saved queries, and historical retrieval against the monitoring duty at 500.14(a)(1) and the access limits at 500.7(a).

Identity proofing, enforced routing, endpoint detection, encryption key management, privileged access tooling, penetration testing, the risk assessment, and the April 15 certification stay with the roles that already own them. Book a demo today.