Does the FDA regulate the LLM API calls my clinical software makes?

The FDA reviews the AI model cleared as part of a device submission, so on-device inference falls squarely inside its authority. The HTTP calls your clinical software makes to an external LLM API are a different matter. They are generally not the cleared device, yet they still land inside the FDA quality system when that software is part of a regulated product, because 21 CFR Part 11 and design-control expectations reach the electronic records the software produces. HIPAA also applies the moment protected health information enters a prompt, which pulls in Business Associate Agreement obligations for the model provider. The practical reading: the model inside the device is governed by clearance, and the model calls your software makes over HTTP are governed by your quality system, your HIPAA obligations, and any post-market traceability you can produce. The audit trail for those calls is the piece that is usually missing.

Is on-device model inference covered by an AI gateway?

No. On-device inference runs inside the medical device, on the device's own compute, and never crosses a network boundary a proxy could inspect. That path is governed by the FDA clearance the manufacturer already holds, along with the verification and validation records behind it. An AI gateway sits on the HTTP path between software and an external model API, so it can only govern calls that travel over that path. The distinction matters when a team maps its controls. The model baked into a cleared insulin dosing calculator or an imaging triage tool is outside the gateway's scope. The calls that the same hospital's scheduling assistant, ambient scribe, or intake service make to a hosted LLM are inside it. Identity-aware policy and a per-decision record apply to the second group, not the first. Placing a proxy in front of on-device inference is a category error.

What records does a Predetermined Change Control Plan expect?

A Predetermined Change Control Plan is the mechanism the FDA offers so a manufacturer can update an AI model after clearance without a new submission for every change. The plan describes the modifications the manufacturer anticipates, the methods used to develop and validate them, and the limits that keep the model inside its authorized performance envelope. That structure carries an implicit records duty. To show a regulator that a deployed change stayed inside the plan, the manufacturer needs evidence of which model version was active, when it changed, and how it behaved against real inputs. For a model reached over an HTTP API rather than embedded in firmware, that evidence lives at the call boundary. A per-decision record that captures the model version and policy in effect at each call gives the change-control story a factual spine instead of an assertion.

How does a per-decision audit record help during an FDA investigation?

An FDA post-market investigation asks what the software did with specific patients and specific data. A per-decision audit record answers at the granularity of a single model call. Each record ties a request to a verified identity, the role in effect, the data classification applied to the prompt, the policy version that governed the call, the outcome, and a timestamp precise enough to correlate across systems. Because the record is signed and tamper-evident, it stands as evidence rather than as a convenience log the application could edit after the fact. During an investigation, the manufacturer can produce the exact set of calls that touched a patient record, name who initiated each one, and show the policy that applied. That reconstruction is what post-market surveillance and the quality system expect. Without it, the manufacturer is left describing intended behavior instead of documented behavior.

Do we still need a BAA if the audit trail is in place?

Yes. A Business Associate Agreement is a HIPAA contractual requirement between a covered entity and any vendor that handles protected health information on its behalf. An audit trail records what happened at each call. A Business Associate Agreement is the legal instrument that binds the model provider to safeguard the data and report breaches, and it has to exist before any PHI is sent. Neither one substitutes for the other. The stronger position combines both: a signed BAA with any provider that will process PHI, and redaction at the AI call boundary so that PHI which does not need to leave never reaches the provider in the first place. Cloud Radix found 57% of healthcare professionals process PHI through unauthorized tools with no BAA at all, which is the failure mode to close first. Redaction plus a per-decision record then shows, on demand, exactly what data crossed the boundary and under what policy.

How does FDA oversight line up with the EU AI Act for a device sold in both markets?

A manufacturer shipping an AI-enabled device into the United States and the European Union answers to two regimes that converge on the same evidence. The FDA expects traceability, post-market surveillance, and records that reconstruct what the software did. The EU AI Act, whose high-risk requirements take effect August 2, 2026, classifies many medical AI systems as high-risk and applies Article 12, which mandates automatic event logging over the system's lifetime with enough detail to identify the natural persons involved. Both point at the AI call boundary for any function that reaches an external LLM. A single per-decision record, generated at that boundary, carries the identity, policy, classification, outcome, and timestamp that an FDA reviewer and an EU auditor each request. Building the evidence once, at the boundary, answers both regimes instead of maintaining two parallel logging schemes that drift apart over time.

← Blog

July 19, 2026

Medical Device AI and FDA Compliance

FDA oversight of AI-enabled medical device software keeps widening, yet the audit trail for the LLM calls clinical software makes over HTTP is usually missing. This walks through what FDA expects, where on-device inference sits outside the boundary, and how identity-bound policy plus per-decision records close the gap.

ByParminder Singh· Founder & CEO, DeepInspect Inc.

Industry Verticalsai-compliancecomplianceregulationai-governanceauditai-security

The FDA reviews artificial intelligence and machine learning Software as a Medical Device through the 510(k) and De Novo pathways, and it evaluates these products across their total product lifecycle rather than at a single moment of authorization. Its Good Machine Learning Practice principles and Predetermined Change Control Plan framework both assume the manufacturer can trace what the software did and produce evidence of it. The model running inside the cleared device is one surface. The LLM calls that clinical software, staff, and connected services make over HTTP to external model APIs are another, and that surface usually carries no audit trail at all.

I want to walk through what FDA oversight actually expects, where on-device inference sits outside the AI call boundary, and how identity-bound policy plus a per-decision record supplies the traceability that FDA quality systems and post-market monitoring look for.

FDA oversight now spans the total product lifecycle

The FDA regulates AI and machine learning software that meets the statutory definition of a medical device, the category it labels Software as a Medical Device. Its approach has shifted from a single premarket decision toward continuous oversight after a product ships. Three parts of that approach set the traceability bar for any manufacturer.

Total product lifecycle and post-market monitoring

The total product lifecycle framing treats a cleared AI device as something to watch in the field, not only before launch. Post-market surveillance expects the manufacturer to detect performance drift, report qualifying malfunctions, and show what the software actually did with real inputs. That holds only when the relevant events were recorded in a form a reviewer can reconstruct months later.

Predetermined Change Control Plans

A Predetermined Change Control Plan lets a manufacturer specify, in advance, how an AI model will change after clearance and how those changes stay inside tested limits. The plan carries an implicit records duty. To prove a modification stayed within the authorized envelope, you need a trail of which versions ran, when, and against which data.

Good Machine Learning Practice

The Good Machine Learning Practice guiding principles, issued by the FDA with Health Canada and the UK's MHRA, call for traceability across data handling, training, evaluation, and monitoring. Each principle assumes a durable record exists. Software that acts on clinical data without a per-event trail cannot demonstrate the practice the guidance describes.

The AI call boundary sits outside the cleared device

Two different things get called medical device AI, and they live at different layers. The first is the model that runs inside the cleared device or its bundled software, on-device inference that the FDA reviews as part of the submission. That path is closed. It is governed by the clearance itself and sits outside the reach of any network proxy, because the computation happens inside the device.

The second is the set of HTTP calls that clinical software, staff, and connected services make to external LLM APIs from OpenAI, Anthropic, Amazon Bedrock, Azure OpenAI, and Google Vertex. A radiology worklist tool that summarizes findings through a hosted model, a scheduling assistant that drafts patient messages, an intake service that structures a referral through an LLM: each crosses an AI call boundary over HTTP. That boundary is where identity-aware authorization and a per-decision record can attach.

Clinical LLM traffic over HTTP usually has no audit trail

That HTTP traffic is already heavy in clinical settings, and much of it is unmonitored. Cloud Radix reports that 57% of healthcare professionals use unauthorized AI tools to process protected health information, including SOAP notes and diagnostic plans, without a Business Associate Agreement in place. When a breach involves that shadow AI usage, IBM's Cost of a Data Breach research finds it takes 247 days to detect and costs $670,000 more than a comparable breach. Three structural gaps sit under those figures.

Identity resolves to a service account

Clinical software calls model APIs with a shared service credential or a single API key. The credential identifies the application, not the clinician or agent acting through it. When an auditor asks who sent a specific prompt containing patient data, the log answers with a service account name.

PHI reaches a model with no BAA

A SOAP note pasted into a general-purpose chatbot leaves the covered entity's control the moment the request is sent. Without a Business Associate Agreement and without redaction at the boundary, protected health information reaches a provider that never accepted HIPAA obligations for it. I covered that exposure in AI and HIPAA compliance.

No per-decision record exists

The application writes its own operational logs, when it writes any. Those entries note that a request was processed. They omit the identity, the data classification, and the policy that applied at the moment the model was called. An FDA reviewer reconstructing a post-market event finds gaps where the AI decisions should sit.

FDA traceability expectations map onto a per-decision record

FDA quality systems have expected traceability and records for decades. 21 CFR Part 11 requires computer-generated, time-stamped audit trails for electronic records in regulated systems, and design controls expect a record of what the software did and why. None of that language names LLM APIs, since it predates them, yet the expectation transfers directly to the AI call boundary.

Surviving an FDA inspection or a post-market investigation means answering pointed questions about specific events. Which model calls touched this patient's record? Who initiated each one? What sat in the prompt, and what data classification applied? What policy governed the call at the moment it was made? An application log written by the same software that placed the call cannot answer those questions, because the identity and policy context were never attached. A per-decision audit record generated at the boundary can, and it holds the same fields a reviewer asks for.

The EU AI Act classifies many medical AI systems as high-risk

FDA oversight is not the only regime pointing at this boundary. The EU AI Act treats medical devices and the AI safety components inside them as high-risk systems, which pulls them under Article 12. On August 2, 2026, the high-risk requirements take effect, and Article 12 mandates automatic recording of events over the system's lifetime, with logs detailed enough to identify risk situations and the natural persons involved.

A device maker selling into both the United States and the European Union meets the FDA's traceability expectations and Article 12's logging mandate at the same boundary. The evidence that answers an FDA post-market question, identity, policy, data classification, outcome, and timestamp for each call, is the evidence Article 12 asks for. I worked through the healthcare reading in EU AI Act for healthcare.

DeepInspect

This is the gap DeepInspect closes at the AI call boundary. DeepInspect is a stateless proxy that sits between authenticated users or agents and any LLM, including OpenAI, Anthropic, Amazon Bedrock, Azure OpenAI, Google Vertex, and self-hosted models. It does not touch on-device inference inside a cleared device, and it is not a model, a DLP appliance, or a guardrails library. It governs the HTTP calls that clinical software and staff make to model APIs.

For every request, DeepInspect resolves the identity behind the call, applies per-route and per-role policy, and detects or redacts protected health information before the prompt reaches the provider. Every decision produces a signed, tamper-evident audit record holding identity, role, data classification, policy version, outcome, and timestamp. The record commits before the response returns to the application, so the software that made the call cannot suppress it.

For an FDA post-market investigation or an EU AI Act Article 12 review, that record is the traceability evidence the quality system expects. A device maker routing clinical LLM traffic through the proxy can reconstruct which model calls touched a patient record, who initiated them, and what policy governed each one.

Book a demo today.

Frequently asked questions

Does the FDA regulate the LLM API calls my clinical software makes?: The FDA reviews the AI model cleared as part of a device submission, so on-device inference falls squarely inside its authority. The HTTP calls your clinical software makes to an external LLM API are a different matter. They are generally not the cleared device, yet they still land inside the FDA quality system when that software is part of a regulated product, because 21 CFR Part 11 and design-control expectations reach the electronic records the software produces. HIPAA also applies the moment protected health information enters a prompt, which pulls in Business Associate Agreement obligations for the model provider. The practical reading: the model inside the device is governed by clearance, and the model calls your software makes over HTTP are governed by your quality system, your HIPAA obligations, and any post-market traceability you can produce. The audit trail for those calls is the piece that is usually missing.
Is on-device model inference covered by an AI gateway?: No. On-device inference runs inside the medical device, on the device's own compute, and never crosses a network boundary a proxy could inspect. That path is governed by the FDA clearance the manufacturer already holds, along with the verification and validation records behind it. An AI gateway sits on the HTTP path between software and an external model API, so it can only govern calls that travel over that path. The distinction matters when a team maps its controls. The model baked into a cleared insulin dosing calculator or an imaging triage tool is outside the gateway's scope. The calls that the same hospital's scheduling assistant, ambient scribe, or intake service make to a hosted LLM are inside it. Identity-aware policy and a per-decision record apply to the second group, not the first. Placing a proxy in front of on-device inference is a category error.
What records does a Predetermined Change Control Plan expect?: A Predetermined Change Control Plan is the mechanism the FDA offers so a manufacturer can update an AI model after clearance without a new submission for every change. The plan describes the modifications the manufacturer anticipates, the methods used to develop and validate them, and the limits that keep the model inside its authorized performance envelope. That structure carries an implicit records duty. To show a regulator that a deployed change stayed inside the plan, the manufacturer needs evidence of which model version was active, when it changed, and how it behaved against real inputs. For a model reached over an HTTP API rather than embedded in firmware, that evidence lives at the call boundary. A per-decision record that captures the model version and policy in effect at each call gives the change-control story a factual spine instead of an assertion.
How does a per-decision audit record help during an FDA investigation?: An FDA post-market investigation asks what the software did with specific patients and specific data. A per-decision audit record answers at the granularity of a single model call. Each record ties a request to a verified identity, the role in effect, the data classification applied to the prompt, the policy version that governed the call, the outcome, and a timestamp precise enough to correlate across systems. Because the record is signed and tamper-evident, it stands as evidence rather than as a convenience log the application could edit after the fact. During an investigation, the manufacturer can produce the exact set of calls that touched a patient record, name who initiated each one, and show the policy that applied. That reconstruction is what post-market surveillance and the quality system expect. Without it, the manufacturer is left describing intended behavior instead of documented behavior.
Do we still need a BAA if the audit trail is in place?: Yes. A Business Associate Agreement is a HIPAA contractual requirement between a covered entity and any vendor that handles protected health information on its behalf. An audit trail records what happened at each call. A Business Associate Agreement is the legal instrument that binds the model provider to safeguard the data and report breaches, and it has to exist before any PHI is sent. Neither one substitutes for the other. The stronger position combines both: a signed BAA with any provider that will process PHI, and redaction at the AI call boundary so that PHI which does not need to leave never reaches the provider in the first place. Cloud Radix found 57% of healthcare professionals process PHI through unauthorized tools with no BAA at all, which is the failure mode to close first. Redaction plus a per-decision record then shows, on demand, exactly what data crossed the boundary and under what policy.
How does FDA oversight line up with the EU AI Act for a device sold in both markets?: A manufacturer shipping an AI-enabled device into the United States and the European Union answers to two regimes that converge on the same evidence. The FDA expects traceability, post-market surveillance, and records that reconstruct what the software did. The EU AI Act, whose high-risk requirements take effect August 2, 2026, classifies many medical AI systems as high-risk and applies Article 12, which mandates automatic event logging over the system's lifetime with enough detail to identify the natural persons involved. Both point at the AI call boundary for any function that reaches an external LLM. A single per-decision record, generated at that boundary, carries the identity, policy, classification, outcome, and timestamp that an FDA reviewer and an EU auditor each request. Building the evidence once, at the boundary, answers both regimes instead of maintaining two parallel logging schemes that drift apart over time.

← All posts