AI-Generated Malware Needs an Abuse-Control Test Pack

AI-generated malware is a request-control and incident-evidence problem before it becomes an endpoint incident. Anthropic's September 2026 threat report describes an actor using AI agents to monitor whether deployed malware was detected, then modify and rebuild it to evade existing detections. The model interaction supplies one controllable decision point. Compilation and execution happen later, often in systems owned by different teams. I would reject an abuse test that proves a model refusal while leaving the caller and authorization state blank. The route and output disposition also belong in the record, followed by the evidence reference and escalation record.

TL;DR

• Build synthetic abuse cases around intent and authorization state. Keep executable malware and real targets out of the test corpus, along with credentials and evasion instructions.
• Preserve caller and target ownership, plus the model destination. Record prompt and output classifications, then the policy verdict and escalation. Add the reviewer and evidence for every case.
• Test malicious requests beside authorized security work and ambiguous dual-use requests. A blanket keyword block produces weak security and noisy operations.
• Keep compilation and execution outside the LLM gateway claim. Delivery, endpoint behavior, C2, persistence, containment, and attribution also sit outside it.

The controllable point is the model request

Anthropic's Acceptable Use Policy prohibits creating or distributing malware and ransomware, along with other malicious code. It also preserves a boundary for security research and testing, including tool development on owned or operated systems. Owner authorization, an authorized bug bounty, or a vulnerability disclosure program can establish that boundary within the policy's conditions.

That distinction belongs in the control. A request for code that steals browser credentials and sends them to an external host carries a different authorization profile from a request to explain a detection rule against a synthetic sample in an owned lab. A keyword list cannot adjudicate ownership and scope, much less permitted purpose. The policy decision needs the authenticated caller and the declared research authorization, followed by the target and destination.

The model route is only one stage. A blocked request never reaches the model through that route. An allowed response may still require quarantine or human review. The AI security incident response process should receive the same request ID and classification rather than a free-text alert with no link to the original call.

Build a safe test corpus

Use descriptions and inert placeholders instead of executable payloads. A fixture can state that the user requests credential theft or persistence. Other fixtures can name destructive behavior, payload delivery, or adaptation intended to evade security detection. It can name a fictional host such as lab-host-17.invalid and a marker such as ORANGE-CANARY-52. The fixture should never contain working malware or live infrastructure. Real credentials and procedural evasion detail are also excluded.

Create cases across four decision classes:

• Prohibited malicious intent: an authenticated user asks for malicious code or modification intended for unauthorized access and theft, or for disruption, persistence, or evasion.
• Authorized research with complete context: the caller supplies an approved lab or bug-bounty identifier and an owned target. The record also contains permitted activity, time window, reviewer, and evidence owner.
• Authorized claim with missing proof: the text says "security research," but the route lacks a valid authorization record or the requested activity exceeds its scope.
• Ambiguous dual use: the request discusses code or security behavior without enough context to decide safely, so the route pauses for review or applies a constrained response policy.

I prefer ten well-labeled cases that exercise these boundaries over a thousand copied jailbreak prompts. High volume can hide bad labels. The AI red-team methodology can expand the corpus after the policy owner has proved that the first labels and expected decisions are defensible.

Define the abuse-control record

Every case needs a stable fixture ID and an expected result. At runtime, join that fixture to the actual policy event. Preserve these fields:

• Authenticated user or agent identity and role, plus the organization and credential route
• Authorization record and target owner, followed by permitted activity, scope, evidence source, and expiry
• Model provider and endpoint, plus the model, application, release, and destination
• Prompt classification and requested behavior, with a confidence or review state
• Policy version and verdict, followed by applied response treatment and reason code
• Output classification and disposition, including block, quarantine, limited return, or review
• Request ID and timestamps, plus evidence locations and the retention owner
• Escalation owner and incident ID, with the reviewer decision and closure state

The reason code should identify the failed condition. "Malware" is too broad for an operational queue. Use a controlled reason-code taxonomy. Candidate codes include unauthorized malicious-code generation and evasion intent. Add target ownership missing, authorization expired, scope exceeded, and ambiguous dual use where they match. Preserve the original content under restricted access according to the organization's evidence rules.

A clean artifact resembles a case file rather than a prompt spreadsheet. On one screen, the reviewer should see the caller and target authorization. The same view should show the model route, classifications, verdict, evidence, and escalation. The dangerous content can remain masked while hashes and controlled references preserve the join.

Test prompt and output handling separately

Prompt policy should reject explicit unauthorized malicious-code requests before they reach the model. It should also catch requests that frame evasion or credential theft as a rewrite task. Test authorized research with both valid and expired authorization. Add one case where the caller has approval for a lab target but asks for behavior outside the permitted scope.

Output handling needs separate fixtures because a benign-looking prompt can produce suspicious code or operational instructions. Use prebuilt inert responses labeled by the security team. Confirm that the response classifier blocks or quarantines the expected classes and preserves a reference for review. Do not ask a live model to generate the dangerous fixture during the test.

The expected result may include human review. Define the queue and service owner before the run. Add severity and a response deadline. Then test the evidence handoff. An alert that reaches a general SOC mailbox without the route and authorization record has failed even if the initial classifier was correct.

Use the Anthropic case as a control pattern

Anthropic's September 2026 threat intelligence report describes AI agents monitoring whether tools evaded known security defenses. When security products detected deployed malware, agents modified and rebuilt it to evade those detections, continuing the cycle until the tools were undetected.

Translate that report into non-operational test intents. The corpus should contain a request whose declared goal is modification after detection, with no code or detector details. Another case should ask for repeated rebuilding tied to a detection result. Expected controls can classify the evasion intent, block the model route, preserve the caller and destination, and escalate the event. This tests the policy mechanism without reproducing the actor's technique.

The report supports a wider incident question as well. Repeated requests with shared targets and reason codes may belong to one incident, especially when authorization failures recur. Correlation happens in the SIEM or incident platform. The gateway supplies stable events, while the response team decides containment and attribution.

Keep endpoint and malware controls with their owners

The abuse-control test pack should name every control owner around the model route. Endpoint protection owns process and file behavior. Network controls own payload delivery, egress, and C2 observations. Sandboxes and malware-analysis teams own safe execution and reverse engineering. Identity teams own compromised accounts and credential response. Incident responders own containment and attribution.

An LLM request log cannot prove that generated code was never compiled. It also cannot observe persistence, a process tree, or a later payload retrieved from another system. Join those records by incident and identity where possible, but preserve the scope of each source. Automated red teaming can test model behavior at scale. It still needs an authorized target and a separate production control owner.

Routed prompts and responses can be classified and blocked. They can also be quarantined, escalated, correlated, and recorded. Endpoint and incident-response systems own the evidence after code leaves the model route, including execution, delivery, persistence, containment, and attribution.

DeepInspect

DeepInspect can govern authenticated user or agent requests deliberately routed over HTTP to an LLM. For the AI-generated malware test pack, it can evaluate caller identity and authorization context, classify the prompt and response, apply a route policy, block or quarantine covered content, and create a per-decision record with the model destination and policy version.

DeepInspect does not compile or execute generated code. It does not inspect endpoint processes, payload delivery, C2, persistence, or later network egress outside the routed model call. Malware analysis and containment stay with their named systems and teams. Attribution, account response, endpoint recovery, and downstream authorization do too. The DeepInspect event becomes one evidence source in that larger incident record. Let's talk today.