RAG Security Checklist: The Pre-Release Evidence Pack

A RAG release can pass answer-quality evaluation while carrying another tenant's document into the model. The RAG security checklist below produces a pre-release evidence pack for the exact corpus and index, retrieval policy and prompt assembly, plus the response path and deletion behavior going live. It complements the RAG security architecture, which explains the trust boundaries. This checklist asks a narrower release question: what evidence proves those boundaries work in this build?

TL;DR

• Tie the evidence pack to one release and corpus snapshot, the index version and retrieval policy, plus the model route and owner.
• Prove source provenance and permission-aware retrieval. Test tenant isolation and assembled-prompt inspection, then verify safe response handling with repeatable tests.
• Run deletion through the source and index, then check the cache and prompt trace before checking derived stores. Record completion or an approved exception.
• Reconstruct one allowed event and one denied event before signing the release disposition.

Check 1: identify every ingestion source and transformation

The corpus owner should list each source repository and connector, the owner and content type, plus the authorization basis and classification. Record the ingestion credential separately. Record extraction and chunking, redaction and enrichment, plus embedding and deduplication. Each indexed chunk needs a link to its source version.

NIST AI 600-1 recommends documenting data provenance and testing data and content flows, including original sources and transformations. OWASP's Top 10 for LLM and GenAI applications also names data and model poisoning plus vector and embedding weaknesses. The evidence pack should include the source manifest, transformation version, rejected-content log, and a sample trace from source file to chunk.

Check 2: prove permission-aware retrieval

Retrieval must carry the authenticated principal and apply source-derived permissions before ranking returns context. The test set needs two users with different entitlements and several documents they share. Include at least one document available to only one user. Put a visible canary string in that restricted document.

Run identical queries as both users. The authorized user should retrieve the canary. The other result set must omit it before prompt assembly. Save the query identity and filters, candidate identifiers and ranked results, plus the policy version. The RAG poisoning prevention guide addresses malicious source content; this check focuses on entitlement enforcement.

Check 3: test tenant isolation under collision and failure

The platform owner should prove that tenant identity scopes storage and retrieval, caches and evaluation traces, plus operational access. Insert the same sentence into two tenant corpora and add a different canary to each document. Query with equivalent language.

Evidence should show index isolation, tenant-bound filters, cache keys, and administrative access. Repeat the test with missing tenant context, then fail the authorization service. The system should stop before retrieval. A release stays blocked when missing context broadens the search namespace.

Check 4: inspect the assembled prompt before transmission

The application team must capture the message payload after retrieved chunks, conversation history, system instructions, and tool context are assembled. A request-layer policy evaluates that payload before it reaches the model. Earlier scans miss content introduced by a fresh sync or retrieval error.

NIST's Generative AI Profile describes indirect prompt injection delivered through data likely to be retrieved. Test with a document containing an instruction to reveal hidden context, plus synthetic regulated data in a separate chunk. Save the assembled request and classification, the policy result and treatment, plus the destination and proof that blocked content never reached the provider. The RAG context redaction pattern covers this runtime control in detail.

Check 5: validate response handling at every sink

Treat model output as untrusted input. The application security owner should list each sink, including rendering, exports, database fields, and tool arguments. Give every sink an encoding, schema, content, and authorization rule.

The evidence set should include malformed structured output, active markup, sensitive canary text, and a response that invokes a tool beyond the user's authority. Record transformation or rejection before rendering. A clean provider response code says nothing about safe handling downstream. The release proof comes from sink behavior.

Check 6: demonstrate deletion across derived stores

The data owner defines deletion scope and completion target. Remove a test document at its source, then verify its chunks and embeddings leave the active index. Check caches and evaluation datasets, prompt traces and response stores, plus backups and derived analytics.

Query the deleted document's canary before deletion and after the active-index update. Query it again after cache expiry. Preserve timestamps and results. Where backups follow delayed expiry, document the schedule and access restriction, restoration treatment and owner, plus the legal basis. A "deleted from source" ticket leaves RAG copies untested.

Check 7: reconstruct an allowed and denied request

The security operations owner should reconstruct two events without asking the development team to explain them. For the allowed event, identify the caller and tenant, query and retrieved source identifiers, assembled-prompt classification and destination, policy version and response treatment, plus the time. For the denied event, show the same chain through the blocking decision.

NIST SP 800-53 Rev. 5 covers event logging and audit-record content in AU-2 and AU-3 in its control catalog. The evidence should live outside the application's sole write control. Picture the review screen with one timeline and seven linked records. Gaps appear as blank intervals, which is exactly what the test should expose.

Check 8: sign the release disposition and exceptions

The release owner signs one disposition tied to the application build and corpus snapshot, index version and retrieval policy, model routes and test results, plus the evidence links. Use go or no-go. When conditions apply, use go with conditions. Every condition needs scope and consequence, a compensating measure and owner, plus an expiry and closure test.

I would stop a RAG release over an untraceable corpus source or a tenant filter that defaults to broad retrieval. Answer quality can wait for another build. Cross-tenant disclosure cannot be pulled back after the response reaches a user. Material changes to sources and permissions, index design and prompt assembly, model routes and sinks, or retention should reopen the checklist.

DeepInspect

DeepInspect covers the assembled-prompt and model-response portion of the RAG evidence pack for managed HTTP traffic. It uses application-supplied identity and context, classifies the complete outbound request, applies content and destination policy, and records any permit, redaction, reroute, or block before the request reaches the model.

The resulting signed records can support prompt-policy tests and event reconstruction. Ingestion provenance and source permissions, tenant isolation and index deletion, plus cache design and sink authorization remain with the systems that own those controls. Book a technical deep dive at deepinspect.ai.