API Key Rotation for LLM Systems: A Runbook That Avoids Downtime
LLM API key rotation is a controlled credential replacement, not a console click. The runbook starts with inventory, creates a restricted replacement, deploys it through a dual-key window, validates every workload, revokes the old key, and preserves a tested rollback path. This guide also separates provider credential controls from HTTP model-request policy and states the stolen-key boundary plainly.
An LLM key rotation fails when the team revokes a credential before finding the notebook or batch job that still uses it. Vendor connectors and fallback workers belong in the same inventory. The provider console shows one credential. Production may contain a dozen copies with different owners and restart behavior.
Treat rotation with the same discipline as a release. The change needs an inventory and a replacement credential. It also needs a bounded overlap window and workload validation. Revocation and rollback complete the change. I would rather see a grease-pencil diagram with every consumer crossed off than a green change ticket that says "secret updated" and proves nothing about live traffic.
TL;DR
- Inventory every workload and environment before creating the replacement. Record each secret location and owner, plus the provider project and downstream model route.
- Use a bounded dual-credential window: create and restrict the new credential, deploy it, confirm traffic on every consumer, then revoke the old credential.
- Validate successful and denied calls. Check usage attribution, cost controls, logs, and fallback behavior before revocation.
- DeepInspect governs only model traffic routed through its HTTP proxy. It cannot stop stolen-credential calls made directly outside that path.
Build the credential inventory first
Start with the provider project and list each active credential. For every credential, record its human-readable label and creation date. Add the expiration, owner, environment, and allowed services. Record all known consumers alongside the secret-store path and deployment mechanism. Include CI jobs and scheduled notebooks. Add evaluation runners, agents, vendor integrations, and developer sandboxes. Include emergency scripts and disaster-recovery environments.
The repository is only one discovery surface. Search deployment manifests and secret-manager access logs. Inspect CI variables and job definitions. Review provider usage by credential where available. OpenAI says credential usage can be monitored on its Usage page after tracking is enabled, while credentials created before December 20, 2023 may require tracking to be enabled. That distinction can explain an "untracked" bucket during inventory.
The existing AI API key sprawl guide explains why owner and purpose belong beside the secret. A credential that appears in five systems has five deployment dependencies, even if the provider dashboard shows one object. Stop the rotation when any production consumer has no owner or update path.
Create a restricted replacement
The official OpenAI production best-practices guide recommends expiration dates for project credentials and a regular rotation process. It gives the core order explicitly: create a replacement before expiration, update applications, verify the replacement, and revoke the old credential. OpenAI also documents maximum credential lifetimes at organization or project level and credential-governance controls for newly created credentials.
Create the replacement in the correct project and environment. Apply the narrowest provider permissions and spend controls available for that workload. Keep staging and production credentials separate. Store the secret directly in the approved secret manager. Do not copy it through chat or a ticket, and do not save it in a local text file.
Google's API key management best practices add useful adjacent controls. Google recommends restrictions that reduce the impact of compromise and individual application isolation. It also recommends monitoring, periodic rotation, and deletion of old credentials. It also recommends keeping credentials out of client code and repositories. Apply the same operating discipline to model provider credentials even when the provider exposes different restriction fields.
Use a bounded dual-credential rollout
A safe rotation has a short period when both credentials remain valid. That overlap allows workloads to move without converting a missed consumer into an outage. Record the start and planned end time before deployment. A dual-credential window without an expiry becomes permanent credential growth.
Deploy the replacement through the normal configuration path. Services that read secrets at startup may require a rolling restart. Long-running workers may cache the old value. Serverless workloads may retain warm instances. Notebooks and vendor connectors may need manual updates. Record the deployment version or configuration revision for each consumer.
Move lower-risk consumers first, then the production path. If the provider exposes usage by credential, watch calls migrate to the replacement. If it does not, add a temporary internal deployment marker to application telemetry without logging the secret. Never place the credential value or a reversible fragment in logs.
The AI agent secrets handling guide covers secret retrieval for agent workloads. Rotation should preserve the same rule: the workload reads the credential at runtime from an approved source and never receives authority beyond its job.
Validate the replacement before revocation
Validation needs more than one successful completion. Send a production-like request through every application route and confirm the expected model and project. Check streaming and non-streaming calls if both exist. Exercise batch processing and retries. Test fallback workers, evaluation jobs, and scheduled tasks. Confirm rate and spend controls under the new credential.
Run a denied test as well. A staging credential should fail against a production-only path. A consumer outside the approved project should fail. Confirm that provider and application logs identify the replacement credential's label or tracked identity without exposing its value. Compare request counts during the overlap window against the expected workload inventory.
Watch for old-credential usage until every known consumer has moved and a full workload cycle has passed. A monthly batch requires a different observation window from a continuously running API. The rotation owner should sign the evidence with the consumer name and deployed revision. Add the test request ID and result. Record the first new-credential use and last old-credential use.
Revoke and prove the old credential is dead
Revoke the old credential only after validation closes. Record the provider event time and the administrator who performed it. Then run one controlled request with the old credential from an isolated validation client. The expected result is an authentication failure. Do not run that test from a production service that could trigger retry storms.
Monitor provider usage and application authentication errors after revocation. An old credential that continues to generate accepted calls indicates a provider or project mistake. A spike in rejected calls identifies a missed consumer or leaked copy. Keep the incident threshold and owner in the runbook before the change begins.
Google says to retain only actively used credentials and delete unneeded ones to reduce exposure. OpenAI says to revoke the old credential after the replacement works. Disabling a credential while leaving undocumented copies scattered across environments may be necessary for investigation, but it is not a completed hygiene outcome.
Rollback without reviving the wrong risk
Rollback during the overlap window means restoring the previous secret reference or deployment revision while the old credential still works. Define the command and approver in advance, then test the procedure in staging. The rollback should restore service without editing source code or pasting credentials into runtime consoles.
Rollback changes after revocation because re-enabling the old credential may recreate the exposure that prompted an emergency rotation. Prefer a newly issued emergency credential with narrow scope and short expiration. Give it explicit incident ownership. Deploy it through the same secret path and validate the minimum service route. Replace it again after the incident closes.
A routine rotation and a compromise response should share mechanics but use different risk decisions. Routine work can allow a measured overlap. Confirmed theft may require immediate revocation and controlled degradation. Keep the business owner involved when service availability conflicts with exposure containment.
The AI gateway rollback strategy covers policy and routing changes at a separate control point. Credential rollback restores provider authentication. Gateway rollback restores a request policy or route. Record both changes when one incident touches both layers.
Schedule and evidence the next rotation
Set the next expiration and rotation owner when the replacement is created. Provider-enforced maximum lifetime can prevent indefinite credentials, but it cannot update every consumer for you. Put rotation lead time ahead of expiration so the dual-credential window closes during staffed hours.
Retain a compact evidence pack with the inventory snapshot and approvals. Include the new-credential settings and secret deployment revisions. Add validation request IDs and the usage transition. Record the old-credential revocation and denied proof. Include the monitoring window and exceptions. The pack must exclude the secret itself. It should let another engineer reconstruct the change without opening a private chat history.
My opinion is that rotation frequency receives too much attention and inventory quality receives too little. A 30-day schedule repeated against an incomplete consumer list creates 12 opportunities for an outage while leaving the forgotten credential untouched. The inventory and last-use evidence determine whether the rotation changed the attack surface.
DeepInspect
DeepInspect evaluates authenticated user or agent HTTP requests that applications deliberately route through its proxy. It can apply identity-aware model policy and produce a per-decision record for those calls. That record helps validate which routed workloads continued to operate during a credential change.
DeepInspect cannot prevent a stolen credential from calling the provider directly outside the routed path. It also does not replace provider-side restrictions or secret storage. Repository scanning and credential creation remain separate controls, as do revocation and compromise response. The identity-aware AI gateway architecture explains how application identity and request policy apply after credential management has established the model connection.
Book a technical deep dive at deepinspect.ai.
Frequently asked questions
- How long should the dual-credential window remain open?
Keep it long enough to exercise every consumer once and short enough to prevent overlap becoming normal. A continuous API may need hours. A scheduled batch may need one complete cycle. Set the deadline before creating the replacement and require an exception for any extension.
- Should applications support two credentials at the same time?
The provider can usually accept both credentials during the overlap. Each application should use one active secret reference at a time, with deployment rollback available if the replacement fails. Avoid custom logic that alternates credentials silently because it obscures migration evidence and can keep the old credential alive.
- What changes during an emergency rotation?
Confirmed exposure can compress or remove the overlap window. Revoke the affected credential and accept controlled service impact where necessary. Issue a narrow emergency replacement and monitor direct provider usage. Preserve the leaked-credential identifier and relevant logs for investigation without retaining the secret in the incident ticket.
- Can an AI gateway rotate provider credentials?
Some gateways can hold or reference provider credentials, but the provider remains the authority that creates, restricts, expires, and revokes them. A gateway deployment still needs secret-manager integration and a tested reload path. Rotation evidence should distinguish the provider event from the gateway configuration change.