NVIDIA NIM Security: What an NGC API Key Authenticates and What It Does Not
An NGC API credential pulls NIM container images from nvcr.io and authenticates calls to NVIDIA-hosted endpoints. It works at the registry and account level rather than as a per-request authorization mechanism. Once a NIM container is running in your own cluster, access control belongs entirely to the deployment platform, and platforms including NVIDIA Run:ai default to public access with no authentication on the inference endpoint.
The NGC API key does two jobs, and neither one is the job most teams assume. NVIDIA's documentation describes it as the credential you need "whether you are interacting with NIM through the Hosted API Endpoints, or pulling NIM from the NVIDIA NGC catalog to host locally or in cloud infrastructure," noting that "the API key is unique to you and tied to your account." I want to walk through what that credential secures, what happens the moment a NIM container starts serving inside your own cluster, and where authorization has to live instead.
TL;DR
- An NGC API credential authenticates image pulls from
nvcr.ioand calls to NVIDIA-hosted endpoints. It is tied to an NGC account, not to an end user of your application. - Personal credentials support expiration dates, revocation, deletion, and rotation from the NGC console, which makes them manageable but still account-scoped.
- A self-hosted NIM container has no built-in authentication on its inference endpoint. Access control is a property of the deployment platform, and NVIDIA Run:ai defaults to public access with no authentication.
- The NVIDIA RAG blueprint resolves credentials in the order service-specific value, then
NVIDIA_API_KEY, thenNGC_API_KEY, then none, which is a fallback chain worth auditing in a running deployment.
What the credential actually authenticates
Two distinct flows use the same credential. Pulling nvcr.io images into a Kubernetes cluster requires NGC credentials, and NVIDIA notes that for a Kubernetes deployment "the cluster must still pull NIM and microservice images from nvcr.io and may need NGC API access," with the Helm chart expecting values derived from a personal NGC token. Calling NVIDIA-hosted NIM endpoints uses the same credential as a bearer token.
NVIDIA's own guidance on handling it is blunt: "it is important to keep your key secret and in a safe place. Do not share it or store it in a place where others can see or copy it." Personal credentials support configuring an expiration date, revoking or deleting one with an action button, and rotating it as needed.
Expiry and rotation are real controls and they are worth using. What the credential cannot express is which human, in which role, is behind a given inference request, because it was issued to an NGC account and the account is a supplier relationship rather than an employee directory.
Self-hosted NIM ships without endpoint authentication
This is the fact that reorders a threat model. With self-hosted NIM, the operating team owns the container, the model, the GPU capacity, the network, and the observability. Authentication on the inference endpoint is not part of the container's job.
NVIDIA's Run:ai documentation makes the default explicit for inference workloads: public access is the default selection, "giving everyone within the network access to the endpoint with no authentication." The platform does offer alternatives, where "authentication is performed either by a user (with a username/password) or by a user application (with client credentials), and authorization to access the endpoint is enforced based on user or group membership," along with restricting access to specific authenticated users, service accounts, groups, or individuals.
Those alternatives exist and have to be chosen. A NIM endpoint stood up quickly for an evaluation, left on the default, and then quietly promoted into a production path is reachable by anything on the network that can resolve its address. The cluster's network policy becomes the entire access control story.
A call to that endpoint is a POST to the chat completions path on an in-cluster service address such as nim-llama-3-1-70b.ml-platform.svc.cluster.local:8000, carrying a model name and a message array. There is no Authorization header on that request and no identity anywhere in it. A pod in a neighbouring namespace, a compromised sidecar, or a developer's port-forward all produce the same request.
The credential fallback chain deserves an audit
The NVIDIA RAG blueprint supports service-specific API credentials for fine-grained control across components, and documents a resolution order: the service-specific credential, then NVIDIA_API_KEY, then NGC_API_KEY, then none.
Fallback chains are convenient during development and they hide state in production. A service intended to run under a narrowly scoped credential silently falls back to the broad account one if the specific variable is unset in a deployment manifest, and nothing surfaces the downgrade. Checking which credential each component resolved at runtime, rather than which one the manifest was supposed to set, is a five-minute exercise that has found real problems.
What is left unprotected on the request
Assume the best case: a NIM deployment behind Run:ai with authenticated access restricted to a named service account. That configuration answers whether a caller may reach the endpoint. It answers nothing about the content of the call.
A service account authorized to invoke the model is authorized for every request it makes. The prompt that carries a routine internal question and the prompt that carries an export-controlled design document are indistinguishable to the endpoint, because the authorization decision was made once, at connection time, against the caller rather than the payload.
That is the post-authentication gap, and it is why self-hosted inference tends to produce a thinner evidence trail than managed platforms. The telemetry position is covered in what NIM records by default, and the data-classification side in NVIDIA NIM DLP. The general problem of credentials standing in for people is described in non-human identity for AI agents.
Where the control has to sit
A NIM container serves HTTP. Everything upstream of the container is where identity, classification, and policy can be evaluated, which makes the placement question straightforward: something has to terminate the request, verify the caller against the identity provider, inspect what the prompt carries, decide against a versioned policy, and only then forward to the inference server.
Network policy narrows who can reach the port. NGC credential hygiene bounds who can pull images and call hosted endpoints. Run:ai's authenticated access modes bind an endpoint to a service account. None of those three inspects a request, and all three are worth having anyway.
DeepInspect
This is exactly what DeepInspect does. DeepInspect is a stateless proxy that sits inline in front of HTTP LLM endpoints, including self-hosted NIM containers running in a Kubernetes cluster, so the request is evaluated before the inference server ever receives it.
Every call carries an identity check against the enterprise identity provider, a classification pass over the prompt content, and a policy decision that allows or blocks, producing a signed record per decision that correlates against the X-Request-Id NIM forwards internally. NGC credentials keep governing image pulls and hosted-endpoint access, and the deployment platform keeps deciding which workloads may reach which service. DeepInspect adds the per-request, identity-bound authorization the container itself was never built to perform. Book a technical deep dive at deepinspect.ai.