Granite Guardian: Govern Remote Risk Classification over HTTP
Granite Guardian can judge prompts, model responses, RAG groundedness, function calls, and custom criteria. This guide defines a production pattern for calling a remotely hosted Guardian endpoint through authenticated HTTP, testing its yes or no verdicts, and keeping the original model route under separate policy. Local or in-process Guardian inference sits outside this scope.

Granite Guardian judges text against a stated criterion and returns a yes or no score. IBM documents built-in criteria for harmful content and jailbreak attempts, along with RAG quality and function-calling hallucinations. It also accepts custom criteria written in natural language. For this guide, Granite Guardian runs behind a remote endpoint. The application reaches it through authenticated HTTP, and that classification call receives its own policy and audit record. Local model loading and in-process inference remain outside scope.
I would use the Guardian verdict as one input to a release policy. A green score without the tested criterion and model version, plus the caller identity and original content reference, leaves an attractive dashboard tile and a poor investigation record.
TL;DR
- Granite Guardian judges prompts and responses, including RAG evidence and function calls, against built-in or custom criteria.
- Production use here means remote classification through authenticated HTTP. Local and in-process inference are outside scope.
- Test each criterion against fixed allowed and disallowed cases. Record the model version and route, criterion and score, result, and caller.
- Keep Guardian classification separate from authorization on the original LLM request and any later tool action.
Granite Guardian provides a configurable judging model
The Granite Guardian model documentation describes a family of models and LoRA adapters that judge input or output against specified criteria. Granite Guardian 4.1 supports built-in definitions and Bring Your Own Criteria (BYOC). IBM's documented built-in areas include harm, social bias, jailbreaking, violence, profanity, sexual content, unethical behavior, RAG context relevance, groundedness, answer relevance, and function-calling hallucination.
IBM defines two operating modes. Think mode emits a reasoning trace before the score. No-think mode returns the score without an explanation and is positioned for lower-latency evaluation. Both modes use a structured Guardian block that states the criterion and a scoring schema. The documented schema asks for yes when the evaluated text meets the criterion and no otherwise. A production adapter needs to preserve the semantic direction of that question. A criterion that describes risk can make yes mean risk found, while a criterion that describes compliance can make yes mean the requirement passed.
Remote classification needs an explicit HTTP contract
IBM's risk-detection recipe describes screening user prompts and LLM responses for harmful content. Its hosted recipe names a watsonx API key, project ID, and URL among the required credentials. That provides a clear remote-service shape: the application authenticates to an endpoint, submits content for classification, and receives a result.
Write that exchange down as an API contract before connecting it to a blocking policy. The request record should identify the authenticated caller, Guardian route, deployed model version, criterion identifier, evaluation mode, and a correlation ID for the original LLM transaction. The result record should preserve the returned score and the action taken by the calling policy. Sensitive prompt or response text may require a reference or digest instead of another stored copy, according to the application's data rules.
The blue box on the architecture diagram should contain the application and its authenticated principal. Put the remote Guardian endpoint in a separate box, with the HTTP route drawn between them.
Criteria require tests before they can block traffic
IBM states that custom risk definitions require testing. Its scope notes also warn that the model can produce unexpected or harmful output when used outside the prescribed scoring mode, may be affected by adversarial attacks, and has been trained and tested on English data. IBM separately warns that reasoning traces may contain unsafe content and may be unfaithful.
Those limits should become acceptance cases. For each built-in or custom criterion, keep a fixed set of content expected to score yes and content expected to score no. Include borderline cases and adversarial wording. Run the set against the exact remote model version and prompt template used in production. Record disagreements for review instead of changing the threshold silently. The wider prompt injection detection guide explains how to retain a stable attack corpus while model and policy versions change.
I would start with no-think mode for an enforcement route unless an investigator has a defined use for reasoning traces. IBM positions that mode for lower latency, and avoiding a stored trace reduces the amount of generated material that needs handling. That is an engineering preference, subject to measured results for the selected criterion.
Guardian scores and authorization remain separate decisions
A Guardian score answers the criterion placed in its judging prompt. Authorization asks whether this authenticated user or agent may send this content to this model route under the current policy. Those records can be correlated, but they have different inputs and owners.
Consider a prompt that passes a harmful-content check but contains data restricted to a finance role. The Guardian result says nothing about the employee's role unless the criterion explicitly asks about that supplied context. Identity-aware authorization still applies to the LLM request. After inference, a function-call hallucination score can flag a malformed or inconsistent call, while the tool endpoint must enforce its own permissions before changing a record. The AI agent security guide covers that wider split between model traffic and action authorization.
Use separate reason codes for classification and authorization. A reviewer should be able to see Guardian's named score and the policy outcome: permit, block, or review. Folding both steps into a single safe field destroys the evidence needed to tune the criterion without rewriting access policy.
Failure handling belongs in the route policy
A remote classifier adds network and service failure modes. Define the policy response for an authentication failure or timeout. Handle a malformed score and unavailable model version as separate cases, alongside a criterion mismatch. A high-sensitivity route may block when classification is unavailable. A lower-risk route may queue the request for review. The chosen response belongs in versioned policy and should be exercised during acceptance testing.
Test the bypass path as carefully as the success path. The application should have one approved remote Guardian destination, and the production caller should use the authenticated route that creates the decision record. A direct call to another Guardian endpoint escapes that route.
The IBM documentation site currently carries a banner saying that it is no longer updated and directs readers to IBM's Hugging Face and GitHub collections for current models. Pin the deployed model identifier in the route inventory and recheck its current model card before a version change. This article relies on the two IBM documentation pages for the behavior described here; deployment approval still needs evidence from the exact hosted version.
Production acceptance record
A release review for remote Granite Guardian classification should contain these artifacts:
- Route identity: authenticated calling principal, approved Guardian URL, project or tenant context, and credential owner.
- Evaluation definition: deployed model identifier, criterion text or version, think or no-think mode, and score semantics.
- Test evidence: fixed positive, negative, borderline, and adversarial cases with expected and actual results.
- Policy action: the rule that maps a Guardian score or service failure to permit, block, or review.
- Correlation: the identifier joining the classification request to the original LLM request and its eventual outcome.
- Exclusions: local inference, in-process calls, direct bypass traffic, and downstream tool authorization with named owners.
Put one failed request in the review packet. An expired credential or unreachable classifier reveals more about the production design than another screenshot of a successful yes score. The record should show which policy branch ran and which event reached the audit store.
DeepInspect
DeepInspect covers the remote Guardian classification call only when an authenticated user or agent reaches the Guardian endpoint through a deliberately governed HTTP route. The application supplies caller and role context. DeepInspect can evaluate the destination, content classification, and route policy before forwarding the request, then record the decision with the Guardian call's correlation ID.
The original LLM request needs its own routed policy decision. Tool authorization, local Guardian execution, in-process inference, model training, reasoning-trace handling, and direct bypass traffic retain separate owners.
Book a technical deep dive at deepinspect.ai.
Frequently asked questions
- Can Granite Guardian inspect both prompts and model responses?
IBM's risk-detection recipe says Granite Guardian can screen user prompts and LLM responses for harmful content. The model page also distinguishes prompt text, model responses, and the last turn of a conversation as separate evaluation uses. Configure and test each direction separately because the evaluated content and policy consequence differ.
- What can Granite Guardian evaluate in a RAG workflow?
IBM documents context relevance, groundedness, and answer relevance. Context relevance concerns the retrieved material's pertinence to the query. Groundedness checks whether claims are supported by or contradict the supplied context. Answer relevance checks whether the response addresses the user's input. The production request must supply the evidence required by the selected criterion.
- Does Granite Guardian authorize tool calls?
Granite Guardian can judge function-calling hallucination, including syntax or semantic errors against the user query and available tool definition. The actual tool endpoint still enforces authorization. Keep the classification score beside the downstream permission result rather than treating one as proof of the other.
- Does this guide cover running Granite Guardian locally?
No. IBM publishes local model examples, but this guide is limited to a remotely hosted Guardian classification endpoint reached through authenticated HTTP. Local loading, an in-process model call, and traffic that bypasses the governed HTTP route remain outside the DeepInspect boundary.