LLM Evaluation Tools: Choose by Test Method and Deployment Stage

An LLM evaluation run combines a dataset with an application or model version. It applies one or more scoring methods. The output might be a pass/fail result for JSON validity, a judge score for tone, a faithfulness score for a RAG answer, or a comparison between two agent versions. Those artifacts answer different engineering questions. Buying an evaluation platform before naming the decision it must support is how teams end up with a handsome dashboard and no release gate anyone trusts.

TL;DR

• Choose the evidence artifact first: a CI pass/fail, per-example rubric feedback, a RAG metric, an experiment comparison, or a production-quality signal.
• Use deterministic checks for rules a machine can verify. Reserve LLM judges for semantic criteria, then calibrate them against human labels.
• OpenAI and MLflow overlap with LangSmith and Ragas. Their centers differ between provider-native testing and experiment management, and between trace evaluation and RAG metrics.
• Evaluation measures behavior. Runtime gateway enforcement decides whether a live authenticated HTTP model request proceeds and records that decision.

The evidence artifact comes first

Start with the meeting where the result will be used. A pull request needs a threshold that can fail the build. A prompt review needs per-example output beside the expected answer and grader feedback. A RAG team needs to separate retrieval quality from answer quality. An incident review needs the production trace that produced a bad response.

Picture a release review on a wall monitor: 198 test rows are green, two are red, and the engineer can open each failure to see the input and retrieved context. The same view shows the output and expectation, plus the scorer and application version. A single "quality: 0.84" tile hides the failure mode and creates an argument about the metric.

I would reject any evaluation-tool shortlist that begins with feature count. The shortlist should begin with fields for the evaluation job and source data. Add the scoring method and deployment stage, then name the evidence consumer. That turns a vague platform purchase into an engineering decision.

This scope also separates evaluation from LLM observability tools. Tracing captures what an application did. Evaluation applies criteria to selected outputs or traces. A platform may support both, but the records serve different questions.

Match the scoring method to the claim

Deterministic scorers fit claims with an objective answer. JSON parses. A required field exists. A tool call used an approved function name. Latency stayed under an internally chosen threshold. Exact match and regular expressions produce repeatable results. Schema validation and executable tests do too, giving developers results they can interpret without asking what a judge "meant."

Rubric and LLM-as-judge scoring fits semantic claims such as correctness against a reference, response style, or adherence to written criteria. The flexibility comes with a governance cost. The judge model and prompt become part of the evaluation definition, along with the rubric and version. Teams need human-labeled examples to check agreement and a review path for borderline failures.

RAG evaluation adds another split. Retrieval metrics examine the contexts selected for a query. Response metrics examine the generated answer. Ragas documents context precision and context recall. Its available metrics also include response relevancy and faithfulness, tool-call accuracy and agent-goal accuracy, plus BLEU and ROUGE alongside exact match. Combining them into one score erases where the system broke.

OpenAI fits provider-native prompt and model testing

OpenAI defines evaluations as tests of model outputs against specified style and content criteria. Its workflow connects test data to an eval definition, runs the candidate prompt or model, and exposes results for analysis. For a team already building on OpenAI, the attraction is proximity: prompts and models can live in the same provider environment. Datasets and graders can live there too, along with results.

The procurement decision needs a date attached. OpenAI is deprecating its Evals platform. The official Evals guide says existing evals become read-only on October 31, 2026, and the dashboard and API are scheduled to shut down on November 30, 2026. New evaluation work should account for that migration rather than treating the current Evals API as a durable foundation.

OpenAI points new users toward Datasets for iterative prompt testing. That makes sense when the test corpus can sit inside the OpenAI platform and the team wants fast comparison during development. Requirements involving external-model comparison or self-hosted execution push the selection toward a different layer. A provider-independent evidence store creates the same pressure.

MLflow GenAI fits experiment and scorer management

MLflow GenAI treats the evaluation dataset as a managed test repository and attaches evaluation activity to experiments and traces. Its evaluation documentation covers dataset management and human feedback. It also covers LLM-as-a-judge scoring and systematic evaluation, including production monitoring. That combination suits teams already using MLflow to organize engineering runs or teams that want evaluation results beside broader experiment metadata.

The scoring model is explicit. MLflow distinguishes built-in and natural-language guideline judges from custom judges and code-based scorers. A judge can inspect inputs and outputs, along with expectations or a full trace, while code-based scorers handle exact logic such as format validation. Feedback attaches to the trace, giving an engineer a path from a failed score to the execution that earned it.

MLflow is a good fit when the evidence artifact is an experiment record containing the dataset version and application trace, plus the scorer result and run metadata. It also supports turning existing traces into evaluation data, which helps convert a production failure into a regression case. Sensitive-prompt handling still needs a deployment review covering storage location and scorer-model calls. Retention and access need review too.

LangSmith fits trace-centered application evaluation

LangSmith divides evaluation by deployment stage. Its official evaluation types guide describes offline benchmarking and unit tests, along with backtesting and pairwise comparison, plus online evaluation for production monitoring and anomaly detection. Evaluator types include LLM-as-a-judge and deterministic code. Composite and summary evaluators are available alongside pairwise approaches.

That structure works well when the unit under review is an application trace rather than a single model completion. An agent may call a retriever and select a tool. It may then retry a model request and produce a final response. The final text can look acceptable even when the route was expensive or the tool sequence violated the expected plan. Trace-aware evaluation lets the scorer inspect the steps that produced the answer.

LangSmith becomes especially useful when production examples feed the offline test set. A failed run can become a dataset example, then a later application version can be backtested against the same input. Teams should choose it for that feedback loop and trace-centered workflow, rather than duplicating the broader comparison in DeepEval alternatives.

Ragas fits metric-led RAG and agent evaluation

Ragas is a framework for teams that want evaluation metrics in their own testing workflow. Its metric catalog is the center of the product. Faithfulness checks whether claims in a response are supported by retrieved context. Context precision examines the ranking of relevant chunks. Agent metrics cover tool calls and goal completion. The framework also provides general-purpose rubric and similarity metrics, along with string metrics.

This makes Ragas a strong starting point when the engineering question is narrow and measurable: did the retriever place useful chunks near the top, did the response stay grounded in those chunks, or did the agent call the expected tools? The artifact can remain close to the codebase and CI process instead of requiring a full hosted evaluation program.

Metric definitions deserve review before they become gates. Faithfulness establishes support from the supplied context while leaving the truth of that context unverified. Response relevancy establishes alignment to the user input while leaving factual accuracy unmeasured. Ragas is most useful when a team preserves those distinctions and exposes per-example results beside aggregate scores.

A practical selection sequence

Write one sentence describing the release decision. "Block a deployment when any golden-case JSON response fails its schema" points to deterministic CI scoring. "Compare two RAG versions for retrieval and grounded response failures" points to Ragas or a platform that runs equivalent metrics. "Review complete agent traces and promote production failures into regression datasets" points to LangSmith or MLflow. "Iterate on an OpenAI prompt inside the provider workspace" points to Datasets, with the Evals retirement dates recorded in the architecture decision.

Then inspect the data path. Record where prompts and retrieved passages are stored. Do the same for expected answers and human labels, then record the location of judge inputs and results. Check which service receives sensitive examples and which model grades them. A security review should also cover retention and tenant access. Regional requirements and the path for deleting a test case need review too.

Finally, define ownership. ML engineers may maintain metrics, while domain experts label disputed examples. A release owner approves threshold changes. A score with no named owner decays into dashboard furniture. For the wider governance record, connect the evaluation artifact to an AI audit log schema rather than treating a test report as proof of every production decision.

Evaluation and runtime enforcement have separate jobs

An evaluation suite asks how a model or application behaved on selected examples. It can run before deployment, against historical traces, or on sampled production output. Its evidence includes datasets and grader definitions. Scores and comments sit beside experiment versions and traces.

Runtime enforcement evaluates a live request before it reaches an LLM. At DeepInspect's exact boundary, an authenticated user or agent sends an HTTP model request through the gateway. DeepInspect applies identity-aware policy to that traffic, permits or blocks the request, and produces a per-decision audit record. Local test execution and benchmark generation remain outside that boundary. So do judge calls and human annotation, along with score calculation.

The two layers can inform each other. An evaluation may reveal a prompt pattern that should become a policy rule. A blocked production request may supply a sanitized case for a later regression dataset. Their artifacts still answer separate questions, a distinction also covered in DeepInspect vs DeepEval.

DeepInspect

For each routed HTTP model request, DeepInspect evaluates identity context and policy before the traffic reaches the model, then writes the per-decision record. Scoring for faithfulness and relevance remains outside that boundary. The same applies to BLEU and ROUGE, plus rubrics and benchmarks.

Use an evaluation tool to decide if a model or prompt meets its quality criteria. The same evaluation can cover a retriever or agent version. Use DeepInspect when production HTTP model traffic needs identity-aware policy enforcement and per-decision audit evidence.

Book a technical deep dive at deepinspect.ai.