Not Diamond Alternatives: A Module Exit Plan

Not Diamond combines model routing with prompt optimization, so a replacement decision crosses two release systems. One chooses a model for each input. The other changes the prompt used with a target model. A team that moves both behind one feature flag loses the evidence needed to attribute a regression.

I would reject a Not Diamond exit plan that starts with a vendor shortlist. Draw the production call on a whiteboard first. Mark the selection request and its result. Add the prompt version beside the evaluation record that justified both choices. Those four marks define the migration.

TL;DR

• Split the exit into routing, prompt optimization, evaluation data, and API integration. Give each module an owner and acceptance test, then define its rollback trigger.
• Baseline the pre-trained router and every custom router against frozen prompts before dual-running another decision service.
• Preserve raw model responses and evaluator configuration. Aggregate scores alone cannot train a replacement router or explain a changed selection.
• Cut over prompt optimization separately from routing, then retain a stable request ID across the old and new paths.

The Not Diamond product boundary has two decision engines

Not Diamond describes itself as an intelligent AI model router and prompt optimization product. Its current documentation exposes Python and TypeScript SDKs alongside a REST API. That product wording identifies two separate migration units even when one account and API key serve both.

The routing unit predicts the best-suited model from a candidate set. Its routing concepts distinguish a general-purpose pre-trained router from a custom router trained on a customer's evaluation data. The request can favor quality or cost, with latency as another routing input.

Not Diamond returns a session ID for tracking the selection, while a preference ID identifies a custom router.

Prompt optimization has another input contract. It accepts an existing system prompt and user-message template, evaluation data, target models, and a metric. The service returns target-specific prompts with pre-optimization and post-optimization results. An exit inventory should keep those objects separate from router sessions and preference IDs.

Pre-trained and custom routers need different exit tests

A pre-trained router contains vendor-supplied routing behavior. Capture its observable contract rather than pretending its internal model can be exported. Freeze a representative request corpus, the candidate model list, and the request-time tradeoff. Record the selected provider and model beside the session ID.

Add the response quality score later through the same evaluator used for the baseline.

The current docs also say the continuous cost_quality_tradeoff applies only to the default pre-trained router. That setting ranges from quality-first to always selecting the cheapest candidate, and requests combining it with the other tradeoff setting receive HTTP 422. A replacement must reproduce the business intent of those states. It need not copy the same parameter names.

Custom routers carry more portable material. Not Diamond's custom-router training guide requires prompt inputs and candidate-model responses scored with numerical evaluations. The published service currently accepts at least 15 samples, with documented upload limits of 5 MB or 10,000 samples per training job. Preserve the full training file, model list, maximize setting, prompt-column mapping, preference ID, and training timestamp. The preference ID alone is a locator, not a recoverable training package.

A separate page confirms that custom routing can include an arbitrary inference endpoint, including a fine-tuned model or an agent, when its evaluation results are present. Those entries also carry context-length and token-price metadata. They carry latency as well. Export that metadata with the scores because a replacement router needs the same cost and latency assumptions to make a comparable decision.

Prompt optimization exits on its own release clock

Not Diamond says its prompt optimization process uses evolutionary algorithms and reinforcement learning. It evaluates the original prompt, generates target-specific variations, validates them with the selected metric, and returns results for comparison.

This module can change model behavior without changing the route.

Inventory every production system prompt and user-message template. For each target model, save the optimized version, originating prompt, optimization run ID, evaluation metric, train and test samples, and recorded scores. Include the application release that first used the prompt. A prompt visible in a dashboard becomes operationally useful only after it can be tied to a deployed version.

Run the prompt move after the routing baseline is stable. Send the same model the old and replacement prompts, then compare sample-level outputs through a fixed evaluator. Keep the production prompt unchanged while testing a new router. Once route parity is accepted, move prompt ownership as a second release. This sequence costs another deployment window, but it buys a diagnosis when results move.

Evaluation data is the portable core

The evaluation dataset links routing and prompt optimization. A custom router learns which candidate performed best for each input. Prompt optimization uses examples to test prompt variations against a metric. Preserve raw inputs, candidate responses, reference answers where present, sample IDs, split membership, and per-response scores.

Not Diamond's current evaluation metrics include an LLM-judged semantic-similarity default, RAGAS measures, custom LLM judges, JSON match, exact match, BLEU, ROUGE, and METEOR. Record the exact metric identifier. For a custom judge, retain its prompt, judge model, parsing rule, and correctness cutoff.

A score of 1 has little meaning after the judge or cutoff changes.

Use LLM evaluation tools to run the same saved dataset outside the migration path. Compare sample-level disagreements before comparing one aggregate number. Print the ten disputed rows and put them on the review table; a principal engineer can inspect those pages faster than a chart with three decimal places.

API integration inventory exposes the real switching cost

Search repositories and secret stores for Not Diamond SDK initialization, REST base URLs, API keys, model lists, preference IDs, session IDs, optimization run IDs, and status polling. Include notebooks, scheduled jobs, CI workflows, and internal tools.

A central service inventory routinely misses the analyst script that still trains a router every Friday.

Group callers by their API contract. Routing clients submit messages and candidate models, then consume a selected model plus tracking data. Training clients upload evaluation files and later use the returned preference ID. Prompt-optimization clients submit asynchronous work and retrieve target-specific results after polling status. Each group needs a fixture that records request shape, response fields actually consumed, timeout handling, retries, and error behavior.

Build an adapter before changing every caller. The adapter should translate one internal routing request into the active provider contract and return one internal decision object. Keep provider-specific fields in a namespaced evidence record. This limits the cutover to an adapter configuration change and gives rollback one switch instead of twelve repository deployments.

Dual-run replay precedes traffic cutover

Create a frozen replay set from production-shaped inputs after removing or protecting sensitive values under the organization's handling rules. Include short and long prompts, domain-specific requests, model failures, cost-sensitive calls, and inputs where candidate scores were close. Reuse the exact candidate set and evaluator version from the baseline.

During dual run, the incumbent remains authoritative. The replacement receives the same normalized request without making the production model call. Compare selected models, expected cost, routing latency, and evaluated response quality. Investigate disagreements by sample class. A different selection can be acceptable when the replacement meets the declared quality threshold at lower cost; undocumented disagreement blocks cutover.

Carry one migration correlation ID through both decisions. Store the Not Diamond session ID and replacement decision ID beside it. The LLM routing strategies guide explains the wider routing choices, while this replay focuses on proving the selected strategy against your own traffic.

Cutover assigns one authority before retirement

Promote the replacement by traffic cohort rather than changing every request at once. Start with a low-impact route whose quality has an observable evaluator. Define rollback triggers before the first production request: evaluator regression, selection-error rate, added latency, provider-budget breach, or an unavailable candidate model. The AI gateway rollback strategy provides the same owner-and-trigger discipline for a request path.

Keep Not Diamond callable during the rollback window, but assign one router as the production authority. Two active authorities create inconsistent model choices and duplicate spend. Continue shadow decisions only for comparison, with model invocation disabled on the shadow side where possible.

Retire the old integration after the agreed observation window passes and every caller has moved. Revoke API keys, stop training jobs and optimization polling, remove preference IDs from active configuration, and preserve the evidence package under its retention owner. Test the old endpoint after revocation. A successful request at that point means the exit remains incomplete.

DeepInspect

DeepInspect can sit on the chosen HTTP route after a router selects the model. It evaluates supplied caller identity, data classification, destination, and active policy before forwarding an approved model request. That position can produce a separate policy decision record for the routed call.

Keep the migration correlation ID in both records. The router record explains why a model was selected; the DeepInspect record explains whether that authenticated request was permitted or denied, including any change required by organizational policy. Signed audit logs for AI requests describes the evidence properties for that second record. Routing experiments and prompt evaluation remain in the systems built for those jobs. Book a technical deep dive at deepinspect.ai.