← Blog

Tokenizer Attacks Exploit the Gap Between Text and Tokens

Parminder Singh
Parminder Singh··3 min read
Summarize with AI

Tokenizer attacks craft text whose token representation degrades model understanding or evades text-level assumptions. Defenses need Unicode normalization, raw and normalized evidence, model-specific token-budget tests, route policy, and response checks rather than one universal character filter.

Problem-Awareai-securityllm-securityprompt-injectionpolicy-enforcementarchitecture
Tokenizer Attacks Exploit the Gap Between Text and Tokens

An LLM receives tokens rather than the characters a reviewer sees. The paper Tokenization Matters, submitted in May 2024 and revised in May 2025, constructed adversarial inputs from model vocabularies to degrade understanding. The work highlights a production gap: text that looks similar to a person can fragment very differently across model tokenizers.

TL;DR

  • Tokenizer attacks target the transformation between visible text and model tokens.
  • Preserve raw input and normalized input.
  • Also preserve tokenizer identity, token count, policy outcome, and request ID.
  • Test every approved model route because the same string can produce different token sequences and costs.

Character checks see only one representation

A security filter may scan Unicode characters, while the model consumes subword tokens produced after normalization and tokenization. Rare scripts and mixed encodings can change meaning and length. Invisible characters and adversarial segmentation can change classifier behavior and request cost.

The practical scene is a review console displaying one short line while the model meter reports hundreds of tokens. That discrepancy deserves a policy event. It may signal obfuscation or a cost attack. It may also signal a policy bypass or an input the model will misunderstand.

OWASP's prompt-injection guidance includes obfuscated and adversarial inputs among common attack forms. Tokenization explains one route by which that obfuscation reaches the model.

Normalization needs an evidence trail

Apply explicit Unicode normalization and remove disallowed control characters. Identify mixed-script or confusable text according to the application's language needs, then calculate the destination token count. Retain the raw bytes because a destructive cleanup without the original makes incident reconstruction impossible.

My preference is to reject ambiguous transformations on privileged routes. Silent repair helps ordinary text entry; it also hides the exact payload that crossed a security boundary.

The approach complements ASCII smuggling controls, while tokenizer testing reaches beyond invisible Unicode into model-specific vocabulary behavior.

Test each approved tokenizer

Tokenizers differ across model families and versions. Build a regression corpus containing rare tokens, multilingual text, confusables, and invisible characters. Add repeated separators and long segmentation cases. Measure token count and classification result. Also measure latency and output behavior for each approved route.

A router that moves the same prompt between providers should run policy on a normalized representation and enforce the destination's token budget. Record the tokenizer or provider estimator used for the decision. Otherwise a fallback route can turn an accepted prompt into a much larger request.

LLM multi-model routing needs these invariants at the point where the destination is chosen.

Response checks catch semantic degradation

Some tokenizer attacks seek a harmful bypass. Others simply make the model misread the request. Inspect responses for policy violations and language drift. Also check for missing required fields and signs that the model ignored task constraints. High-risk workflows should fail closed when the normalized prompt and output disagree with the declared task.

A tiny visual payload that expands into a large token sequence can consume shared capacity even when the response stays benign, so enforce rate limits and cost budgets on the destination route.

DeepInspect

DeepInspect can inspect normalized and raw-derived classifications on authenticated HTTP model requests when the application routes them through the proxy. Policy can use identity and destination. It can also use data class, token estimate, and risk signals before transmission.

Tokenizer implementation and model internals remain outside the proxy. The enforceable slice includes the routed request and its declared context. It also includes the policy decision and the response that returns through the same boundary.

Book a demo today.

Frequently asked questions

Normalization and token counting solve different parts of the problem. Normalization creates a consistent representation for policy checks, while token counting shows how a selected model will process that representation. Applications need both measurements attached to the same request because either one alone can conceal the discrepancy that made the input risky.

Preserve the received payload first. Produce the normalized form next, then evaluate it against the tokenizer assigned to the chosen model route. The resulting evidence lets an investigator compare what arrived with what policy inspected. It also shows what the provider was expected to receive without reconstructing those stages from separate logs.

Privileged workflows need stricter handling because a misunderstood instruction can trigger a consequential model action. A route serving document summaries can tolerate different failure behavior from a route that drafts database queries or prepares tool arguments. The policy should reflect the declared task and the identity making the request.

Is Unicode normalization enough?

Normalization removes some representational differences. It cannot make all model tokenizers equivalent or detect every adversarial sequence. Pair it with model-specific tests and token budgets. Add classification and response validation.

Normalization also needs a declared form and consistent placement in the request path. If one service normalizes before classification while another counts tokens from the original input, their records describe different representations. Label both representations with the stage that produced them so later checks can reproduce the policy decision.

Should applications expose token counts to policy?

Yes. Pass the estimated input count and tokenizer identity into the decision. Include the model route and normalized-content hash. Reconcile estimates with provider usage records after the response.

The estimate should be evaluated before transmission, when policy can still reject the request or select another approved route. Provider usage then supplies a post-response check. A persistent gap between the estimate and recorded usage should create an operational signal because model changes or estimator drift can weaken an otherwise valid budget rule.