A language model does not read your words. It reads tokens, the small chunks a tokenizer splits text into before the model ever sees it. Most tokens are common and the model has seen them millions of times. A few are not. A glitch token attack abuses the rare tokens that sit in the vocabulary but were almost never seen during training, so the model has no real idea what they mean. Feed one in and the model can stall, hallucinate, ignore its own rules, or refuse to repeat a string it just received. This post takes the mechanism apart: what tokenization is, how these dead tokens get into the vocabulary, why they break the model, and how to detect and defend.
First, what a tokenizer does
Before a model processes text it turns that text into numbers. A tokenizer holds a fixed vocabulary, a list of strings each mapped to an integer id. Splitting happens by frequency, not by words. Common sequences become one token, rare ones get broken into smaller pieces, so an everyday word might be a single token while an odd string gets chopped into four:
text: "the invoice is ready" tokens: ["the", " invoice", " is", " ready"] ids: [464, 18923, 318, 4604]
The vocabulary itself is learned. A training process scans a large pile of text and merges the byte sequences that show up most often, so whatever appears a lot earns its own token. The model later assigns meaning to each token id through its embeddings, the vectors that represent each id.
How a dead token ends up in the vocabulary
The tokenizer is built from one pile of text. The model is trained on another, usually cleaner one. These two piles are not identical, and that gap is where the trouble starts. The tokenizer scan often includes raw, noisy data: scraped web dumps, code repositories, log files. A weird string can appear often enough in that noise to win its own token, like a forum username repeated across thousands of posts, or a substring a broken scraper duplicated everywhere:
// invented examples of strings that could become single tokens " zztopforumuser_4417" -> id 51234 "_______BEGIN_LOG_______" -> id 51987 " qWeRtY_placeholder_99" -> id 52640
Each earns a token id because it was frequent in the tokenizer corpus. But the model trains on the cleaner corpus, where these strings barely appear, so the token exists while the model almost never practiced with it. Its embedding stays near random, because almost no gradient ever pushed it anywhere useful. That is an under trained token, a live wire in the vocabulary.
A glitch token is a word the model can read but never learned to mean. The id is valid, the embedding is noise, and the model has no honest answer for what comes next.
Why a glitch token breaks the model
Everything downstream depends on the embedding being meaningful. Feed in a token whose embedding is near random and you inject noise into the first layer, with no learned pattern to fall back on. The output goes strange in repeatable ways:
- It cannot repeat the token. Ask it to echo the string back and it returns a different word, a blank, or an apology. It cannot map the token to characters it can reproduce.
- It hallucinates. The near random vector lands in an unrelated region of the model’s space, so it free associates and produces text with no link to the input.
- It evades or refuses. The model dodges the request, changes the subject, or insists the token is not there.
- It destabilizes. Output can loop, emit broken characters, or run on until a length limit, which starts to look like a denial of service.
How a glitch token attack works
Once an attacker finds these tokens, they become a tool. A glitch token attack means feeding the model anomalous tokens on purpose, to push it into behavior the builders never planned for. The aims fall into three.
Filter evasion
Safety filters and input checks are usually written against normal tokens. They were never tuned on a string that tokenizes to a single dead id, so it can slip past pattern matching while still steering the model. An attacker splices a glitch token next to a banned request: the surface text looks harmless to a scanner, but the model reads an unstable instruction. This is a cousin of indirect prompt injection, where the input that reaches the model is not the input a reviewer thought they were checking.
Instability and denial of service
A handful of glitch tokens can make a model loop or generate long runs of garbage. If a request can be made to cost the maximum output tokens every time, an attacker has a cheap way to waste compute and slow the service.
Guardrail probing
Because a glitch token drops the model into an untrained region, its usual guardrails may not apply there. An attacker probes for tokens that make the model drop its format, ignore a system instruction, or reveal scaffolding. It overlaps with system prompt extraction, since a model knocked off its normal track sometimes spills context it was told to keep. The vocabulary holds inputs the model cannot handle.
How to detect glitch tokens
Detection starts with the vocabulary itself.
- Audit the vocabulary against the training data. For each token, check how often it actually appeared in the training corpus, not the tokenizer corpus. Tokens with near zero training frequency are your suspect list.
- Measure embedding anomalies. Under trained embeddings cluster oddly, often sitting unusually close to the origin or far from every normal token. Flag the outliers and review them.
- Probe behavior directly. Ask the model to repeat each candidate. One it cannot echo back, or that produces wild output, is a glitch token by behavior.
- Monitor live input. Watch production traffic for rare token ids and strings that tokenize into your flagged list.
How to defend against a glitch token attack
Detection finds the live wires. These controls keep them from being touched.
- Filter and normalize input tokens. Strip or reject the suspect ids before they reach the model, and normalize unusual strings rather than passing them straight through.
- Treat token level input as untrusted. Do not assume the tokens a request produces are the words a person typed. Validate at the token layer, not just the text layer, because that is the layer the attack lives on.
- Retrain or penalize the dead embeddings. The deeper fix is at training time. Give under trained tokens real exposure, prune them, or push their embeddings toward a safe default so an unseen token degrades gently instead of exploding.
- Bound the output. Cap output length and watch for looping generation, so one anomalous prompt cannot burn unlimited compute.
They share one idea. The vocabulary is part of the attack surface, not an internal detail you can ignore. Every token id is an input a user can reach, so it deserves the same scrutiny as any untrusted input.
The assumption that breaks
Under all of this sits one quiet assumption: that every entry in the vocabulary means something the model learned. Most do. A few are valid ids the model never really trained on, and the gap between a token existing and a token being understood is the whole bug. You find this kind of flaw by asking what a system assumes and where that assumption fails, not by matching known bad strings. An autonomous researcher built to test assumptions instead of payloads is built to find exactly this gap. As an early signal, a frontier model drove that full methodology on its own and identified and verified real access control and injection issues in test applications it had not seen before. More on our about page.
This attack is one entry in our AI Agent Security Field Guide, a map of how AI agents get attacked and how to defend each one.
Frequently asked questions
What is a glitch token?
A glitch token is a string that has its own entry in a model’s tokenizer vocabulary but was almost never seen during the model’s training. The token id is valid, yet the model never learned what it means, so its embedding stays near random. Feeding it in produces strange, repeatable behavior such as the model failing to repeat the string, hallucinating, or refusing the request.
How does a glitch token attack work?
An attacker first finds the under trained tokens in a model’s vocabulary, then feeds them in on purpose to push the model into behavior its builders never planned for. The aims include slipping past input filters that were never tuned on these tokens, triggering unstable or looping output that wastes compute, and probing for cases where the model drops its normal guardrails. It works because the model has no learned response for a token it never really trained on.
Why does an under trained token break the model?
Everything the model does depends on each token having a meaningful embedding, the vector that represents it inside the model. An under trained token barely appeared in the training data, so almost no gradient ever shaped its embedding and it stays close to its random starting point. Feeding that near random vector into the first layer injects noise the model has no learned pattern to handle, so the output goes unpredictable.
How do glitch tokens end up in the vocabulary in the first place?
The tokenizer vocabulary is built by scanning one pile of text and giving a token to whatever strings appear most often, while the model is trained on a different, usually cleaner pile. Noisy strings like a repeated username or a log line can be frequent enough in the tokenizer corpus to earn a token, yet rare in the training corpus. The token then exists but the model almost never practiced with it.
How do you detect glitch tokens?
Start by auditing the vocabulary against the actual training data and flag tokens that appeared near zero times during training. Measure embedding anomalies, since under trained tokens cluster oddly, and probe behavior directly by asking the model to repeat each candidate token. A token the model cannot echo back is a glitch token by behavior, which is the test that matters most.
How do you defend against a glitch token attack?
Filter or reject your flagged token ids before they reach the model and normalize unusual strings instead of passing them straight through. Treat token level input as untrusted and validate at the token layer, not just the text layer, since that is where the attack lives. The deeper fix is at training time, retraining or pruning the under trained embeddings, and bounding output length so a single anomalous prompt cannot burn unlimited compute. The vocabulary is part of the attack surface.
Put an autonomous researcher on your own systems
UnboundCompute is an autonomous security researcher that reasons about how an application fits together and proves the access control and injection bugs it finds. We are opening a small number of founding design partner seats: private early access pointed at a staging target you choose, a say in what it looks for, and founding pricing. If your team ships software worth pressure testing, apply to the design partner program.