You can copy a model without ever seeing its weights. A model extraction attack turns a paid API into a free teacher: the attacker sends a stream of inputs, records what comes back, and trains a fresh model to imitate those answers. The original took data, compute, and months of tuning. The clone takes a credit card and a script.
What a model extraction attack is
The target is a model behind an API. You send it text or an image, it returns a label, a score, or a generated answer. That is the only access the attacker has, and it is enough. Each query is a free training example: an input the attacker chose, paired with an output the target produced. Collect enough pairs and you can train a substitute to map the same inputs to the same outputs. The substitute need not share the target’s architecture. It only needs to agree with it on the cases that matter.
People do this for plain reasons. One is theft of a paid product: a competitor pays per call for a while, harvests a few hundred thousand answers, then runs their own clone for nothing. Another is dodging the cost of training from scratch, since the target hands over the labeled data one response at a time. The worst reason is the offline one. Once an attacker owns a local copy, they can probe it with no rate limits and no logging to design other attacks against the real one.
How the copying works
The loop is short. Pick inputs, query the target, store the input and output together, and train on the collected set. The whole job is bounded by how many queries you can afford and how much each one reveals.
for x in probe_inputs:
y = target_api.query(x) # the only access you have
dataset.append((x, y))
substitute = train(dataset) # your cloneWhat the target returns changes everything. A bare top label leaks the least: one decision per call. A full set of class probabilities leaks far more, because it shows how sure the model is and how it ranks the runners up. Raw logits leak the most. With richer outputs the attacker learns the shape of the decision boundary, not just which side a point landed on, so each query teaches more and the clone converges in fewer calls.
If your API returns confidence scores on every call, assume every caller is also collecting a training set. Verbosity is the leak.
What makes a model cheap or expensive to steal
Three things set the price for the attacker. Get them wrong and a clone is cheap.
- Output verbosity. Labels only is the stingiest answer. Probabilities and logits hand over gradient like signal that slashes the number of queries needed.
- Query limits. If a caller can fire millions of requests with no ceiling, they can sample the whole input space. Tight per key limits force the attacker to be efficient or give up.
- Task narrowness. A model that sorts email into three buckets is easy to mimic. A general text generator, with a wide open output space, needs vastly more queries to approximate, and the copy is rougher.
A scenario: the cloned classification API
Picture an invented startup, Acme Triage, that sells a support ticket classifier. You post a ticket, the API returns a category and a confidence for each of forty classes. The scores are detailed because customers asked for them. A competitor signs up under a throwaway account and submits two hundred thousand realistic tickets pulled from public forums, saving every category and score. Two weeks later they train a substitute on those pairs. It agrees with Acme Triage on most tickets, so they ship it as their own feature, undercut on price, and never pay Acme again. Acme sees only a paying customer with steady traffic, because every request was a normal request.
The second order risk
The stolen substitute is not just a cost problem. It is a workbench. Adversarial examples, the tiny crafted perturbations that make a classifier confidently wrong, often transfer between models that solve the same task. The attacker searches their offline clone for inputs that fool it, then a good fraction of those same inputs fool the real Acme Triage on the first try. The clone turned a black box into a white box, and every probe that used to cost a logged API call now costs nothing.
How it differs from membership inference and denial of wallet
These get mixed up, so be precise. An embedding inversion attack tries to rebuild inputs from internal representations. Membership inference asks a narrow question about one record: was this exact example in the training set. Model extraction asks for the whole behavior: copy what the model does across the board, not what it remembers about any single row. Membership is a yes or no about one point; extraction is a wholesale clone of the function.
It also rides the same traffic as denial of wallet. Bulk querying to steal a model runs up the target’s bill at the same time, whether the attacker means to or not. One campaign can drain the budget and lift the IP in a single pass, so both belong on any map of the agent attack surface.
Detecting a model extraction attack
You cannot see the attacker’s training run, so watch the only thing you control: the query stream.
- Query pattern monitoring. Honest users cluster around real tasks. Extraction traffic often spreads evenly across the input space, sampling regions a real user never visits.
- Volume and rate anomalies. A single key pulling far more varied queries than any real workload is the loudest tell.
- Per account baselining. Flag callers whose inputs look like coverage of the decision space rather than a stream of real tickets.
Preventing a model extraction attack
No one control stops this. They stack, and each raises the query cost of a usable clone.
- Rate limit per identity. Cap calls per key and per time window so wide sampling becomes slow and expensive.
- Reduce output granularity. Return the top label, or a coarse confidence band, instead of full probabilities or logits. Less signal per call means more calls for the same clone.
- Require strong authentication. Tie every call to a verified account so throwaway keys are harder to spin up.
- Watermark the outputs. Bias the responses in a faint, consistent way that a substitute absorbs during training. If a competitor’s model carries your watermark, you have evidence it learned from your API.
- Monitor for systematic probing. Treat coverage style traffic as a security event, not just usage, and step up friction when a caller starts mapping the space.
The assumption that breaks
The whole API rests on one belief: that query access is harmless because the weights stay hidden. Extraction breaks that belief. The outputs are the model, just sampled slowly, and a determined caller can reassemble enough to compete. You find this risk by asking what each response gives away and how cheaply it can be collected, not by scanning for a known payload. An autonomous researcher that tests the assumptions an API makes, rather than a fixed list of attacks, is built to surface this kind of gap. As an early signal, a frontier model drove the full methodology on its own and identified and verified real access control and injection issues in test applications it had not seen before. You can read more on our about page.
This attack is one entry in our AI Agent Security Field Guide, a map of how AI agents get attacked and how to defend each one.
Frequently asked questions
What is a model extraction attack?
It is when an attacker with only query access to a model, through its API, copies its behavior without ever seeing the weights. They send many chosen inputs, record the outputs, and pair each input with its output to build a training set. Training a substitute model on those pairs produces a clone that agrees with the target on the cases that matter. The original cost data, compute, and tuning to build, while the copy costs only the price of the queries and a script to run them.
Why would someone steal a model this way?
To get capability they did not pay to build. A competitor can clone a paid product, then run the copy for free instead of paying per call. It also skips the cost of training from scratch, since the target hands over labeled data one response at a time. The worst motive is offline probing: once an attacker holds a local copy, they can study it with no rate limits and no logging to design further attacks against the real model.
What makes a model cheaper or more expensive to steal?
Three factors. Output verbosity is the biggest: returning full probabilities or logits leaks far more per query than a bare top label, so the clone needs fewer calls. Query limits matter too, since loose limits let an attacker sample the whole input space cheaply. Task narrowness is the third: a classifier with a few buckets is easy to mimic, while a general text generator with a wide output space needs far more queries and yields a rougher copy.
How is model extraction different from membership inference?
They answer different questions. Membership inference asks a narrow yes or no about one record: was this exact example in the training set. Model extraction asks for the whole behavior: copy what the model does across all inputs, not what it remembers about any single row. Membership is about one point of training data, while extraction is a wholesale clone of the function the model computes.
How do you detect and prevent a model extraction attack?
Detect it by watching the query stream: monitor query patterns for traffic that covers the input space evenly, flag volume and rate anomalies, and baseline each account against real workloads. Prevent it by stacking controls. Rate limit per identity, reduce output granularity to a top label or coarse band instead of full logits, require strong authentication, watermark outputs so a clone carries proof it learned from you, and treat systematic probing as a security event.
Put an autonomous researcher on your own systems
UnboundCompute is an autonomous security researcher that reasons about how an application fits together and proves the access control and injection bugs it finds. We are opening a small number of founding design partner seats: private early access pointed at a staging target you choose, a say in what it looks for, and founding pricing. If your team ships software worth pressure testing, apply to the design partner program.