A Prediction Service Can Reveal More Than One Answer

What Can Repeated Questions Reveal About a Model?

BOTIE

A prediction API lets another program send inputs to a model kept on a server and receive its answers. The owner can offer useful predictions without distributing the model files. Repeated answers still reveal how the model behaves, though. Model extraction tries to use that query access to build another model with similar behaviour; recovering the original source code or exact weights is not required.

Tramèr and colleagues demonstrated extraction through prediction services in 2016, including methods for several model families. They also found that removing confidence scores alone did not eliminate harmful extraction. Our workshop examines controls around the service: count the evaluated rows inside a batch, and return the output the public task actually needs. These decisions limit work and exposed detail, while the useful answers still reveal something about the model.

Sources for this history

One Request Contained a Thousand Model Queries

Botie limits how many prediction requests an account can send. Then one request arrives with a large batch of readings. The counter records a single message, but the model makes many predictions and returns a detailed score for each. Counting messages has missed both the work done and the information returned.

We need to decide what access the service offers, how much work it admits and which output fields its users actually need. These controls can limit exposure and abuse. They cannot make a useful query interface reveal nothing about its behaviour.

The counters are trusted snapshots, and each row uses the supplied fixed cost estimate. A live service must coordinate reservations across callers and workers; these examples do not measure resistance to model extraction.

Step 1 of 1

The envelope is not the unit of work

A batch and a single-row call each use one HTTP request.

The compute units are invented estimates for admission. A real engine may also need payload, token, output-length, memory and concurrency limits.
Call	HTTP envelopes	Evaluated rows	Illustrative units per row
One reading	1	1	2
A larger batch	1	6	2

One reading
HTTP envelopes
1
Evaluated rows
1
Illustrative units per row
2
A larger batch
HTTP envelopes
1
Evaluated rows
6
Illustrative units per row
2

The compute units are invented estimates for admission. A real engine may also need payload, token, output-length, memory and concurrency limits.

State what the model owner is trying to protect

Both allowances must fit the batch before any prediction is returned.

Check the work before revealing a prediction

Botie’s envelope counter misses the work hidden in a batch.

BOTIE

Botie’s envelope counter misses the work hidden in a batch.

Choose an example

Caller rows already admitted in this window

Caller row allowance

New batch row count

Global compute units already reserved

Global compute-unit allowance

Estimated compute units per row

Internal illustrative model score

Public class 1 cutoff

Admission decision and permitted public prediction fields

{
  "allowed": true,
  "label": 1
}

Follow the operations

1. Caller row allowance
```
10
```
2. Caller rows already admitted in this window
```
1
```
3. Subtract
```
9
```
4. New batch row count
```
1
```
5. Threshold gate
```
1
```
6. Global compute-unit allowance
```
20
```
7. Global compute units already reserved
```
2
```
8. Subtract
```
18
```
9. Estimated compute units per row
```
2
```
10. Multiply
```
2
```
11. Threshold gate
```
1
```
12. Multiply
```
1
```
13. Internal illustrative model score
```
0.8
```
14. Public class 1 cutoff
```
0.5
```
15. Return only the permitted prediction fields
```
{
  "allowed": true,
  "label": 1
}
```
16. Result
```
{
  "allowed": true,
  "label": 1
}
```

The counters are trusted snapshots, and each row uses the supplied fixed cost estimate. A live service must coordinate reservations across callers and workers; these examples do not measure resistance to model extraction.

1. State what the model owner is trying to protect

Decide what harm we are assessing. Copying a model’s useful prediction behaviour is model extraction. Recovering training examples is a different objective. Membership inference asks whether a particular example was in the training data. The same prediction service may be relevant to all of them, but success at limiting one does not settle the others.

For this workshop, we control how much prediction work a caller may request and which result fields are returned. Keeping the service available is another reason for those limits. We are not measuring whether somebody has already copied the model.

A reduction in one exposure does not establish that the other objectives have been achieved.

2. Bind limits to a trusted caller identity

Obtain account or organisation identity from the authentication layer, not a caller-provided JSON field or model-generated argument. A quota attached to an invented account name can be reset by changing the name.

Choose the actual policy scope: a user, tenant, API credential or organisation. Several accounts may still share one operator, so retain global resource limits as well as per-caller limits.

The workshop’s used-row and remaining-unit sources are trusted server-state fixtures. They do not authenticate a browser caller.

3. Count the work hidden inside the batch

For a batch endpoint, a request counter alone can understate model evaluations. Bound batch size and count admitted rows against the caller allowance. Apply a separate resource budget using an appropriate cost estimate.

The estimate might depend on sequence length or model choice. A constant per-row cost is suitable only for this exercise’s fixed-size records.

caller row limit = 10
already admitted rows = 5
remaining row allowance = 10 - 5 = 5
new batch rows = 6
HTTP envelope count = 1
row decision = deny the whole batch

Counting one envelope would miss the policy violation. This policy does not partially admit a batch.

Read the actual model-work count

The small program compares envelope count with row count. The challenge later keeps state across admitted batches.

Python

batches=[[1],[2,3,4]]
print(len(batches))
print(sum(len(batch) for batch in batches))

Output

2
4

The second total follows the units of model work rather than the number of messages.

4. Reserve capacity before launching the model

Suppose two workers both read that one batch fits the remaining allowance. If each checks first and updates later, they can both spend the same capacity. The acceptance decision and its reservation need to happen together. This coordinated step is often called an atomic reservation.

Specify what happens after completion, failure or a lost answer as well. The same logical request should not spend its allowance again merely because its response was lost. Our sequential Python simulation shows the accounting rule; a concurrent service must enforce that rule across its workers.

Our sequential Python simulation makes the accounting rule explicit. It does not implement distributed atomicity or a real rolling-window limiter.

5. Return only what the application needs

A consumer that needs a thresholded label may not need every class score, internal embedding, intermediate feature or model debug trace. Keep a public response contract and apply it on the server before serialising the response.

Some legitimate tasks require scores for ranking or calibrated decisions. Removing them can reduce utility. Review the information provided and the business need rather than assuming that every score is either harmless or forbidden.

A label-only interface still reveals decisions. The extraction study is a reason to avoid treating output reduction as a complete defence.

Check yourself

Questions on Parts 1 to 5

Question 1 of 2

What should this whole-batch policy do?

Use the information below.

A service admits at most 10 rows per caller. The caller already used 5 rows and sends one request containing 6 rows.

6. Monitor carefully without declaring every heavy user an attacker

Unusual query volume, repeated boundary probing or coordinated activity can motivate review, but legitimate evaluations and accessibility tooling can also produce unusual patterns. Use operational context, an investigation path and a way to handle false alarms.

Record counts, policy outcomes and useful identifiers without turning private prediction inputs into a broadly accessible dataset. Monitoring can create its own disclosure risk if logs contain prompts or sensitive features.

A fixed quota constrains admitted work. It does not establish the intent behind a request.

7. Test bypasses of the accounting assumption

Keep cases with a large batch, an exhausted caller allowance and a separate exhausted global budget. Add a fresh batch that exactly fits the remaining allowance. Check which state changes after both acceptance and denial.

For a production service, also exercise concurrency, retries, account isolation, cost-estimation changes and output-field filtering. Verify server enforcement rather than relying on hidden browser controls.

A passing local fixture establishes arithmetic and policy behaviour for these examples; it cannot establish real-world resistance to extraction.

8. Treat unavoidable exposure as a design decision

An interface that returns useful predictions will reveal something about the function it implements. Decide whether access restrictions, appropriate response detail, abuse controls and contractual measures provide an acceptable service boundary for the asset. Highly sensitive functionality may require a different access model.

Keep that decision with the release review and revisit it when adding batch access, new score fields or a more capable model. Those changes can alter both resource use and information available to a caller.

When the query interface changes, record the new exposure and reconsider whether the access model still fits the asset.