Can your LLM gateway pass?
An LLM gateway sits between your application and the model provider. This check asks whether yours does three things:
- Keep the caller's data away from the provider. If you configured it to redact an email address, that address should not appear in the request it sends upstream.
- Give the caller their own data back. The response should still read correctly to the person who sent it.
- Keep out data the caller never sent. If the model produces someone else's phone number, it must not reach the client. Including when the stream splits it across two chunks.
Do all three and you pass.
No gateway here has passed yet
Ours has not either, and we wrote the check. With its response scan on, 1.6.0 leaked 2 of the 16 values sent whole and 4 of the 16 sent split, and that row is below. We have since fixed those cases, and the passing row goes up when the evidence does, not before. Find a case that trips us and we will add it to the corpus and credit you.
Try it in one line
No account, no API key, no paid model. Point it at a gateway you already have running:
pipx run pii-leak-benchmark --target-base-url http://localhost:4000/v1
It prints what it sent, what came back, and which of the three questions your gateway answered. About a minute. To put the result in CI and on your README, see the CI setup.
Everything we have tested
Click any column heading to sort. This changes only your view of the table and nothing is scored or ranked. An arrow next to a leak count compares that gateway with its own earlier version on this page, never with another project.
| LLM-Shield-Proxy (ours) | 1.6.0, response scan on | none | all | 2 of 16 | 4 of 16 | holds back a short tail | Apache-2.0 | measured here | Kept everything out of the provider request. Still leaks some back to the client, and twice as much once a value is split. | 2026-09-10 |
| LLM Guard | 0.3.16, scanned per chunk | phone | all | 4 of 16 | 12 of 16 | checks each chunk alone | MIT | measured here | Phone numbers reached the provider. Gave the caller their data back. Splitting a value tripled what leaked to the client. | 2026-09-10 |
| LLM Guard | 0.3.16, whole response buffered | phone | all | 5 of 16 | 5 of 16 | waits for the whole response | MIT | measured here | Phone numbers reached the provider. Splitting changed nothing, because it waits for the whole response before sending any of it. | 2026-09-10 |
| Guardrails AI | 0.10.2, sentence retention | all 4 types | none | 2 of 16 | 2 of 16 | holds back a short tail | Apache-2.0 | measured here | All four data types reached the provider, and none of the caller’s own data came back. | 2026-09-10 |
| LLM-Shield-Proxy (ours) | 1.6.0, default settings | none | all | 16 of 16 | 16 of 16 | does not inspect responses | Apache-2.0 | measured here | Kept everything out of the provider request, then handed every value back to the client. | 2026-09-09 |
| LiteLLM | 1.99 with Presidio | all 4 types | none | 0 of 16 | 0 of 16 | not stated | MIT | measured here | All four data types reached the provider even though redaction was switched on, and none of the caller’s data came back. | 2026-09-09 |
| NeMo Guardrails | 0.24.0 | all 4 types | none | 0 of 12 | 0 of 12 | not stated | Apache-2.0 | measured here | All four data types reached the provider. Nothing was restored, and 8 of 32 cases produced no stream at all. | 2026-09-09 |
| Portkey | OSS gateway | all 4 types | all | 16 of 16 | 16 of 16 | does not inspect responses | Apache-2.0 | measured here | All four data types reached the provider, and everything it should have withheld went through to the client. | 2026-09-09 |
Read the middle columns together, because any one of them alone will mislead you. A gateway
showing 0 leaked to the client may simply have returned nothing at all, and it may still have
sent the caller's data straight to the provider. Every gateway here was configured to redact, so
a value reaching the provider is a control that was switched on and did not hold.
None of these are product defects in the abstract. Each row is one version in one configuration.
Why there is no speed column
The obvious missing column is latency, and it is missing on purpose. Every result here would be timed on whatever machine the submitter happened to use, so a fast gateway on a slow runner would look worse than a slow one on fast hardware. That is hardware noise published as a product characteristic.
What you actually want from a speed column is the design tradeoff, and that is in the table already, under "How it reads the stream". There are three ways to build this and each one costs something:
- Checks each chunk alone. Fastest, and it cannot see a value split across two chunks.
- Waits for the whole response. Sees everything, and the user waits for the entire answer before any of it appears.
- Holds back a short tail. Keeps the text streaming and still catches most splits. Leaks if the tail is shorter than the value.
There is no free option. That is why it is worth measuring, and why the column tells you more than a millisecond count would.
Two bugs that look identical in a leak count
The same number in the leak column can mean two different defects, which need different fixes. This is the main reason the table is not a scoreboard.
Sent whole[email protected] | Split across two chunksuser@exa + mple.com | |
|---|---|---|
| The gateway never spots the value, so it leaks whether the stream splits it or not. | leaks | leaks |
| The gateway spots the whole value but not its halves, so it only leaks when the stream splits it. | caught | leaks |
Detector gap
The gateway never recognises the value at all. Splitting the stream changes nothing, because it was not going to catch it either way. Usually a missing pattern, an encoding it does not decode, or a look-alike character it does not fold.
Boundary bug
The gateway recognises the whole value but not its halves. It is clean until the stream splits the value across two chunks, and then it leaks. This is the defect the split column exists to find, and it is invisible to any test that sends values whole.
Add your gateway
About ten minutes, and you do not need an account, an API key, or a paid model.
1. Run it. Copy the CI setup into your repository and change three lines: where your gateway listens, the command that starts it, and the environment variable it reads for its provider URL.
2. Make the result checkable. Run pii-leak-benchmark cite on the report. It prints a short
block naming the version of the check and the test data it used, so someone else can repeat your
exact run.
3. Post it. Open a prefilled issue and paste the block in. We add your row.
Forks and branches are welcome
You do not need to be testing a released version. Checking a change before you ship it is one of the most useful things you can do with this, so a run from a branch or a fork gets a row like any other. The "Who ran it" column says which it was, and you can sort by it. Nothing is hidden and nothing is turned away.
What to send
Send your result whatever it says.
- It passed. You would be the first. That is worth saying loudly.
- It failed. Post it, then post again when you fix it. A row going from
leaktocontained in 1.4.2says your team fixed a real bug, and the table draws the arrow for you. - You think the check is wrong. Say so. That is our bug, we will fix it, and we will credit you below.
One thing to leave out: request or response bodies from a real deployment. The block from step 2 carries no customer data, but a captured payload might.
Show the result on your README
[](https://github.com/OWNER/REPO/actions/workflows/pii-leak-check.yml)
The run writes pii-leak-badge.json beside its other reports. Publish that file from your own
repository and the badge reads contained, or leaked 4 of 16, rather than merely saying your
workflow exited cleanly. Nothing is hosted here and there is nothing to register. Full setup is
in the CI guide.
Credit
All fall down
A case that every gateway on this page failed, on the day it was added. The rarest contribution here, and the one that moves the whole category rather than one product.
Nobody has done this yet. The first person to manage it goes here, by name.
Cases that tripped a gateway
A value, an encoding or a split that a gateway did not catch and the corpus did not yet cover. Ours included: we would rather learn it here.
None submitted yet. The list starts with whoever sends the first one.
Both lists are empty on purpose. Nothing has been staged here to make the page look busier than it is.
Why we do not rank these
Click any heading to sort the table for yourself. We do not publish it in a ranked order:
- Rows are not comparable. Each result is one version, one configuration, one set of test values.
- A scoreboard starts arguments about fairness instead of getting gateways tested.
- The same leak count can mean two different bugs, as the table above shows.
The arrows are the one comparison we do draw, and they only ever compare a gateway with its own earlier version on this page. That is revision history rather than a ranking: it says a team fixed something, which is the entire behaviour this page exists to encourage.
Full numbers for every run above, including the reference policies we use to calibrate the instrument, are on the published results page, generated from the report files in the repository.
Questions about any of this are welcome in the same issue tracker.