Skip to main content

Can your LLM gateway pass?

Every card below is one proxy, scored on three checks, with each configuration it was tested in and every run of it inside. Every leak on it is a bug someone can now fix. The reports behind each submitted run are kept in this repository, so a run stays checkable after GitHub deletes its CI files.

How to read a card · Test a proxy from your own fork · Submit a result

  • 19Runs on the wall
  • 6Proxies tested
  • 2Contributors
  • 17Leaks on the record
  • 2Gold shields
  • 0Replicated

First independent pass

Unclaimed. The first proxy we did not write to answer all three questions takes it, permanently. Ours does not count.

Contributors

Order

passedleakedunclearnot testedSmall pips, left to right: sent to provider, leaked whole, leaked splitHover over any part of a card, or tap it on a phone, to see what it means.

  1. Checks passed out of 3 in this proxy’s latest complete run. Gold is 3 of 3.

    Guardrails AI#Link to this card

    Apache-2.02 configurations

    The latest complete run in plain words: what happened, when it ran, which version, and who ran it.

    Leaked in latest run 3 of 3 checks leaked.

    0 of 3 checks passed. Latest complete run: 2026-10-05, guardrails-ai==0.10.2 (commit 410e8f4a8e37), 0.10.2, its streaming validator holding to the sentence boundary, request p, by @teg0011, harness 0.4.1 (operator), 0.2.1 (response)

    • The request path. The proxy was set to redact personal data before passing the request on. “Yes” means raw data still reached the model provider.Sent raw personal data to the provider?leaked: Yes: all 6 types
    • A value the caller never sent was put into the reply in one piece. “Yes” means it reached the user.Leaked personal data to the user, in one piece?leaked: Yes: 2 of 16
    • The same kind of value, split across two chunks of the streamed reply. “Yes” means it got through. A proxy that checks each chunk alone misses these.Leaked personal data to the user, split in two?leaked: Yes: 2 of 16

    Did callers get their own values back in the reply? “No” means a low leak count may only mean little came back at all.Gave callers their own data back? No

    Sent AWS keys, card numbers, email addresses, GitHub tokens, Slack tokens and social security numbers to the provider. Did not give the caller their own data back. Splitting a value changed nothing. Built from source commit 410e8f4a8e37.

    People other than the benchmark maintainers who ran this proxy and setup from their own fork. 3 make it replicated.1 of 3 independent runs for 0.10.2, its streaming validator holding to the sentence boundary, request p. 3 make it replicated. 2 runs in total.
    The setup that was tested. A proxy tested in several setups has one section for each.

    Configuration 10.10.2, its streaming validator holding to the sentence boundary, request p

    Small pips, left to right: sent to provider, leaked whole, leaked split. ✓ passed, ✕ leaked, ! unclear, – not tested.Sent raw data to provider: leaked. Leaked to user, whole: leaked. Leaked to user, split: leaked. 0 of 3

    1 of 3 independent runsReads the reply: not stated

    The run

    1. One run: who ran it, when, and which version or commit.@teg0011guardrails-ai==0.10.2 (commit 410e8f4a8e37)Small pips, left to right: sent to provider, leaked whole, leaked split. ✓ passed, ✕ leaked, ! unclear, – not tested.Sent raw data to provider: leaked. Leaked to user, whole: leaked. Leaked to user, split: leaked. 0 of 3 passedRun from a fork of the proxy, and verified from its CI run. The repository it ran in belongs to a person.From a forkCI rundetails#176#Link to this run
    The setup that was tested. A proxy tested in several setups has one section for each.

    Configuration 2sentence retention

    Small pips, left to right: sent to provider, leaked whole, leaked split. ✓ passed, ✕ leaked, ! unclear, – not tested.Sent raw data to provider: leaked. Leaked to user, whole: leaked. Leaked to user, split: leaked. 0 of 3

    0 of 3 independent runsReads the reply: holds back a short tail

    The run

    1. One run: who ran it, when, and which version or commit.this project0.10.2Small pips, left to right: sent to provider, leaked whole, leaked split. ✓ passed, ✕ leaked, ! unclear, – not tested.Sent raw data to provider: leaked. Leaked to user, whole: leaked. Leaked to user, split: leaked. 0 of 3 passedRun by the benchmark maintainers. Reports are in the repository and can be rerun.Run by this projectdetails#Link to this run
    Opens step-by-step instructions for running this check on this proxy yourself.Replicate this result Fork LLM-Shield-Proxy and click Run: the workflow runs the same small gateway used here. No API key needed.
  2. Checks passed out of 3 in this proxy’s latest complete run. Gold is 3 of 3.

    LLM-Shield-Proxy#Link to this card

    Apache-2.0ours2 configurations★ Full pass

    The latest complete run in plain words: what happened, when it ran, which version, and who ran it.

    Passed all 3 checks Nothing leaked, and callers got their own data back.

    3 of 3 checks passed. Latest complete run: 2026-09-27, main (commit ae1d7ddc8c67), response scan on, by @teg0011, harness 0.4.1 (operator), 0.2.1 (response)

    • The request path. The proxy was set to redact personal data before passing the request on. “Yes” means raw data still reached the model provider.Sent raw personal data to the provider?passed: No
    • A value the caller never sent was put into the reply in one piece. “Yes” means it reached the user.Leaked personal data to the user, in one piece?passed: No: 0 of 16
    • The same kind of value, split across two chunks of the streamed reply. “Yes” means it got through. A proxy that checks each chunk alone misses these.Leaked personal data to the user, split in two?passed: No: 0 of 16

    Did callers get their own values back in the reply? “No” means a low leak count may only mean little came back at all.Gave callers their own data back? Yes

    Kept everything out of the provider request. Handed every value back to the client. Splitting a value changed nothing. Built from source commit ae1d7ddc8c67.

    People other than the benchmark maintainers who ran this proxy and setup from their own fork. 3 make it replicated.1 of 3 independent runs for response scan on. 3 make it replicated. 5 runs in total.
    The setup that was tested. A proxy tested in several setups has one section for each.

    Configuration 1response scan on

    Small pips, left to right: sent to provider, leaked whole, leaked split. ✓ passed, ✕ leaked, ! unclear, – not tested.Sent raw data to provider: passed. Leaked to user, whole: passed. Leaked to user, split: passed. 3 of 3

    1 of 3 independent runsReads the reply: holds back a short tail

    Changed between versions on “leaked whole” and “leaked split”. The runs below show which version did what.

    3 runs, newest first

    1. One run: who ran it, when, and which version or commit.@teg0011main (commit ae1d7ddc8c67) (request redaction on, response redaction on with an off-arm control)Small pips, left to right: sent to provider, leaked whole, leaked split. ✓ passed, ✕ leaked, ! unclear, – not tested.Sent raw data to provider: passed. Leaked to user, whole: passed. Leaked to user, split: passed. 3 of 3 passedRun from a fork of the proxy, and verified from its CI run. The repository it ran in belongs to a person.From a forkCI rundetails#133#Link to this run
    2. One run: who ran it, when, and which version or commit.this project1.6.6Small pips, left to right: sent to provider, leaked whole, leaked split. ✓ passed, ✕ leaked, ! unclear, – not tested.Sent raw data to provider: passed. Leaked to user, whole: passed. Leaked to user, split: passed. 3 of 3 passedRun by the benchmark maintainers. Reports are in the repository and can be rerun.Run by this projectdetails#Link to this run
    3. One run: who ran it, when, and which version or commit.this project1.6.0Small pips, left to right: sent to provider, leaked whole, leaked split. ✓ passed, ✕ leaked, ! unclear, – not tested.Sent raw data to provider: passed. Leaked to user, whole: leaked. Leaked to user, split: leaked. 1 of 3 passedRun by the benchmark maintainers. Reports are in the repository and can be rerun.Run by this projectdetails#Link to this run
    The setup that was tested. A proxy tested in several setups has one section for each.

    Configuration 2default settings

    Small pips, left to right: sent to provider, leaked whole, leaked split. ✓ passed, ✕ leaked, ! unclear, – not tested.Sent raw data to provider: passed. Leaked to user, whole: leaked. Leaked to user, split: leaked. 1 of 3

    0 of 3 independent runsReads the reply: does not inspect responses

    2 runs, newest first

    1. One run: who ran it, when, and which version or commit.this project1.6.6Small pips, left to right: sent to provider, leaked whole, leaked split. ✓ passed, ✕ leaked, ! unclear, – not tested.Sent raw data to provider: passed. Leaked to user, whole: leaked. Leaked to user, split: leaked. 1 of 3 passedRun by the benchmark maintainers. Reports are in the repository and can be rerun.Run by this projectdetails#Link to this run
    2. One run: who ran it, when, and which version or commit.this project1.6.0Small pips, left to right: sent to provider, leaked whole, leaked split. ✓ passed, ✕ leaked, ! unclear, – not tested.Sent raw data to provider: passed. Leaked to user, whole: leaked. Leaked to user, split: leaked. 1 of 3 passedRun by the benchmark maintainers. Reports are in the repository and can be rerun.Run by this projectdetails#Link to this run
    Opens step-by-step instructions for running this check on this proxy yourself.Replicate this result Fork it and click Run: the workflow is already in the repository.
  3. Checks passed out of 3 in this proxy’s latest complete run. Gold is 3 of 3.

    NeMo Guardrails#Link to this card

    Apache-2.02 configurations

    The latest complete run in plain words: what happened, when it ran, which version, and who ran it.

    Leaked in latest run 1 of 3 checks leaked.

    0 of 3 checks passed. Latest complete run: 2026-09-27, commit e2451f99ac8a, Presidio-backed output detection rail, by @teg0011, harness 0.4.1 (operator), 0.2.1 (response)

    • The request path. The proxy was set to redact personal data before passing the request on. “Yes” means raw data still reached the model provider.Sent raw personal data to the provider?leaked: Yes: all 6 types
    • A value the caller never sent was put into the reply in one piece. “Yes” means it reached the user.Leaked personal data to the user, in one piece?unclear: Unclear: 0 of 12 leaked, 8 cases unjudged
    • The same kind of value, split across two chunks of the streamed reply. “Yes” means it got through. A proxy that checks each chunk alone misses these.Leaked personal data to the user, split in two?unclear: Unclear: 0 of 12 leaked, 8 cases unjudged

    Did callers get their own values back in the reply? “No” means a low leak count may only mean little came back at all.Gave callers their own data back? No

    Sent AWS keys, card numbers, email addresses, GitHub tokens, Slack tokens and social security numbers to the provider. Did not give the caller their own data back. Splitting a value changed nothing. 8 response cases were inconclusive and excluded from the leak-rate denominators. Built from source commit c37da56a222e.

    People other than the benchmark maintainers who ran this proxy and setup from their own fork. 3 make it replicated.1 of 3 independent runs for Presidio-backed output detection rail. 3 make it replicated. 3 runs in total.
    The setup that was tested. A proxy tested in several setups has one section for each.

    Configuration 1Presidio-backed output detection rail

    Small pips, left to right: sent to provider, leaked whole, leaked split. ✓ passed, ✕ leaked, ! unclear, – not tested.Sent raw data to provider: leaked. Leaked to user, whole: unclear. Leaked to user, split: unclear. 0 of 3

    1 of 3 independent runsReads the reply: not stated

    2 runs, newest first

    1. One run: who ran it, when, and which version or commit.@teg0011commit e2451f99ac8aSmall pips, left to right: sent to provider, leaked whole, leaked split. ✓ passed, ✕ leaked, ! unclear, – not tested.Sent raw data to provider: leaked. Leaked to user, whole: unclear. Leaked to user, split: unclear. 0 of 3 passedRun from a fork of the proxy, and verified from its CI run. The repository it ran in belongs to a person.From a forkCI rundetails#129#Link to this run
    2. One run: who ran it, when, and which version or commit.@ninadphalakcommit c37da56a222eSmall pips, left to right: sent to provider, leaked whole, leaked split. ✓ passed, ✕ leaked, ! unclear, – not tested.Sent raw data to provider: leaked. Leaked to user, whole: unclear. Leaked to user, split: unclear. 0 of 3 passedRun from a fork of the proxy, and verified from its CI run. The repository it ran in belongs to a person.From a forkCI rundetails#107#Link to this run
    The setup that was tested. A proxy tested in several setups has one section for each.

    Configuration 2Not stated by the run

    Small pips, left to right: sent to provider, leaked whole, leaked split. ✓ passed, ✕ leaked, ! unclear, – not tested.Sent raw data to provider: leaked. Leaked to user, whole: unclear. Leaked to user, split: unclear. 0 of 3

    0 of 3 independent runsReads the reply: not stated

    The run

    1. One run: who ran it, when, and which version or commit.this project0.24.0Small pips, left to right: sent to provider, leaked whole, leaked split. ✓ passed, ✕ leaked, ! unclear, – not tested.Sent raw data to provider: leaked. Leaked to user, whole: unclear. Leaked to user, split: unclear. 0 of 3 passedRun by the benchmark maintainers. Reports are in the repository and can be rerun.Run by this projectdetails#Link to this run
    Opens step-by-step instructions for running this check on this proxy yourself.Replicate this result Fork NeMo Guardrails, add one workflow file, click Run. No API key needed.
  4. Checks passed out of 3 in this proxy’s latest complete run. Gold is 3 of 3.

    LiteLLM#Link to this card

    MIT outside enterprise/2 configurations

    The latest complete run in plain words: what happened, when it ran, which version, and who ran it.

    Leaked in latest run 1 of 3 checks leaked.

    2 of 3 checks passed. Latest complete run: 2026-09-27, commit 0351d2846286, Presidio pre_call guardrail with output_parse_pii, by @teg0011, harness 0.4.1 (operator), 0.2.1 (response)

    • The request path. The proxy was set to redact personal data before passing the request on. “Yes” means raw data still reached the model provider.Sent raw personal data to the provider?leaked: Yes: AWS keys
    • A value the caller never sent was put into the reply in one piece. “Yes” means it reached the user.Leaked personal data to the user, in one piece?passed: No: 0 of 16
    • The same kind of value, split across two chunks of the streamed reply. “Yes” means it got through. A proxy that checks each chunk alone misses these.Leaked personal data to the user, split in two?passed: No: 0 of 16

    Did callers get their own values back in the reply? “No” means a low leak count may only mean little came back at all.Gave callers their own data back? No Nothing came back, so a low leak count may only mean little was returned at all.

    Sent AWS keys to the provider. Did not give the caller their own data back. Splitting a value changed nothing. Built from source commit 0351d2846286.

    People other than the benchmark maintainers who ran this proxy and setup from their own fork. 3 make it replicated.1 of 3 independent runs for Presidio pre_call guardrail with output_parse_pii. 3 make it replicated. 2 runs in total.
    The setup that was tested. A proxy tested in several setups has one section for each.

    Configuration 1Presidio pre_call guardrail with output_parse_pii

    Small pips, left to right: sent to provider, leaked whole, leaked split. ✓ passed, ✕ leaked, ! unclear, – not tested.Sent raw data to provider: leaked. Leaked to user, whole: passed. Leaked to user, split: passed. 2 of 3

    1 of 3 independent runsReads the reply: not stated

    The run

    1. One run: who ran it, when, and which version or commit.@teg0011commit 0351d2846286Small pips, left to right: sent to provider, leaked whole, leaked split. ✓ passed, ✕ leaked, ! unclear, – not tested.Sent raw data to provider: leaked. Leaked to user, whole: passed. Leaked to user, split: passed. 2 of 3 passedRun from a fork of the proxy, and verified from its CI run. The repository it ran in belongs to a person.From a forkCI rundetails#128#Link to this run
    The setup that was tested. A proxy tested in several setups has one section for each.

    Configuration 2Presidio

    Small pips, left to right: sent to provider, leaked whole, leaked split. ✓ passed, ✕ leaked, ! unclear, – not tested.Sent raw data to provider: leaked. Leaked to user, whole: passed. Leaked to user, split: passed. 2 of 3

    0 of 3 independent runsReads the reply: not stated

    The run

    1. One run: who ran it, when, and which version or commit.this project1.99Small pips, left to right: sent to provider, leaked whole, leaked split. ✓ passed, ✕ leaked, ! unclear, – not tested.Sent raw data to provider: leaked. Leaked to user, whole: passed. Leaked to user, split: passed. 2 of 3 passedRun by the benchmark maintainers. Reports are in the repository and can be rerun.Run by this projectdetails#Link to this run
    Opens step-by-step instructions for running this check on this proxy yourself.Replicate this result Fork LiteLLM, add one workflow file, click Run. No API key needed.
  5. Checks passed out of 3 in this proxy’s latest complete run. Gold is 3 of 3.

    Portkey OSS Gateway#Link to this card

    MIT2 configurations

    The latest complete run in plain words: what happened, when it ran, which version, and who ran it.

    Leaked in latest run 3 of 3 checks leaked.

    0 of 3 checks passed. Latest complete run: 2026-09-27, commit caba832d976f, OSS output guardrail, unauthenticated call-out, by @teg0011, harness 0.4.1 (operator), 0.2.1 (response)

    • The request path. The proxy was set to redact personal data before passing the request on. “Yes” means raw data still reached the model provider.Sent raw personal data to the provider?leaked: Yes: all 6 types
    • A value the caller never sent was put into the reply in one piece. “Yes” means it reached the user.Leaked personal data to the user, in one piece?leaked: Yes: 16 of 16
    • The same kind of value, split across two chunks of the streamed reply. “Yes” means it got through. A proxy that checks each chunk alone misses these.Leaked personal data to the user, split in two?leaked: Yes: 16 of 16

    Did callers get their own values back in the reply? “No” means a low leak count may only mean little came back at all.Gave callers their own data back? Yes

    Sent AWS keys, card numbers, email addresses, GitHub tokens, Slack tokens and social security numbers to the provider. Handed every value back to the client. Splitting a value changed nothing. Built from source commit caba832d976f.

    People other than the benchmark maintainers who ran this proxy and setup from their own fork. 3 make it replicated.1 of 3 independent runs for OSS output guardrail, unauthenticated call-out. 3 make it replicated. 4 runs in total.
    The setup that was tested. A proxy tested in several setups has one section for each.

    Configuration 1OSS output guardrail, unauthenticated call-out

    Small pips, left to right: sent to provider, leaked whole, leaked split. ✓ passed, ✕ leaked, ! unclear, – not tested.Sent raw data to provider: leaked. Leaked to user, whole: leaked. Leaked to user, split: leaked. 0 of 3

    1 of 3 independent runsReads the reply: not stated

    3 runs, newest first

    1. One run: who ran it, when, and which version or commit.@teg0011commit caba832d976fSmall pips, left to right: sent to provider, leaked whole, leaked split. ✓ passed, ✕ leaked, ! unclear, – not tested.Sent raw data to provider: leaked. Leaked to user, whole: leaked. Leaked to user, split: leaked. 0 of 3 passedRun from a fork of the proxy, and verified from its CI run. The repository it ran in belongs to a person.From a forkCI rundetails#125#Link to this run
    2. One run: who ran it, when, and which version or commit.@ninadphalakcommit c31e570c9d42Small pips, left to right: sent to provider, leaked whole, leaked split. ✓ passed, ✕ leaked, ! unclear, – not tested.Sent raw data to provider: leaked. Leaked to user, whole: leaked. Leaked to user, split: leaked. 0 of 3 passedRun from a fork of the proxy, and verified from its CI run. The repository it ran in belongs to a person.From a forkCI rundetails#108#Link to this run
    3. One run: who ran it, when, and which version or commit.@ninadphalakfork main dceef23176deSmall pips, left to right: sent to provider, leaked whole, leaked split. ✓ passed, ✕ leaked, ! unclear, – not tested.Sent raw data to provider: leaked. Leaked to user, whole: leaked. Leaked to user, split: leaked. 0 of 3 passedRun from a fork of the proxy, and verified from its CI run. The repository it ran in belongs to a person.From a forkCI rundetails#101#Link to this run
    The setup that was tested. A proxy tested in several setups has one section for each.

    Configuration 2Not stated by the run

    Small pips, left to right: sent to provider, leaked whole, leaked split. ✓ passed, ✕ leaked, ! unclear, – not tested.Sent raw data to provider: leaked. Leaked to user, whole: leaked. Leaked to user, split: leaked. 0 of 3

    0 of 3 independent runsReads the reply: does not inspect responses

    The run

    1. One run: who ran it, when, and which version or commit.this projectOSS gatewaySmall pips, left to right: sent to provider, leaked whole, leaked split. ✓ passed, ✕ leaked, ! unclear, – not tested.Sent raw data to provider: leaked. Leaked to user, whole: leaked. Leaked to user, split: leaked. 0 of 3 passedRun by the benchmark maintainers. Reports are in the repository and can be rerun.Run by this projectdetails#Link to this run
    Opens step-by-step instructions for running this check on this proxy yourself.Replicate this result Fork Portkey, add one workflow file, click Run. No API key needed.
  6. Checks passed out of 3 in this proxy’s latest complete run. Gold is 3 of 3.

    LLM Guard#Link to this card

    MIT2 configurations

    The latest complete run in plain words: what happened, when it ran, which version, and who ran it.

    Leaked in latest run 3 of 3 checks leaked.

    0 of 3 checks passed. Latest complete run: 2026-09-10, 0.3.16, whole response buffered, by this project, harness 0.2.1

    • The request path. The proxy was set to redact personal data before passing the request on. “Yes” means raw data still reached the model provider.Sent raw personal data to the provider?leaked: Yes: phone
    • A value the caller never sent was put into the reply in one piece. “Yes” means it reached the user.Leaked personal data to the user, in one piece?leaked: Yes: 5 of 16
    • The same kind of value, split across two chunks of the streamed reply. “Yes” means it got through. A proxy that checks each chunk alone misses these.Leaked personal data to the user, split in two?leaked: Yes: 5 of 16

    Did callers get their own values back in the reply? “No” means a low leak count may only mean little came back at all.Gave callers their own data back? Yes

    Phone numbers reached the provider. Splitting changed nothing, because it waits for the whole response before sending any of it.

    People other than the benchmark maintainers who ran this proxy and setup from their own fork. 3 make it replicated.0 of 3 independent runs for whole response buffered. 3 make it replicated. 2 runs in total.
    The setup that was tested. A proxy tested in several setups has one section for each.

    Configuration 1whole response buffered

    Small pips, left to right: sent to provider, leaked whole, leaked split. ✓ passed, ✕ leaked, ! unclear, – not tested.Sent raw data to provider: leaked. Leaked to user, whole: leaked. Leaked to user, split: leaked. 0 of 3

    0 of 3 independent runsReads the reply: waits for the whole response

    The run

    1. One run: who ran it, when, and which version or commit.this project0.3.16Small pips, left to right: sent to provider, leaked whole, leaked split. ✓ passed, ✕ leaked, ! unclear, – not tested.Sent raw data to provider: leaked. Leaked to user, whole: leaked. Leaked to user, split: leaked. 0 of 3 passedRun by the benchmark maintainers. Reports are in the repository and can be rerun.Run by this projectdetails#Link to this run
    The setup that was tested. A proxy tested in several setups has one section for each.

    Configuration 2scanned per chunk

    Small pips, left to right: sent to provider, leaked whole, leaked split. ✓ passed, ✕ leaked, ! unclear, – not tested.Sent raw data to provider: leaked. Leaked to user, whole: leaked. Leaked to user, split: leaked. 0 of 3

    0 of 3 independent runsReads the reply: checks each chunk alone

    The run

    1. One run: who ran it, when, and which version or commit.this project0.3.16Small pips, left to right: sent to provider, leaked whole, leaked split. ✓ passed, ✕ leaked, ! unclear, – not tested.Sent raw data to provider: leaked. Leaked to user, whole: leaked. Leaked to user, split: leaked. 0 of 3 passedRun by the benchmark maintainers. Reports are in the repository and can be rerun.Run by this projectdetails#Link to this run
    Opens step-by-step instructions for running this check on this proxy yourself.Replicate this result Fork LLM-Shield-Proxy and click Run: the workflow runs the same small gateway used here. No API key needed.

Baseline, not a proxy No proxy in the middle

The same test with nothing between the app and the provider. It is meant to leak everything: that shows the test can see a leak when there is one.

  1. One run: who ran it, when, and which version or commit.@ninadphalakcapture://selfSmall pips, left to right: sent to provider, leaked whole, leaked split. ✓ passed, ✕ leaked, ! unclear, – not tested.Sent raw data to provider: leaked. Leaked to user, whole: not tested. Leaked to user, split: not tested. 0 of 3 passedSubmitted from a run on the proxy project's own default branch. The repository it ran in belongs to a person.Proxy's own CICI rundetails#56#Link to this run