Skip to main content

LLM-Shield-Proxy

LLM-Shield-Proxy is a self-hosted proxy for OpenAI-compatible LLM APIs. It replaces the personal data and secrets it detects (emails, card numbers, SSNs, API keys and more) before a request goes to the model provider, and puts the original values back into the streamed reply before your application sees it. Your application changes only its base_url and the key it sends.

Try it in a minute, with no API key​

pip install llm-shield-proxy

UPSTREAM_BASE_URL=http://127.0.0.1:8765 UPSTREAM_API_KEY=unused VALID_VIRTUAL_KEYS=sk-demo llm-shield-proxy --port 4000 &

pii-leak-benchmark selfcheck --target-base-url http://127.0.0.1:4000/v1 --target-api-key sk-demo

The second command starts the proxy. The third sends prompts full of synthetic personal data through it and plays the model provider, so it sees exactly what the proxy forwarded. It should print CLEAN. On Windows PowerShell, use the PowerShell version.

Use it with your application​

The proxy needs two keys:

  • VALID_VIRTUAL_KEYS: the keys your clients send to the proxy. Any other key gets a 401.
  • A provider key, here OPENAI_API_KEY: what the proxy sends upstream. Your clients never hold it.
export VALID_VIRTUAL_KEYS=sk-my-client-key
export OPENAI_API_KEY=sk-your-openai-key
llm-shield-proxy --host 127.0.0.1 --port 8000
from openai import OpenAI

client = OpenAI(api_key="sk-my-client-key", base_url="http://localhost:8000/v1")

How the pieces fit​

What it isWho uses itAnalogy
The specificationA written rulebook, with JSON schemas for the reports. No runtime code.Anyone defining what "safe" means for streamed redactionThe safety standard
pii-leak-benchmarkAn outside tester. It talks to a running gateway over HTTP, plays both the client and the model provider, and scores what leaked.Auditors, researchers, and teams checking a gateway they runThe crash-test lab
chunk-invarianceA test library a developer adds to their own test suite: the output must not depend on how the stream is cutDevelopers building streaming filtersThe gauge on the factory line
LLM-Shield-ProxyA working privacy gatewaySomeone who needs a filter, or a gateway to compare againstA car built to the standard

The proxy goes through the same lab as any other gateway: release 1.7.1 fixed two streamed-response redaction cases in the proxy that new stream-shape tests exposed. pip install pii-leak-benchmark and pip install chunk-invariance each install on their own, without the proxy. pip install llm-shield-proxy also installs pii-leak-benchmark.

Next steps​

You want toRead
Run it in Docker or KubernetesDeployment
See which data types it findsSupported types
Understand the designArchitecture
Know what it does not doLimitations
Put it in front of LiteLLM, Open WebUI or LangChainIntegrations
Check it yourselfLeak benchmark