All postsEngineering

Your LLM gateway is a credential vault, so run it like one

The LiteLLM compromise showed what an LLM proxy holds. How to isolate the gateway, pin it properly and stop a payload from phoning home.

Johnny Unar7 min read

01

what the gateway holds

If your product talks to more than one model provider, there's a good chance it goes through an LLM gateway: LiteLLM, Portkey, a homegrown proxy, or the gateway feature of your cloud. The job is useful, because one service handles routing, retries, budgets and logging for every model call. The side effect is that this one service holds the OpenAI key, the Anthropic key, the Bedrock IAM role, the Vertex service account and often a database URL for its own spend tracking. In most stacks no other process holds that many third-party credentials at once, yet the gateway usually gets installed like a library, with one pip install and no threat model.

The LiteLLM incident in March 2026 is the clearest example of why that matters. On 24 March, malicious litellm 1.82.7 and 1.82.8 were uploaded to PyPI. LiteLLM's own security update says the compromise likely came through the Trivy scanner used in its CI security-scanning workflow, which researchers tie to the TeamPCP campaign that rewrote Trivy's GitHub Action tags on 19 March, and Snyk's analysis says the stolen credential was the PyPI publishing token in the GitHub Actions environment. Version 1.82.7 hid a credential stealer inside litellm/proxy/proxy_server.py, so it ran when anything imported the proxy module. Version 1.82.8 kept that and added litellm_init.pth to site-packages, which Python executes at interpreter startup, with no import needed, every time any Python process starts in that environment. The payload collected environment variables, SSH keys, cloud credentials, Kubernetes tokens and database passwords and posted them, encrypted, to models.litellm.cloud, a lookalike domain. PyPI quarantined the package within hours and the versions were removed.

LiteLLM's advisory also lists who wasn't affected: users of the official proxy Docker image, which pins its dependencies, and anyone on 1.82.6 or earlier who didn't upgrade during that window. That list is the most useful part of the incident, because it shows which controls worked.

02

pins and cooldowns, not just hashes

The common advice after an incident like this is "use hash pinning", and it's worth being precise about what hashes do. A hash in your lockfile proves that the file you download today is byte for byte the file you locked. That protects you if someone replaces the artifact behind an existing version. The LiteLLM attack didn't do that. It published new version numbers, and the .pth file was listed properly in the wheel's RECORD, so a hash check of 1.82.8 against 1.82.8's own published hashes would have passed. What kept people safe was an exact pin to a version that already existed, and not bumping it during those hours.

So the control that matters is that nothing new enters your dependency tree without a person deciding it should, and nothing enters while it's hours old. With uv that means a lockfile you install from strictly, plus a release-age cutoff:

toml
1[project]
2name = "gateway"
3version = "0.1.0"
4requires-python = ">=3.12"
5dependencies = ["litellm[proxy]==1.82.6"]
6
7[tool.uv]
8exclude-newer = "7 days"

exclude-newer accepts a timestamp or a relative duration like "7 days" or P7D, and the resolver ignores any distribution uploaded more recently than that, so a release has to survive a week of public scrutiny before your next uv lock can pick it up. In CI, install with uv sync --locked, which fails if the lockfile doesn't match pyproject.toml instead of quietly re-resolving. If you're on pip, pip 26.0 added --uploaded-prior-to and pip 26.1 lets it take a relative duration such as P7D. Renovate's minimumReleaseAge does the same job for automated update PRs.

Pinning litellm[proxy]==1.82.7 today fails resolution outright, because the version no longer exists on PyPI. That's the behaviour you want from every pinned dependency: if the exact thing you reviewed isn't available, the build stops.

Keep hashes as well. uv.lock records them automatically, and uv pip compile requirements.in --generate-hashes followed by an install with --require-hashes gives you the same for a requirements file. They just answer a different question, and the bump PR is where a person should look at what changed and why litellm moved a patch version this morning.

03

where the gateway runs and who it runs as

The second question is what a compromised gateway process can reach. If it runs as a sidecar in your application pod, or shares a service account and node with the app, then a payload in the gateway sees the app's credentials too, and a bug in the app that gives an attacker code execution also gives them every provider key. Give the gateway its own deployment, its own service account and its own namespace, and inject provider keys only into that workload, ideally from a secrets manager at start-up rather than from a values file in the repository.

On the provider side, scope keys as tightly as each provider allows: separate keys or projects per environment, spending limits on every key, and no organisation-admin keys in the gateway at all. Gateways like LiteLLM can issue their own virtual keys with budgets to the services that call them, which means application code never sees a real provider key and you can revoke one service's access without rotating anything upstream.

Write the rotation runbook before you need it. List every secret the gateway holds, where each one is issued, who can rotate it and how long it takes. If a compromise like March's happens, the people rotating keys shouldn't be discovering for the first time that the Vertex service account was created by someone who has since left.

04

egress control is what stops the payload

Everything above reduces the chance that malicious code runs in the gateway. Egress control decides what happens when it runs anyway. The March payload needed one thing to succeed: an outbound HTTPS connection to an attacker-controlled domain. A gateway that can only reach the model providers it actually uses would have collected the secrets and failed to send them anywhere.

Kubernetes NetworkPolicy can't filter by hostname, only by IP, namespace and pod labels, and provider APIs sit behind CDNs whose addresses change. A practical pattern is to deny all egress from the gateway except DNS and a forward proxy, and let the proxy enforce the hostname allowlist:

yaml
1apiVersion: networking.k8s.io/v1
2kind: NetworkPolicy
3metadata:
4 name: llm-gateway-egress
5 namespace: llm-gateway
6spec:
7 podSelector:
8 matchLabels:
9 app: llm-gateway
10 policyTypes: ["Egress"]
11 egress:
12 - to:
13 - namespaceSelector:
14 matchLabels:
15 kubernetes.io/metadata.name: kube-system
16 podSelector:
17 matchLabels:
18 k8s-app: kube-dns
19 ports:
20 - { protocol: UDP, port: 53 }
21 - { protocol: TCP, port: 53 }
22 - to:
23 - namespaceSelector:
24 matchLabels:
25 kubernetes.io/metadata.name: egress-proxy
26 ports:
27 - { protocol: TCP, port: 3128 }

The proxy itself can be a few lines of Squid that only allow HTTPS CONNECT to the provider hosts:

text
1http_port 3128
2acl llm_providers dstdomain api.openai.com api.anthropic.com
3acl SSL_ports port 443
4acl CONNECT method CONNECT
5http_access deny CONNECT !SSL_ports
6http_access allow CONNECT llm_providers
7http_access deny all

Tested with Squid in Docker, a request through this proxy to api.anthropic.com reached Anthropic (and got the expected 401 without a key), while models.litellm.cloud and pypi.org were refused at the proxy. Point the gateway at it with HTTPS_PROXY, which httpx and most Python HTTP clients honour by default, and add your Postgres and Redis to the NetworkPolicy as separate rules. If you run Cilium or a service mesh, an FQDN-aware egress policy does the same job without a separate proxy. Start in log-only mode for a few days to learn what the gateway really talks to, then switch to deny.

05

the build pipeline that publishes it

If you publish your own packages, the LiteLLM chain is also a lesson about CI. The token was stolen from the environment of a job that ran a third-party scanner. Trusted publishing with OIDC, which PyPI supports, removes the long-lived token, and you should use it, but it doesn't make the job safe to share with untrusted code. In the TanStack npm compromise two months later, according to TanStack's postmortem, attacker code running in the release workflow read a short-lived OIDC token from the runner's memory and published with it.

The rule that covers both incidents is that scanners, linters and anything else from a third party run in jobs with no publishing rights and no secrets, and the job that holds id-token: write or a registry token runs only your own pinned build steps. That's more YAML and one more job, and it turns a compromised scanner into a failed build instead of a poisoned release.

Thanks for reading.
Johnny Unar

Written by

Johnny Unar

Keep reading

Stop patching.Start building.

Tell us what's slowing you down. We'll bring coffee and opinions.