Deciding Before You Know: What Willison's LiteLLM Response Reveals About Incident Response

Simon Willison published a minute-by-minute account of his response to the LiteLLM malware attack on March 26, 2026. The incident itself, a compromised litellm release that shipped a malicious .pth file to steal LLM API credentials from every Python invocation, has been covered thoroughly elsewhere. What deserves its own attention is what the write-up format reveals about incident response as a discipline, and specifically about a class of incident that the standard playbooks are not written for.

Two Incidents That Happened to Coincide

Before getting to the response, it’s worth separating what actually happened. The LiteLLM event was two distinct attacks.

The first was a compromised PyPI release: version 1.82.8 shipped a file named litellm_init.pth in site-packages. Python’s site module processes .pth files on every interpreter startup, and any line beginning with import is executed as code rather than treated as a path entry. The file contained a base64-obfuscated credential stealer that exfiltrated environment variables to an attacker-controlled server silently, on every subsequent Python invocation, for as long as the compromised version remained installed.

The second was a breach of BerriAI’s managed LiteLLM service, affecting 47,000 users. Different threat model, different exposure, different remediation.

These overlapped in time, which made them harder to reason about clearly at the moment of discovery. A developer who ran the managed service and also had the PyPI package installed locally faced a compound exposure. A developer who only used the package had no managed service exposure at all. Disentangling the two is a prerequisite for scoping the response correctly, and Willison’s real-time account captures the work of doing that disentangling under pressure.

What You Don’t Know When You Need to Know It

Most incident response guidance assumes you are the organization that was breached. You own the logs. You can determine when an intrusion started from system events. You can audit access, check for lateral movement, and bound the blast radius through evidence you control. Frameworks like NIST SP 800-61 are built around this model.

Downstream consumers of a compromised package are in a fundamentally different position. You did not own the build pipeline. You cannot pull the attacker’s exfiltration logs. You cannot confirm whether your specific machine was successfully targeted or whether the outbound request failed due to a network condition. The .pth mechanism in this attack was designed to suppress all errors, which means there is no traceback, no failed request, no system alert. You are trying to determine your exposure using only the information that predates your knowledge of the incident.

Specifically, you know:

• Which environments you installed the package into
• When you installed it
• Which credentials were likely present in those environments
• What your provider dashboards show for the relevant time window

You do not know:

• Whether the exfiltration requests succeeded
• Whether your keys were actively used after capture
• Which environments you may have forgotten, including Docker images built during the window, CI runners, or colleague machines

This is not a complete picture. The question is how to act rationally with an incomplete one.

The Case for Rotation Before Confirmation

Willison’s account makes visible a priority ordering that is easy to state but hard to execute under pressure: rotate first, investigate second.

The asymmetry that justifies this is straightforward. Revoking and reissuing an API key costs minutes. OpenAI’s key management, Anthropic’s console, AWS IAM, and the credential dashboards for every major LLM provider all support instant revocation with self-service flows. A valid key being actively used by an attacker compounds its damage by the minute. The cost of rotating a key that turns out not to have been compromised is negligible. The cost of not rotating one that was is not.

The temptation in a suspected compromise is to wait for confirmation before acting. The natural question is: did this actually affect me? But for a credential stealer that runs silently, confirmation is not available from the victim’s side. The conservative answer, to treat confirmed installation of the compromised version as equivalent to confirmed capture, is uncomfortable because it requires acting on inference rather than evidence. Willison’s account makes the alternative visible: if you wait for the logs that don’t exist, you have not been more careful, you have just spent more time with a potentially live credential in attacker hands.

What Retrospective Write-Ups Smooth Over

The reason a minute-by-minute account produces something qualitatively different from a post-mortem is that post-mortems are written by someone who knows the outcome.

When you reconstruct an incident response after the fact, you know which hypotheses were correct. The ones that turned out to be wrong get compressed or omitted. The decision points that were genuinely ambiguous get resolved into clean forks. The responder’s uncertainty, which was real and consequential at the time, disappears from the narrative. What remains reads as a sequence of rational steps rather than a series of guesses made under pressure.

Willison’s real-time account preserves the uncertainty because it was written into it. It shows that investigating scope while simultaneously rotating credentials is the actual shape of the task, not a sequential process. It shows the specific questions you run into when the standard IR frameworks assume access you do not have. It shows the false starts.

For practitioners, this is more instructive than a clean post-mortem. Security training relies heavily on tabletop exercises that simulate incident scenarios, but the simulation collapses the epistemic situation: participants in a tabletop know they are in a training exercise and know the scenario has a defined scope. A real account of someone working through a real incident with incomplete information is harder to come by and more useful.

The Check Worth Running Right Now

For anyone using Python who wants to verify their own exposure to .pth-based malware, past or future, the check is direct:

ls $(python -c "import site; print(site.getsitepackages()[0])")/*.pth

For each .pth file that appears, open it and look at the content. Legitimate .pth files contain either path entries, one per line, or comments. A file with base64 content, an inline exec call, or a semicolon-chained import chain is not legitimate. A file named after its parent package with a suffix like _init, _config, or _setup that contains anything executable is worth treating as suspicious until proven otherwise.

Socket and Phylum have both documented this attack surface over multiple years of PyPI malware analysis. It remains effective because standard tooling does not surface it. pip show, pip list, and pip audit will not reveal a malicious .pth file. The check above is something worth adding to deployment verification scripts for any environment that handles credentials.

Why AI Tooling Changes the Stakes

The LiteLLM incident sits in a lineage that includes the pytorch-nightly dependency confusion attack in late 2022 and the XZ Utils backdoor in early 2024. Supply chain attacks against trusted open source infrastructure are not new.

What has changed is the value of the credentials that AI tooling packages hold in scope. LLM API keys carry spending limits and, in enterprise deployments, access to fine-tuned models trained on proprietary data. A developer whose LiteLLM installation was compromised did not just lose some credentials; they potentially exposed every prompt their application sent to every model they were routing through, their system prompts, their model selection logic, and their spending patterns across multiple providers. This is harder to rotate than a key and harder to quantify as a loss.

Packages like LiteLLM, LangChain, and LangFuse occupy an aggregation position in the AI tooling stack. They sit between your application and your providers, and they see everything. The correct security posture for that class of dependency is the same as for authentication middleware: pinned to a specific hash, updated deliberately rather than silently, and treated as critical infrastructure. Willison’s write-up is a useful forcing function for teams to ask whether their current practices reflect that.

PyPI supports hash-verified installs. pip-compile from the pip-tools package makes it straightforward to maintain a locked requirements file with hashes across all transitive dependencies. The tooling exists. The question is whether the cultural norm around AI tooling has caught up to the risk the tooling’s position actually represents.