PyTorch supply chain attack: Dependency confusion burns DevOps

A classic dependency confusion attack revealed itself last week. The PyTorch open source software supply chain was compromised by a hacker publishing a malicious torchtriton clone on PyPI.

The perp was pretending to be an ethical researcher. However, the alarm was raised by their efforts to obfuscate the malware and exfiltrate sensitive data. Not only that, but the stolen data could have been viewed in transit.

It’s proof, once again, that DevOps needs to get serious about mitigation. In this week’s Secure Software Blogwatch, we have nothing to lose but our chains.

Your humble blogwatcher curated these bloggy bits for your entertainment. Not to mention: Update your software.

Flaming security posture

What’s the craic? Careful with that Ax Sharma — “PyTorch discloses malicious dependency chain compromise over holidays ”:

“Malicious payload”
PyTorch has identified a malicious dependency with the same name as the framework's 'torchtriton' library. … From computer vision to natural language processing, the open source machine learning framework PyTorch has gained prominence in both commercial and academic realms. … This type of supply chain attack is known as "dependency confusion."
…
The warning follows a … dependency that appeared over the holidays on the Python Package Index (PyPI) registry, the official third-party software repository for Python. … The malicious 'torchtriton' dependency on PyPI shares name with the official library. … But, when fetching dependencies in the Python ecosystem, PyPI normally takes precedence, causing the malicious package to get pulled on your machine instead of PyTorch's legitimate one.
…
This won't be the first time … a hacker claims that their actions constitute ethical research, just as they are caught exfiltrating secrets. … Unlike several research packages and PoC exploits that are conspicuous in their intent and behavior, 'torchtriton' employs known anti-VM techniques to evade detection. More importantly, the malicious payload is obfuscated.

Who did what? Jeff Burt blurts, “PyTorch dependency poisoned with malicious code”:

“Securing software repositories”

Developers who last week downloaded the nightly builds of the … PyTorch framework also unknowingly installed a malicious version of the torchtriton dependency. … A person taking responsibility for the incident said it was part of a security research project that went awry. They apologized, saying they erred in not making this clear.

…

The [exfiltrated] sensitive data includes nameservers, hostnames, the current username and working directory name. In addition, it accessed a range of files, including /etc/hosts, /etc/password/, the first 1,000 files in $HOME/*, $HOME/.gitconfig, and $HOME/.ssh*. … The PyTorch maintainers have taken several steps to fix the issue, including removing torchtriton as a dependency … they registered a dummy package on PyPI [and] packages that depend on torchtriton were removed from the package indices.

…

PyPI and other open source code repositories have become a target in supply chain attacks. … The Open Source Security Foundation [has] created a community-based working group to address the issue of securing software repositories.

Horse’s mouth? The PyTorch Team — “Compromised PyTorch-nightly dependency chain”:

“Delete the malicious version”
If you installed PyTorch-nightly on Linux via pip between December 25, 2022 and December 30, 2022, please uninstall it and torchtriton immediately. … Since the PyPI index takes precedence, this malicious package was being installed instead of the version from our official repository. This design enables somebody to register a package by the same name as one that exists in a third party index, and pip will install their version by default.
…
We have reached out to the PyPI security team to get proper ownership of the torchtriton package on PyPI and to delete the malicious version. … All nightly packages that depend on torchtriton have been removed from our package indices … until further notice.

What’s the solution? See p-e-w’s laser focus: [You’re fired—Ed.]

“I doubt there is a good solution”

The only thing that surprises me about such attacks is that they don't happen more often. With Python and Node.js it's now the norm that large packages have hundreds of transitive dependencies.

…

By installing a single package, you're potentially trusting hundreds of authors. And yet we do want all these packages, because they solve specific problems in an optimized manner, and we do want anyone to be able to publish packages. … We all need tensor algebra, video decoding, cutting-edge network protocols, compression, cryptography, dozens of data exchange formats, syscall bindings for three or more platforms, fuzzing, containers, and I don't know what else.

…

"Dependency hell" is here to stay. … I doubt there is a good solution here.

It’s the tip of the iceberg, says u/Gentleman-Tech:

“Other people don't play nice”

I firmly believe that we're going to see a huge wave of supply chain attacks over the next decade or so, and it's going to change the way we do open source. Just as IP, HTTP and the other core internet protocols had no security elements because everyone just assumed everyone else would play nice, our current OSS protocols have no security elements and assume everyone else is going to play nice.

We're going to learn, again, that other people don't play nice. Every dependency is a security risk.

It’s worse than we first thought, according to Paul Ducklin — @duckblog:

This malware deliberately steals your data… and transmits it scrambled [but] not encrypted … so anyone on your network path who recorded it can trivially decode it.

So? So ShivShankaran seems somewhat concerned:

The malicious package uploads your ssh private keys. … This is extremely concerning. … Judging from the package installation stats this was installed around 2,500 times.

Meanwhile, u/DoughyInTheMiddle gives their impression of the perp’s response:

The full version of that notice was like, "Yeah, if your systems were compromised, sorry. Our bad. Whoopsie doodles!"

And Finally:

Patch now

Previously in And finally

You have been reading Secure Software Blogwatch by Richi Jennings. Richi curates the best bloggy bits, finest forums, and weirdest websites … so you don’t have to. Hate mail may be directed to @RiCHi or ssbw@richi.uk. Ask your doctor before reading. Your mileage may vary. Past performance is no guarantee of future results. Do not stare into laser with remaining eye. E&OE. 30.

Image sauce: Aziz Acharki (via Unsplash; leveled and cropped)