A software supply chain meltdown: What we know about the XZ Trojan

Security experts are sounding alarms about what some are calling the most sophisticated supply chain attack ever carried out on an open source project: a malicious backdoor planted in xz/liblzma (part of the xz-utils package), a popular open source compression tool.

A months-long campaign of tampering and social engineering intended to plant malicious code in major Linux distributions is behind the compromise of the open-source compression library xz/liblzma called the XZ Trojan.

The details of the attack and the extent of the compromise are still emerging. Here’s what we know so far.

Join Webinar: Unraveling XZ: A Software Supply Chain Under SiegeSee Special Report: The State of Software Supply Chain Security (SSCS) 2024Download: State of SSCS

What happened in the XZ Trojan supply chain attack?

On Friday, Andres Freund, a Principal Software Engineer at Microsoft and PostgreSQL developer posted a message to Openwall, an online forum focused on open source security, to warn about a “backdoor in upstream xz/liblzma leading to ssh server compromise.” In the post, Freund described observing “a few odd symptoms around liblzma on Debian sid installations” in recent weeks. Specifically: he noticed that logins via ssh were slower and “taking a lot of CPU” as well as “generating valgrind errors” (Valgrind is a debugging tool).

Freund said he initially assumed he had stumbled on a compromise of the Debian linux package in question, but that the actual compromise was “upstream” from Debian: versions 5.6.0 and 5.6.1 of xz (aka liblzma) a well known open-source compression library.

The public disclosure led to a rapid response. CISA issued a security alert and the issue was assigned the identifier CVE-2024-3094. Vendors including Red Hat wrote to urge customers using vulnerable versions of the Fedora operating system to either cease use of the software (Fedora Rawhide) or revert to a safe version of the xz software (Fedora Linux 40).

What software is affected?

A wide range of open-source software implemented the affected versions of xz (5.6.0 and 5.6.1), as documented by the security firm Lacework. That includes Red Hat’s Fedora 40 and Fedora Rawhide Linux distributions, Debian unstable (Sid), Arch Linux, openSUSE Tumbleweed and MicroOS, and homebrew, an open source package manager for the MacOS operating system. Organizations using affected software are urged to revert to a known-good version of xz, such as 5.4.x.

What does xz do?

According to this write up over at GitHub, xz is used for compression, including “release tarballs, software packages, kernel images, and initramfs images.” The package is widely distributed and often included with Linux distributions for convenience.

How does the XZ Trojan work?

The malicious backdoor is still being analyzed, however a good amount of information is available from a variety of sources. Researchers said the backdoor enables affected systems to be accessed via remote, unprivileged systems connecting to public SSH ports. Those systems may then execute malicious code on the affected systems.

The FAQ on GitHub says that the malicious code was distributed via release tarballs (compressed files) that were altered substantially from previous versions in the Git repository to include the malicious backdoor. In particular, the version of the file build-to-host.m4, found in the altered release tarballs, calls a script that checks for various conditions like the architecture of the machine. According to research into the attack, the attacker appears to target systems using the amd64 architecture and running glibc using either Debian- or Red Hat derived distributions.

The backdoor discovered by Freund is sophisticated. It only springs into action when a predetermined set of criteria are met. If and when those conditions are met, the script unpacks malicious data hidden in files in the tests/ folder that were added to the xz git repository via a series of commits by the malicious author. Those contain obfuscated/encrypted bash stages and the binary backdoor in two test files: tests/files/bad-3-corrupt_lzma2.xz and tests/files/good-large_compressed.lzma.

ReversingLabs Spectra Assure flagged a sudden loss of application hardening functions in the malicious version of liblzma.

That unpacked malicious test data is used to modify the build process. At the same time, IFUNC- a legitimate function in glibc used for indirect function calls- is used (maliciously) to perform runtime hooking/redirection of OpenSSH's authentication routines.

After that, a malicious payload is injected into the source tree that activates if the running program has the process name /usr/sbin/sshd. Systems that put sshd in /usr/bin or another folder may, or may not be, vulnerable. It may be the case that the malicious code activates when other, as yet unobserved conditions are met, also. The malicious payload has been observed being loaded into sshd indirectly after which the RSA_public_decrypt function is redirected into a malicious implementation that can be used to bypass authentication. The exact purpose behind this isn’t yet clear and more research is being done to identify the purpose and function of the malicious payload.

Which systems are vulnerable to the backdoor?

According to the GitHub FAQ, vulnerable systems need to be:

• Using a .deb or .rpm based Linux distribution
• Using the glibc for IFUNC
• Using versions 5.6.0 or 5.6.1 of xz or liblzma installed. (The liblzma library is included with the xz-utils package)
• Using systemd on publicly accessible ssh

However, research is ongoing as to the length and extent of this compromise. Given the long and intimate involvement of the likely malicious author with the xz open source project, it is highly possible that versions of xz or liblzma prior to 5.6.0 contain malicious elements and that the reach of this malicious campaign is far wider.

Who is behind XP Trojan attack?

The identity of the malicious actor(s) behind the compromise of xz isn’t known. What is known is that this was a sophisticated and long-term campaign in which a malicious actor posed as a legitimate open source developer, Jia Tan (JiaT75), using an open source contributor account that was created in 2021and has submitted patches and feature updates to other projects. (Submissions that are now being analyzed.)

As Evan Boehs notes in his write-up of the xz incident, Tan’s involvement in the xz project is no accident. Indeed, starting in 2022 xz's sole maintainer, Lasse Collin, faced mounting pressure online from a number of newly minted developer accounts. His work on xz had tapered off, with Collin citing “long term mental health” issues and other conflicts as the cause of his reduced attention to the xz project. The developers were critical of the lack of activity on xz and pressured Collin to add a new maintainer to address their calls for expanded integrations, features and patches.

Through frequent, volunteer contributions, Jia Tan ingratiated him or herself with Collin starting in 2022. Soon after, the accounts pressuring Collin, with names like Jigar Kumar and Dennis Ens, went quiet, suggesting a coordinated campaign using “sock puppet” developer accounts to pressure Collin to accept help from the malicious JiaT75 account. Soon after, JiaT75 made their first commit to xz. The account eventually became the second most active contributor to the project.

By 2023, the JiaT75 account was merging their commits to xz, indicating that Tan was trusted as a maintainer of the xz project. By March of that year, the primary xz contact email in Google’s oss-fuzz was updated to be Jia’s, instead of Lasse Collin, while testing infrastructure that was subsequently leveraged by the exploit was committed, with code from another contributor: Hans Jansen another suspicious developer account with little track record of contributions before or after the work with xz.

The same developers pressuring Collin to accept help and contributing suspicious updates shifted to pressuring downstream organizations like Debian to integrate the latest (compromised) updates. And that's not all. In July, 2023, Jia Tan submitted a pull request in OSS-Fuzz, which is used to scan open source packages for malicious code. The request was to disable fuzzing of the IFUNC feature, making it less likely that OSS-Fuzz would identify the malicious tampering within the xz code.

The origin and purpose of this campaign isn’t known, however the subtlety and length of the operation point to sophisticated, possibly nation-state actors that were clearly looking to gain a foothold on systems running the affected open source operating systems.

What is the impact of this supply chain attack?

The impact of this compromise is still a matter of debate. At first glance, this attack looks like a “near miss,” that’s because the compromised code was spotted before it could be integrated into popular open source distributions. As a result, the impact of the compromise was felt by the subset of users who ‘updated religiously’ and had quickly adopted the latest versions of xz (5.6.0 or 5.6.1) or liblzma. Furthermore, the affected Linux distributions were development versions like Fedora Rawhide, Arch Linux and Debian Unstable that were unlikely to be deployed in production environments.

The apparent focus on AMD64 based systems and the reliance on the glibc library also limit the impact of the attack.

However, given the long history of the JiaT75 account with both the xz project and other open source projects, the potential exists that earlier iterations of xz or other packages — now deemed stable — actually contained suspicious or malicious code. For that reason, the full extent and impact of this campaign has yet to be determined.

Takeaways and recommendations

Obviously, those organizations using affected versions of the xz or xz-utils packages should downgrade to an known-good version. CISA recommends the XZ Utils 5.4.6 Stable version. Those organizations should also commence hunting for malicious activity, CISA has asked that any organizations that detect such activity should report their findings.

However, even organizations not directly impacted by this incident should continue to follow it closely, as the extent of this malicious campaign may broaden to include more stable versions of commonly used Linux distributions and/or additional open source packages with a broader reach than xz.

The bigger “fix” is for organizations to adopt tools and processes that allow them to identify signs of tampering and malicious features within both open source and commercial code used in their own development pipeline. For example, ReversingLabs' Spectra Assure platform flagged the sudden loss of application hardening functions between the last known good version of the liblzma library and the first known malicious version.

While not explicitly malicious, the regression of hardening features are unusual and should always be investigated further, said Tomislav Peričin, co-founder and Chief Software Architect at ReversingLabs.