A data compression library called XZ Utils that ships as part of major Linux distributions has had a backdoor inserted into it by one of its trusted maintainers. Luckily the backdoor, which enables remote code execution on systems that accept SSH connections, was only present in the latest versions of the library and was spotted before poisoned versions made it into stable Linux releases.
That said, some Linux distros, such as Arch Linux, that use a rolling update model as well as the development branches of Debian, Fedora OpenSUSE, and Alpine distributed the backdoored packages.
As a result, some developers and power users are likely using the impacted Linux distributions in their containers, virtual machines, and other projects for testing or even as the main operating systems on their desktop machines because they like having access to the latest versions of various Linux applications and tools.
If the carefully constructed backdoor, which is now tracked as CVE-2024-3094, had gone undiscovered for a few more months and made it into stable Linux distributions, countless production servers would likely have been impacted around the world. It could easily have been one of the highest-impact software supply chain breaches to date. Even so, it stands as one of the most sophisticated ones.
An example of how trust gets abused in open-source
The incident highlights how trust can be abused in the open-source ecosystem especially when many projects struggle with a lack of manpower and development resources, not to mention volunteers with cybersecurity knowledge and skills.
There are thousands of important applications, command-line utilities, and libraries that make up the Linux ecosystem and many of them are maintained by very small teams or even single individuals who have been doing it for years in their spare time. XZ Utils is just one of them.
“This might be the best-executed supply chain attack we’ve seen described in the open, and it’s a nightmare scenario: malicious, competent, authorized upstream in a widely used library,” security researcher and cryptographer Filippo Valsorda noted on Bluesky. “Looks like this got caught by chance. Wonder how long it would have taken otherwise.”
XZ Utils backdoor discovered by chance
The backdoor was first spotted by Andres Freund, a Microsoft software engineer working on PostgreSQL, who announced it on the popular Open Source Software Security mailing list on Friday. Freund found the issue more or less by accident, not while doing any intentional code security review.
He was in the process of running some benchmarking tests on a system in order to try to reduce noise when he noticed abnormally high CPU utilization spikes from the sshd process (the SSH service daemon) that were caused by login failures due to wrong usernames.
“Profiled sshd, showing lots of cpu time in liblzma, with perf unable to attribute it to a symbol,” he wrote on Mastodon. “Got suspicious. Recalled that I had seen an odd valgrind complaint in automated testing of postgres, a few weeks earlier, after package updates. Really required a lot of coincidences.”
Liblzma is a popular library that implements data compression and decompression using the Lempel-Ziv-Markov chain algorithm known as LZMA. The library is part of a package called XZ-Utils (previously LZMA Utils) which also includes the xz command-line utility for compressing and decompressing data.
Freund was doing his tests on a Debian Sid installation — Sid is the unstable branch of Debian — so he looked at the xz-utils package distributed from that branch’s repositories. He then found an obfuscated script that is executed at the end of the configure step of the installation process and traced it back to the original tarballs from the XZ-Utils project on GitHub.
“That line is *not* in the upstream source of build-to-host, nor is build-to-host used by xz in git,” Freund said in his report. “However, it is present in the tarballs released upstream, except for the ‘source code’ links, which I think GitHub generates directly from the repository contents.”
What does the XZ Utils backdoor do?
The obfuscated script is meant to inject malicious code into the sshd process, which is possible because liblzma is a dependency for certain builds of OpenSSH. The payload hooks the RSA_public_decrypt function, an OpenSSH function that is used to validate RSA signatures, and uses it to check a certain value (the RSA public modulus) from a RSA signature that is controlled by a connecting client.
The hooking code then takes this client-controlled value and decrypts it using a hard-coded ChaCha20 key. The resulting data has two parts: a digital signature generated using the Ed448 asymmetric elliptic curve signing algorithm and a string that represents shell commands to be executed on the system. The commands will only be executed if the signature passes validation using a hard-coded Ed448 public key.
However, the catch is that only the attackers have the private key needed to generate valid signatures. This ensures that only they can send rogue SSH requests to the backdoor that would result in the included shell commands being executed, basically ensuring that no one else aside from them can exploit the backdoor.
“The sophisticated nature of this attack and the use of highly future-proof crypto algorithms (Ed448 vs the more standard Ed25519) led many to believe that the attack may be a nation-state level cyberattack,” researchers from security firm JFrog noted in an analysis.
Who is affected by the XZ Utils backdoor?
The backdoor is present in versions 5.6.0 and 5.6.1 of xz-utils and particularly in the .deb and .rpm packages distributed as part of certain Linux distributions, including the following: Fedora 40 and 41 Rawhide (active development); Debian testing, unstable (sid) and experimental; Alpine Edge (active development); openSUSE Tumbleweed; as well as Kali Linux and Arch Linux which follow a rolling release or update model where non-security updates to applications and packages are released continuously as they become available instead of on a planned basis as part of major OS upgrades.
Users should refer to the guidance put out by their Linux distribution maintainers in their respective advisories. In some cases, it might be recommended to completely reinstall the operating system because it’s hard to know if the backdoor was actively exploited or whether malicious commands were executed on the system as a result and what those commands did.
How was the backdoor added?
XZ-Utils dates back to 2009 and was created by a developer named Lasse Collin who is known as Larhzu on GitHub. He also served as the sole maintainer of the project until around 2023 when another developer who identified as Jia Tan (JiaT75) received commit permissions and was added as a second maintainer. It is Jia Tan’s account that introduced the malicious code and signed the backdoored tarballs for versions 5.6.0 and 5.6.1.
While there’s a theoretical possibility that Jia Tan’s account was compromised, mounting evidence suggests that it’s more likely this is a fake identity and part of a well-planned and executed years-long software supply chain campaign.
The JiaT75 account was created on GitHub in 2021 and started making contributions to multiple projects and submissions that are now being scrutinized and in retrospect look very suspicious. For example, a patch he submitted to the libarchive repository in 2021 replaced a safe function safe_fprintf() with the unsafe version fprintf() in the code, potentially introducing a character escape vulnerability. The issue is currently being investigated.
In February 2022, JiaT75 submitted a patch to XZ-Utils which received comments from never-before-seen accounts complaining that XZ-Utils is not maintained well enough and could use more developers. These could have been sockpuppet accounts created for the purpose of legitimizing Jia’s contributions and pressuring Collin into giving him commit rights.
Groundwork for backdoor was laid in early 2023
Starting in January 2023, Jia Tan started being more involved in the XZ-Utils project and over the course of the year made various contributions, some of which seem to have laid the groundwork for the backdoor and were aimed at gaining more trust. Eventually, he received direct commit permissions and took over some management of parts of the project.
He also made a pull request to oss-fuzz, a project that automatically performs fuzz testing on XZ Utils and many other open-source projects, with the intention of disabling fuzz testing for ifunc, a feature added to XZ and which was leveraged by the backdoor. It’s now believed this was clearly meant to prevent OSS Fuzz from potentially detecting any subsequent malicious code in XZ that leveraged ifunc.
The actual code that makes up this backdoor was added by Jia over the course of several days in February this year, culminating with the release of the backdoored version 5.6.0 on Feb 24th. Then he submitted the new version for inclusion in various Linux distributions.
In an update on his personal website following this incident, Collin wrote: “Only I have had access to the main tukaani.org website, git.tukaani.org repositories, and related files. Jia Tan only had access to things hosted on GitHub, including xz.tukaani.org subdomain (and only that subdomain).”
Based on the community’s findings so far, this appears to be a well-planned attack, possibly a campaign to target many open-source projects, that spanned multiple years and was patiently executed by a sophisticated threat actor.
Similar compromises could be lurking in other projects
The concern is that such compromises could easily happen again or might have already happened in other projects and have yet to be discovered because unfortunately many open-source tools and libraries suffer from a shortage of volunteers and often have a single maintainer. This makes them more susceptible to trusting and accepting work from new people who show an interest in helping those projects.
“Situations like this remind us all that we need to remain vigilant within the open source software ecosystem,” the Open Source Security Foundation (OpenSSF) said in a statement on its website.
“Open source is about well-intentioned humans donating their time and talents to help solve problems, and sadly this can be compromised. As we all learn more details about the anatomy of this attack and the upstream and downstream response, it will give us time to reflect upon how we all can do more to secure open-source software and help maintainers and consumers alike.”