Software supply chain incidents have been making headlines recently. Despite similarities among these security incidents, not all supply chain attacks are created equal.
What is a software supply chain attack?
The umbrella term “software supply chain attack” covers any instance where an attacker interferes with or hijacks the software manufacturing process (software development lifecycle) such that multiple consumers of the finished product or service are impacted detrimentally. This can happen when code libraries or individual components being used in a software build are tainted, when software update binaries are Trojanized, code-signing certificates are stolen, or even when a server hosting software-as-a-service (SaaS) is compromised.
With any software supply chain attack, attackers interject themselves either upstream or midstream to cast their malicious activities and their after-effects downstream to many users. As such, compared to an isolated security breach, successful supply chain attacks are of a much a larger scale with a far-reaching impact.
How big is the software supply chain risk?
Any code repository or library might contain malicious code despite efforts to monitor for and remove suspect packages. While repositories such as GitHub are taking positive measures to keep malware off their sites, attackers are finding ways to successfully make their code available to unsuspecting developers.
The problem is growing. Phylum’s Q3 2023 Evolution of Software Supply Chain Security Report revealed increases in suspicious packages across most categories. Of 3 million packages scanned, more than 10,000 files referenced known malicious URLs. Nearly 86,000 contained pre-compiled binaries, which makes them hard to scan for malware. Another 5,500 attempted to obfuscate their code, and 5,000 were registered by authors with throw-away email accounts.
More significant, the report showed a shift in tactics where more malware authors are targeting specific companies rather than taking a broader approach. Nearly 1,000 malicious packages targeted specific groups or businesses, representing a 47.4% increase over Q2 2023. The goals of these targeted attacks include credential harvesting and source code or intellectual property theft.
Developers don’t appear to be making the software supply chain safer with their practices. Phylum noted that developers download about 24 billion packages from the Javascript package registry npm each week, but almost none of them verified the integrity of the downloaded code. Attackers know this, and that could encourage them to launch broader campaigns such as large-scale ransomware attacks or botnets in the future, the report stated.
Software supply chain attack examples
Here we examine six different techniques used in recent real-world, successful software supply chain attacks.
1. Upstream server compromise: Codecov attack
With most software supply chain attacks, an attacker breaches an upstream server or code repository and injects malicious payload (e.g., malicious line of code or Trojanized updates). That payload is then distributed downstream to many users. From a technical standpoint, however, this isn’t always the case.
The Codecov supply chain attack is one such example. Although the incident has drawn comparisons to the SolarWinds breach, there are stark differences between the two attacks. The SolarWinds supply chain breach was the work of sophisticated threat actors who altered a legitimate update binary, SolarWinds.Orion.Core.BusinessLayer.dll, which was part of the SolarWinds IT performance monitoring product Orion.
As previously analyzed by FireEye, the malicious code contained in the RefreshInternal() method of the counterfeit DLL is shown below. This method invokes the HTTP-based backdoor when Orion loads the Inventory Manager plugin:
Version 2019.4.5200.9083 of the backdoored DLL with the malicious RefreshInternal method
The SolarWinds upstream attack, however, only took full force when the altered binary made its way downstream to over 18,000 SolarWinds Orion customers including US government agencies.
In Codecov’s case, however, no malicious code was distributed downstream, but the after-effects of the attack were. Codecov attackers had obtained credentials from a flawed Docker image creation process that could be used to modify the Codecov Bash Uploader, as per the official security advisory. The attackers then modified the Codecov Bash Uploader hosted on the Codecov server itself to collect environment variables being uploaded from customers’ continuous integration/continuous delivery (CI/CD) environments:
Modified line of Codecov’s Bash Uploader that collected environment variables and sent these to the attacker IP address
Although Codecov Bash Uploader script lived (and continues to live) on the Codecov server itself at codecov[.]io/bash, thousands of repositories were pointing to the link already to send information upstream from their CI/CD environments to this Bash Uploader. As such, the malicious code remained present only at the (compromised) upstream server without any downstream distribution of the code occurring, because the so-called downstream repositories were already pointing to the Codecov server that hosted the Bash Uploader script. Yet, these downstream repositories were affected by this attack as they were configured to upload their data to the Codecov’s Bash Uploader:
In fact, Codecov attackers reportedly breached hundreds of customer networks using credentials collected from the hacked Bash Uploader. Not long afterwards, HashiCorp disclosed that the Codecov incident led to the exposure of its GPG private key used to sign and verify software packages. Twilio has also disclosed some impact from the attack, and other companies are stepping forward with similar disclosures.
2. Midstream compromise to ship malicious updates
The term “midstream” is being loosely applied here to refer to instances of attackers compromising an intermediary software upgrade functionality or CI/CD tool rather than the original upstream source-code base. Last month, Click Studios, makers of the Passwordstate enterprise password manager used by many Fortune 500 companies, notified customers of a supply chain attack. Attackers compromised Passwordstate’s “In-Place Upgrade functionality” to distribute malicious updates to Passwordstate users.
The illicit update contained a modified DLL file titled, Moserware.SecretSplitter.dll, a small part of which is shown below:
In a security advisory, Click Studios stated:
“The compromise existed for approximately 28 hours before it was closed down. Only customers that performed In-Place Upgrades between the times stated above are believed to be affected. Manual Upgrades of Passwordstate are not compromised. Affected customers password records may have been harvested.”
Unsurprisingly, phishing attacks against Click Studios users soon followed with attackers putting illicit links to an updated malware version in these emails.
In addition to having a technical aspect to it (i.e., the upgrade process being tampered with), this supply chain attack had a social engineering aspect as well. In the counterfeit update zip file, which is over 300 MB in size, I discovered the attackers had managed to alter the user manuals, help files, and PowerShell build scripts to point to their malicious content distribution network (CDN) server:
One of the help manual documents stating the malicious CDN server as the official one
A PowerShell install script containing malicious CDN server links
The social engineering aspect to this attack also demonstrates another weakness: that humans (especially newer developers or software consumers) may not always suspect links to content distribution networks (CDNs), whether these are suspicious or not. This is because CDNs are legitimately used by software applications and websites to deliver updates, scripts, and other content.
Online credit card skimming attacks such as Magecart are yet another example of this kind of supply chain attack. In some attacks, Amazon CloudFront CDN buckets have been compromised to distribute the malicious JavaScript code to a wider number of websites that rely on such CDNs.
3. Dependency confusion attacks
In 2021, no piece on supply chain attacks can be complete without mentioning dependency confusion, especially because of the simplistic and automated nature of this attack. Dependency confusion attacks work with minimal effort on the attacker’s end and in an automated fashion due to an inherent design weakness found in multiple open-source ecosystems.
Put simply, dependency confusion (or namespace confusion) works if your software build uses a private, internally created dependency that does not exist on a public open-source repository. An attacker is able to register a dependency by the same name on a public repository, with a higher version number. Then, very likely, the attacker’s (public) dependency with the higher version number will be pulled into your software build, as opposed to your internal dependency.
Illustration of dependency confusion weakness puzzling multiple ecosystems
By exploiting this simple weakness across commonly used ecosystems including PyPI, npm and RubyGems, ethical hacker Alex Birsan was able to hack into 35 big tech firms and walk away with over $130,000 in bug bounty rewards.
Days following Birsan’s research disclosure, thousands of dependency confusion copycat packages began flooding PyPI, npm and other ecosystems. Although most of these copycats were created by other aspiring bug bounty hunters, some went a step too far by targeting known companies in a malicious manner.
There are multiple ways to resolve dependency confusion, including registering (reserving) the names of all your private dependencies on public repositories before an attacker does and using automated solutions–such as a software development lifecycle (SDLC) firewall that prevents conflicting dependency names from entering your supply chain.
Owners of open-source repositories can adopt a more stringent verification process and enforce namespacing/scoping in place. For example, to post packages under the “CSO” namespace or scope, an open-source repository could verify if the developer uploading a package has rights to do so under the name “CSO.”
Java component repository Maven Central employs a simple domain-based verification to verify namespace ownership–a practice that can easily be modeled by other ecosystems. [Full disclosure: Maven Central is maintained by my employer, Sonatype].
Similarly, packages published to the Go package repository are named after the URL to their GitHub repository, making dependency confusion attacks much more challenging, if not outright impossible.
4. Stolen SSL and code-signing certificates
With an increase in HTTPS websites, SSL/TLS certificates are now ubiquitously present and protect your online communications. A compromise of an SSL certificate’s private key could therefore threaten the secure communication and assurance offered by an end-to-end encrypted connection to the end-users.
In January 2021, Mimecast disclosed a certificate used by its customers to establish connection to Microsoft 365 Exchange services had been compromised, potentially impacting communications of about 10% of Mimecast users. While Mimecast did not explicitly confirm whether this was an SSL certificate, this largely appears to be the case, as suspected by some researchers.
While a compromised SSL certificate is problematic, a stolen code-signing certificate (i.e., a compromised private key) can have far wider consequences for software security. Attackers who get their hands on the private code-signing key can potentially sign their malware as an authentic software program or update being shipped by a reputable company.
Although Stuxnet remains a significant example of a sophisticated attack in which attackers used stolen private keys from two prominent companies to sign their malicious code as “trusted,” such attacks have flourished before Stuxnet and continue to even years later. This is also why the aforementioned example of HashiCorp’s GPG private key exposure in the Codecov supply chain attack is problematic. Although there is no indication yet that HashiCorp’s compromised key was abused by attackers to sign malware, such an incident was a real possibility until the compromised key was revoked.
5. Targeting developers’ CI/CD infrastructure
Sonatype recently observed a multi-faceted software supply chain attack that not only relied on the introduction of malicious pull requests to a user’s GitHub project, but also abused GitHub’s CI/CD automation infrastructure, GitHub Actions, to mine cryptocurrency. GitHub Actions provides developers with a way to schedule automated CI/CD tasks for repositories hosted on GitHub.
The attack consisted of attackers cloning legitimate GitHub repositories that used GitHub Actions, slightly altering the GitHub Action script in the repository, and filing a pull request for the project owner to merge this change back into the original repository.
Attacker (edgarfox1982) filing pull request for a legitimate project owner to merge altered code
Should a project owner casually approve the altered pull request, a supply chain attack has succeeded, but that wasn’t even the prerequisite here. The malicious pull request contained modifications to ci.yml, which were automatically run by GitHub Actions as soon as the pull request was filed by the attacker. The modified code essentially abuses GitHub’s servers to mine cryptocurrency.
Such an attack kills two birds with one stone: It tricks a developer into accepting a malicious pull request, and should that fail, it abuses the automated CI/CD infrastructure in place for conducting malicious activities.
Likewise, researchers who successfully breached United Nations (UN) domains and accessed over 100,000 UNEP staff record could do so mainly because they had found exposed Git folders and “git-credentials” files on these domains. A threat actor who obtains access to Git credentials can not only clone private Git repositories, but potentially introduce malicious code upstream to trigger a supply chain attack which would have much harsher consequences.
The primary focus of those looking to prevent supply chain attacks has been on recommending secure coding practices to developers or using DevSecOps automation tools in development environments. However, securing CI/CD pipelines (e.g., Jenkins servers), cloud-native containers, and supplementary developer tooling and infrastructure has now become just as important.
6. Using social engineering to drop malicious code
As any security professional knows, security is only as strong as its weakest link. Because the human element remains the weakest link, exploitation may come from places where it is least expected. The Linux Foundation recently banned University of Minnesota researchers who were suggesting intentionally buggy “patches” that in turn introduced vulnerabilities in the Linux kernel source code.
Although the instance was caught and has now been dealt with, it demonstrates a few simple facts: Developers are spread thin and may not always have the bandwidth to vet every single code commit or a proposed code change that may be buggy or outright malicious. More importantly, social engineering may come from least suspected sources–in this case seemingly credible university researchers with an “.edu” email address.
Another recent example includes how any collaborator contributing to a GitHub project can alter a release even after it is published. Here again, the expectation of a project owner is that most contributors are submitting code and commits to their project in good faith. It takes just one collaborator to go rogue and compromise the security of the supply chain for many.
Over the last year, attackers creating typosquatting and brandjacking software packages have repeatedly targeted open-source developers to introduce malicious code in their upstream builds which then propagates to many consumers.
All these real-world examples demonstrate different weaknesses, attack vectors, and techniques that threat actors employ in successful supply chain attacks. As these attacks continue to evolve and pose challenges, more innovative solutions and strategies are needed when approaching software security.