Security researchers demonstrated a software supply-chain attack that could have allowed them to backdoor the codebase of Bazel, a Google-developed open-source tool for automating software building and testing. The attack exploited vulnerabilities in a custom GitHub Action used by the project in its CI/CD workflows, highlighting how security issues can be inherited from third-party CI/CD dependencies.
“We found that a GitHub Actions workflow could have been injected by a malicious code due to a command injection vulnerability in one of Bazel’s dependent actions,” researchers from application security firm Cycode said in a blog post. “This vulnerability directly impacts the software supply chain, potentially allowing malicious actors to insert harmful code into the Bazel codebase, create a backdoor, and affect the production environment of anyone using Bazel. This vulnerability could have affected millions of projects and users who use Bazel, including Kubernetes, Angular, Uber, LinkedIn, Databricks, DropBox, Nvidia, Google, and many more.”
Custom GitHub Actions can introduce hidden security risks
GitHub Actions is a CI/CD service offered by GitHub that allows developers to automate the building and testing of software code by defining workflows which execute automatically inside containers on either GitHub’s or the user’s own infrastructure. This is a popular service that many GitHub-hosted projects rely on to run various automated tests or actions on code contributed to their repositories.
However, the functionality offered by GitHub Actions can be used insecurely and researchers have highlighted multiple such mistakes in the past that could have resulted in software supply-chain compromises. In December 2022, researchers from security firm Legit Security showed how attackers could poison binary artifacts that would then be used as input for a project’s GitHub Action workflows. Earlier this month another team of researchers from Praetorian showed how self-hosted GitHub Actions runners can be exploited to infiltrate an organization’s development infrastructure. Likewise, the new research from Cycode doesn’t exploit any inherent vulnerability in GitHub Actions itself, but rather in the way some projects choose to use some of its features without considering the risks.
Users define GitHub Actions workflows by creating YAML files within the .github/workflows directory of a repository. These workflow files contain a series of jobs and steps that should be executed to achieve a task and they often involve calling predefined “actions.” These actions are like small scripts or code functions and some of them are provided by GitHub itself while others are created and provided by third parties. The latter are known as Custom Actions and they allow a level of code reuse and nested dependencies that is similar to that seen with various package managers like npm for JavaScript or pip for Python.
Just as vulnerabilities can be inherited from package dependencies in npm or pip, transitive vulnerabilities can make their way into a workflow from custom GitHub Actions written by other people. In fact, it’s even worse, because custom GitHub Actions can execute not just additional actions but also JavaScript and Python packages as well as shell commands. These are known as composite actions.
“When creating new workflows, we heavily rely on various third-party dependencies, including custom actions that, in turn, use other actions,” the Cycode researchers warned. “These actions execute programs written in different languages, such as JavaScript or Python, and leverage libraries from various package managers like NPM or PyPI, forming an extensive chain of dependencies.”
Cycode scanned 3.4 million workflows pulled from GitHub repositories and over 98% of them incorporated one or more custom actions, making this a widespread problem.
Exploiting a Bazel composite action
The Cycode researchers looked at Bazel’s workflows and found one called cherry-picker.yml that executed with a token that had read/write permissions every time an issue in the repository’s bug tracker was closed or milestoned. One of the jobs defined in the workflow performs a check on milestoned issues — resolved as part of a milestone defined in the project — to see if it begins with the text “### Commit IDs” and then executes a composite action from a Bazel repository called continuous-integration.
By inspecting the composite action, the researchers realized that it was taking the issue title and body and passed it a bash shell script. One problem with this approach is that anyone can open an issue on a project to report a bug and they control both the title and the body of that issue. This means they will essentially control the input passed to the bash script as long as they can meet the conditions required to trigger this workflow — their issue was important enough to be milestoned.
Cycode
“By leveraging a shell feature called command substitution, we can include a shell command using the $( ) characters,” the researchers said. “Anything within these brackets will be treated as a system command and will be executed.”
The researchers had to find a way where they could include malicious shell commands between the $( ) characters in the body of the issue they opened without it looking suspicious. So, they wrote a command to launch the curl command-line tool to download a remote script and execute it, they encoded it in base64 to make it unreadable, they put it in between $( ) characters and they inserted it in a part of the issue where they pasted the system log.
Developers will often ask users to include information about the environment where they encountered the bug or error they’re reporting along with a print of the system log which could help debug the issue. However, the Linux system log has many entries and not all output is relevant, so it’s easy for a reader to not notice that in some non-relevant part there’s a base64 encoded string that looks suspicious.
Cycode
Tool available to scan repositories for exploitable Bazel instances
To find the vulnerability, the Cycode researchers used an open-source scanner they developed dubbed RAVEN (Risk Analysis and Vulnerability Enumeration for CI/CD) along with a specific query they devised for finding vulnerabilities in composite actions. In addition to the issue in Bazel, they also found one in the Apache Camel project. Both flaws were reported and patched.
“Custom actions add a significant burden on the organization’s software supply chain. A few lines of code in the top-level workflow can translate into thousands or even millions of lines of code, many of which we may not even be aware of,” the researchers said. “Also, they contain many components that undergo frequent changes and updates, potentially addressing new vulnerabilities on a daily basis.”