Security researchers warn that certain commands executed in the AWS and Google Cloud command-line interfaces (CLIs) will return credentials and other secrets stored in environment variables as part of the standard output. If such commands are executed as part of build workflows in CI/CD tools the secrets will be included in the returned build logs.
AWS and Google Cloud consider this expected behavior and it is up to users to take steps to ensure sensitive command outputs are not saved in logs or that sensitive credentials are stored securely and not in environment variables. The Microsoft Azure CLI had a similar behavior but Microsoft flagged it as an information disclosure vulnerability and fixed it back in November.
Researchers find dozens of projects that leaked information
To prove that this credential leaks can actually occur in practice due to this CLI behavior, researchers from Orca Security searched GitHub repositories for potentially leaky AWS and Gcloud CLI commands used in Github Actions, CircleCI, TravisCI and Cloud Build scripts.
“We entered a few repositories and looked at their build logs,” the researchers said in a report. “We encountered many dozens of projects that inadvertently leak information that could be considered sensitive in environment variables, including passwords and keys.”
In addition to credentials, the researchers identified other potentially sensitive information such as project names, service accounts, S3 bucket names, account IDs and IAM roles.
Saving secrets in environment variables is a bad idea
All major cloud providers offer command-line interfaces that have a set of commands for interacting with the compute and other services provided by the cloud vendors. While many developers use these CLIs on their local machines, they are also commonly used by workflows in CI/CD tools, for example to automatically execute build instructions when new code is submitted and to perform various tests. The execution of these workflows will generate logs that can be publicly accessible in many cases if the repositories are public.
According to the Orca researchers, it is a common practice to store credentials needed by these commands to execute successfully in environment variables in the Linux command-line environments used by these CLIs. The problem is that some of the AWS and Gcloud CLI commands also return these environment variables to stdout (standard output on Unix systems) as part of the command’s execution.
For AWS CLI the Lambda get-function-configuration, get-function, update-function-configuration, update-function-code and publish-version exhibit this behavior. Lambda is AWS’s serverless computing platform that allows developers to execute code and applications directly without provisioning virtual servers. For Gcloud CLI the gcloud functions deploy <func> –set-env-vars, –update-env-vars and –remove-env-vars returns values stored in environment variables.
“If the developer isn’t aware of it, even using secret masking via GitHub Actions / Cloudbuild will not do, because there may be pre-existing environment variables in the cloud function,” the researchers said.
Mitigation to avoid the leak of secrets
AWS will update its documentation to make the risks clearer to users. The company advises customers not to store sensitive values in environment variables and instead use the purpose-built secure secrets store such as AWS Secrets Manager. Users are also advised to review their build logs to ensure there are no secrets in them and to suppress sensitive command outputs by directing it to /dev/null. Access to build logs should also be restricted to only users who need to have it.
Google Cloud had similar recommendations, according to the Orca researchers. The company noted that command output can be suppressed by using the “–no-user-output-enabled” flag and that secrets can be stored securely by using the “gcloud deploy command” with the “–set-secrets” and “–update-secrets” options.