The runaway success of Kubernetes adoption by enterprise software developers has created motivation for attackers to target these installations with specifically designed exploits that leverage its popularity. Attackers have become better at hiding their malware, avoiding the almost trivial security controls, and using common techniques such as privilege escalation and lateral network movement to spread their exploits across enterprise networks. While methods for enforcing Kubernetes security best practices exist, they aren’t universally well known and require specialized knowledge, tools, and tactics that are very different from securing ordinary cloud and virtual machine use cases.
The evidence of increasing Kubernetes-focused attacks comes from direct telemetry, reinforced by anecdotal reports. Security vendors such as Palo Alto Networks, Wiz, and Aqua Security all have put in place Kubernetes honeypots that see rapid attempts at compromising new clusters, with a newly created cluster being attacked within minutes or a few hours. These attacks scan well-known TCP/IP ports 10250-9 used by containers to communicate with each other, for example. These pre-attack scans happen dozens of times daily, an indication of automated and programmatic efforts to compromise the underlying code.
Let’s examine the threat landscape, what exploits security vendors are detecting, and ways that enterprises can better harden their Kubernetes installations and defend themselves.
Kubernetes threat landscape
A report from the Cloud Native Computing Foundation illustrates the complexity of the landscape, with an interlocking collection of data flows, dependencies, and processes. These components require their own protective methods to encrypt communications, authenticate containers, storage repositories and users appropriately, and protect containers from being exploited. The diagram below (from the Foundation, interpreted by Trend Micro) shows that there is a steep learning curve to understand all the relationships and how the various parts fit together.
Kubernetes data flow diagram by CNCF Financial Services User Group
The complexity of this architecture was done deliberately, because Kubernetes “was designed to allow users to have a lot of freedom and used an open architecture and a default security model of being open by default,” Assaf Morag, a data analyst with Aqua Security, tells CSO. But despair not: “The fact that Kubernetes is such a sprawling platform with so many integrations presents an opportunity. It makes it easy to build an automated, systematic set of processes that bakes security into the core of the Kubernetes build and deployment process,” as Palo Alto Networks’ recent “Complete Guide to Kubernetes Security” mentions in its introduction. However, the inherent openness of Kubernetes means that there is no single security toolset that does everything.
Going back to security basics
Often, basic network security postures such as not exposing secret encryption keys, using complex and non-default administrative passwords, deploying various segmentation schemes and following least privilege access rights are seemingly forgotten when it comes to how containers are secured. “We have reverted back to the threats we saw across our 1990s networks,” Palo Alto Networks manager of cloud threat intelligence Nathaniel Quist tells CSO.
This latter effort of controlling access rights, particularly with enforcing role-based access, can be more difficult to implement than for other cloud applications. “Kubernetes has a very complex model to implement this correctly,” says Quist. Analysts from Aqua Security discovered one of the first backdoor attacks on role-based controls back in April 2023. This targeted at least 60 different clusters and had successfully installed cryptomining malware code. Access rights were manipulated so that the malware would persist and have administrative privileges.
“These crypto-based attacks are on the rise, mainly because Kubernetes clusters are highly efficient execution platforms and are ideal for spinning up these workloads,” Shay Berkovich, a threat researcher at Wiz, tells CSO. The company’s researchers have seen examples of cryptomining attacks such as PyLoose and newhello. These aren’t new — Tesla’s cloud estate was compromised by a poorly designed Kubernetes dashboard back in 2018 and ended up running cryptominers.
“Providing users of Kubernetes access to the Kubernetes API in a least privileged manner is a difficult but crucial task,” Venkat Thiruvengadam, CEO of DuploCloud tells CSO. “Adopting a standardized, and automated, method for creating this access is a key step in having a secure Kubernetes environment.”
“Once an attacker is past the initial access, the opportunities are ample for lateral movement and privilege escalation within a cluster,” wrote Berkovich in a blog for Wiz last November. He recommends better zone defense, meaning that every defensive security staffer is given a specific area to cover the most vulnerable entry points.
Another privilege escalation bug in Google’s Kubernetes cluster was discovered by Palo Alto’s Unit 42 researchers last December, and fortunately quickly fixed. But this illustrates how because of “its complexity, many Kubernetes environments are susceptible to security breaches due to misconfiguration and excessive privileges.”
Another problem stems from Kubernetes’ inherent distributed architecture. “This means that often there are too many cooks creating code,” Aqua Security SVP of strategy Rani Osnat tells CSO. “There can be separate teams responsible for running a cluster, managing the development pipeline and another to enforce access controls, and this could result in situations where governance is lacking because these teams don’t always talk to each other.”
One major distinction between Kubernetes and other cloud environments is that the former blends both personal choices with leveraging shared code repositories. While sharing can be effective at quickly building an application, it does introduce risk if the shared code is compromised. “Separate teams need to run their code in separate clusters,” Ami Lutwak, CTO at Wiz, says. This is still far from common practice, according to numerous security researchers.
Perhaps the biggest issue for Kubernetes insecurity is how quickly it can automate a supply chain attack, particularly when it comes to exposing encryption and other secret keys. This was summarized by Aqua’s Morag in a blog post last November, what he called a ticking time bomb. They found hundreds of records that exposed these secrets, searching Github’s API for credentials. One of the major contributors is that developers thought encoding these keys in base64 was sufficient protection, which fooled even some very experienced developers. “It is merely a single base64 decode command away from being revealed as plaintext secrets,” he wrote. “There is a clear need for open-source tools and secrets scanners to improve their detection capabilities to include encoded secrets.”
Another search by Morag found over a period of three months that 60% of the clusters were actively being attacked by several different cryptominer campaigns. He says in that blog post that a simple misconfiguration — which he has observed from their telemetry — of security policy rules can lead to an exposed cluster. “Developers don’t understand the gravity of their code or haven’t learned how to implement better security,” he tells CSO.
Best practices to avoid Kubernetes exploitation
A good first place to start is reviewing the OWASP Kubernetes “Cheat Sheet” which has lots of specific recommendations on best practices. As previously reported, these include locking down containers and adopting more comprehensive role-based access controls. Palo Alto’s Quist emphasizes this and recommends that developers “need to have different accounts for different operational groups and make proper usage of key and secret management services, such as Key Vault in AWS.”
Another practice of better secret management is also important. Wiz’s Berkovich recommends aggressive usage of in-cluster separation security controls, first and foremost. He also suggests using smart namespace-based isolation with RBAC that separates network policies and user namespaces to prevent attackers from any lateral movement between containers.
Regular auditing of clusters can help developers identify and fix misconfigurations promptly. “It’s critical to meticulously configure the clusters, ensuring that network policies are correctly implemented, and access controls are robust,” says Berkovich.
Finally, provide better employee and developer training. “Organizations must invest in training their staff about the potential risks, best practices, and correct configurations. It is important to remember that Kubernetes security is not a one-time task but a continuous process of learning, adapting, and improving as the technology and threat landscape evolve,” wrote Moran in Aqua’s blog.