Kubernetes Ransomware Challenge: How to Mitigate and Recover

Kubernetes continues to grow in popularity for the automation of at-scale software delivery, deployment and management in containerized environments. But many organizations see the threat of ransomware attacks as a showstopper to adopting it. A Red Hat survey of more than 500 DevOps, engineering and security professionals found that 55% of them had delayed deploying Kubernetes applications into production because of security concerns.

While security worries can postpone a Kubernetes rollout, the rising threat of ransomware doesn’t seem to have dampened Kubernetes adoption. In Red Hat’s survey, 88% of respondents said their organization uses Kubernetes for container orchestration and 74% have adopted Kubernetes in production environments.

As Kubernetes deployments continue to increase, the number of attack vectors, in theory, is expanding as well, helping to explain in part the surge in ransomware attacks and resulting damages for these environments. Kubernetes clusters, as well as containers in general, are vulnerable as entry points for intruders seeking to orchestrate ransomware attacks, due largely to their highly distributed nature. As they scale, the number of microservices deployed results in many interdependencies that can be exploited.

Ransomware attacks have understandably garnered attention by the U.S. government, culminating in an official memo from the White House, issued back in June 2021, urging corporate executives and business leaders to take specific steps to protect their assets. In July, the National Security Agency (NSA) issued a report specific to Kubernetes security. Each offers a trove of information, with plenty of overlapping guidance, on how to protect Kubernetes clusters from ransomware attacks.

The White House report makes it clear that ransomware attacks are usually preventable. Once Kubernetes ransomware risks are understood, there are specific and reasonable steps an organization can take for protection.

The Weakest Links

Vulnerabilities in Kubernetes environments that ransomware attackers exploit are much like those hit by other kinds of attacks. They often result in data theft and destruction, the theft of expensive computing resources via a cloud provider account, illicit mining for cryptocurrency, denial-of-service attacks (DoS) and other security-related incidents. Ransomware, specifically, involves an attacker blocking access to data and applications — typically through encryption — and extorting an organization to pay a ransom to resume access.

The underlying framework of a Kubernetes cluster lends itself to numerous attack entry points among Kubernetes components:

• The Kubernetes API server.
• The etcd server client for key-value storage.
• Kubelet for node management.
• A kube-scheduler used to assign pods to nodes.
• A kube-controller-manager.
• And for those relying on cloud providers, a separate kube-controller manager for cloud environments.

While less complex to implement and manage, virtual machines (VMs) do not offer the immense advantages that Kubernetes does for developing and managing applications, but they are easier to lock down, as VMs running Linux, Windows or another operating system are largely self-isolated. They don’t share the underpinning operating system, whereas in a Kubernetes deployment, all the containers running in each node do.

If a Kubernetes node is compromised, all pods on the node are affected, which can put the entire cluster to which the node belongs at risk. Consequently, all containers within that cluster can be exploited because unlike VMs, they share the same kernel of the same host.

In the world of Kubernetes, every node, cluster and container can share a number of resources — and vulnerabilities — in addition to a common operating system. All it takes is for a single microservice to introduce vulnerabilities among multiple containers. The potential attack vectors lurking within a container supply chain are as numerous as the microservices connecting the containerized environment.

Inherent within the Kubernetes cluster is another source of trouble: secrets management. Used to provide tokens for APIs, passwords and other sensitive data, secrets have their own vulnerabilities. Ingress controllers and other components, for example, are configured to access secrets within the entire cluster. Moreover, secrets are largely unencrypted. While some encryption schemes do exist — including an option Kubernetes offers for encrypting secrets — those alternatives are still being beta tested or are not 100% secure. DevOps teams are rightly wary of using them in production environments.

The Lurking Predators

A ransomware attacker seeking to exploit a vulnerability in a Kubernetes environment will likely use automated tools to scan for vulnerabilities. Many scanning tools can be bought online from the dark net, even sometimes via public forums such as Reddit, for just a few hundred dollars. The attacker finds a way to penetrate a cluster and then waits while their automated scanning tool uncovers an angle of attack.

More direct routes to data exist for ransomware attackers. For more than eight years until the vulnerability was recently widely disclosed, a simple scanning tool could seek and find standard ports for MongoDB databases that were exposed by default for several years. During that time, any MongoDB customer admin — or intruder posing as one — had read-write access to these MongoDB databases through the unsecured ports, which were also unencrypted. This meant that an organization that deployed a MongoDB database in a Kubernetes or containerized environment on, for example, Amazon Web Services (AWS) could expose the database to the world through commands that required no credentials for access.

The Insider Challenge

Even security and IT staff members can fall for a phishing scam. All it takes is one inadvertent click on a malicious link by someone with advanced network access privileges to provide an attacker with a direct link to potential ransomware gold. Potentially lethal targets include admins, especially those with control plane API access.

With these privileges required to manage them, Kubernetes clusters are vulnerable to ransomware attacks through tricking an administrator into providing access to, for example, a terminal to the host. Such access can compromise the underpinning operating system and thus a particular container or pod. In some cases, an attacker can gain access to a pod and escalate their privileges further for a pivotal attack if root access has been secured. They may gain access to and control critical data that extends to the on-premises data stores or on the cloud with the etcd server client. By escalating access privileges, the attacker may reach into local and cloud-based data stores.

Siloscape is a more recent example of how ransomware attackers can gain access to Microsoft Windows containers. Using a Tor proxy and an .onion domain, Siloscape takes advantage of vulnerabilities through remote code execution (RCE). From inside the container, the attacker runs a binary to gain access to the entire cluster. In this way, a ransomware attacker can gain access to any data stored within the cluster, including passwords and customer information. In many cases, a cluster consisting of containers distributed across a multicloud environment can engender a more massive supply chain attack.

Immutability: The Golden Standard of Ransomware Readiness

There are key best practices and tools to prevent Kubernetes ransomware attacks. Proper patching and updates and entrusting security management through a reliable third party — simpler API-security management solutions have become available — can provide protection against the attacks described in this article. For infrastructure protection in a cluster environment, a service mesh can provide a sidecar model — in addition to helping to orchestrate the traffic between services within a cluster — to prevent ransomware or other attacks with an adjacent proxy daemon that verifies both inbound and outbound traffic for the containers.

Damage control — limiting the lateral spread of the attack — is arguably the best shield against ransomware attacks. Using microsegmentation, limited access can be granted for certain data and applications while the rest of the environment can be assumed to be insecure, despite consistent patching and other security best practices. This practice, known as zero trust, limits admins or users’ access privilege to a range of APIs while restricting access to more sensitive and mission-critical data stores that might include direct object storage components.

Backup components — typically the lifeblood of the organization with the data and applications it needs to function — can benefit from immutability: once created, they can’t be modified. Data in the direct-object storage backups remains read-only and encrypted so that even if an attacker gains access, data remains out of reach to the attacker and could therefore be used to restore from safely, thereby thwarting the ransomware.

If a ransomware attack does occur, a viable backup-and-restore system should be in place to restore the data and applications in a matter of minutes — again, the data and applications remain immutable. The decryption key that allows for this restoration process to take place remains under lock and key outside the network, where a ransomware attacker cannot reach it, either in the cloud or on-premises. In this way, any data deployed in Kubernetes remains the least-privileged possible. In the MongoDB attack described earlier, the attacker could not have simply taken control of a direct-object storage component through the API.

Immutability offers protection beyond ransomware attacks. A backup admin or attacker with access to AWS-housed containers would not be able to delete any data. Data could be backed up at set intervals depending on the organization’s requirements. If a data restore and replication was required, true immutability would enable a rollback to different versions of backups for different timestamps. While an aspiring ransomware attacker might be able to gain control of part of the system, everything would be restored to a previous timestamp in minutes, almost as if the organization had gone back in time to before the attack.

Action Now

Preventing Kubernetes ransomware attacks requires awareness of both the nature of the attacks and of the security constraints involving Kubernetes. However, awareness is just the first step; mitigation tactics and best practices must be implemented.

• Immutability is the gold standard for preparing for ransomware attacks in a Kubernetes environment. Without it, your organization will likely cease operations and/or have to pay a ransom if your data is locked or deleted during an attack — and there is never an absolute guarantee you will ever get your data back.

• Direct-object storage immutability enables your organization to simply and rapidly restore data, applications and clusters, as if the attack never occurred. If an attacker does gain access and has the ability to delete cluster infrastructure, the organization should also be able to invoke disaster recovery involving the restoration of data and workloads to a brand-new cluster. The organization should also have the right tools in place so that the new cluster can be located in a different public cloud or location, if needed.

• Disaster recovery must extend across all types of Kubernetes deployments and use cases, including edge environments. Data snapshots should be exported to an object store at regular intervals so that the objects with retention applied cannot be accessed. Following an attack, the organization must be able to restore encrypted data using credentials. As a final protective measure, backups and their immutability should be tested regularly.

Protecting against ransomware requires significant changes and ongoing work, but as ransomware attacks increase, it will be worth it. Someday, if operations suddenly shut down and a message comes in that it will cost $100 million to get your customer data back, in just a few minutes, you’ll be back to business as usual without paying the attacker a dime.

Feature image via Pixabay