Editor's note: This post was updated on 12/17/21 at 10:09am PST.
In this post, we provide recommendations from the Google Cybersecurity Action Team and discuss Google Cloud and Chronicle solutions to help security teams to manage the risk of the Apache “Log4j 2” vulnerability (CVE-2021-44228 and CVE-2021-45046 ).
For the latest updates on our assessment of the potential impact of the vulnerability on Google Cloud products and services, please visit Google Cloud’s security advisory page.
Background
The Apache Log4j 2 utility is an open source Apache framework that is a commonly used component for logging requests. On December 9, 2021, a vulnerability was reported that could allow a system running Apache Log4j version 2.15 or below to be compromised and allow an attacker to execute arbitrary code on the vulnerable server. On December 10th, 2021, NIST published a critical CVE in the National Vulnerability Database identifying this as
. The official CVSS base severity score has been determined as a severity of 10. We strongly encourage customers who manage environments containing Log4j 2 to update to the latest version or take the mitigation actions outlined in this post. The Cybersecurity and Infrastructure Security Agency (CISA) released additional recommendations for immediate steps regarding this vulnerability.
Technical Details
The “Log4j 2 vulnerability” can be exploited by sending specially crafted log messages into Log4j 2. The attacker does this by leveraging the Java Naming and Directory Interface (JNDI) – which is a Java abstraction layer included by default in the Java SE Platform that allows a Java application to retrieve data from directory services (such as LDAP). JNDI uses HTTP URIs to resolve to a specific directory service – and the URI can be adjusted as needed to resolve to the right directory location. Using Log4j 2, an attacker sending a log message with a specially crafted URI can cause the application to execute arbitrary code (such as by providing a Base64 encoded script in the path).
This is due to specific behavior in Log4j 2 that allows for the input of variable data into the log (called Lookups). In a Lookup, the reference is queried and evaluated for input into the log. By using this feature during an exploit, the attacker uses the URI input to instruct Log4j 2 to resolve an object they input (such as an encoded script). This issue has been resolved in the latest version of Log4j 2.
Updates and Recommendations to Identify, Detect & Protect
Customers should look to upgrade to the latest version of Log4j2 as soon as possible. Taking this action (and following additional advice from the Apache foundation) is your best immediate action.
In addition to updating Log4j 2, some Google Cloud Security products can help detect and temporarily mitigate exploitation until the patch can be applied. Additionally, if you have third party WAF, IDS/IPS, and/or NDR solutions deployed in Google Cloud, please consult with your vendors for more guidance related to this vulnerability.
To identify potentially impacted systems we recommend the use of a vulnerability scanner. These tools have reported identifying the vulnerability referenced in National Vulnerability Database (NVD) and will help to locate impacted systems.
Where possible, we recommend implementing all of the below for a layered defense in depth strategy.
Cloud Armor WAF Log4j 2 Detection and Blocking Rules
Relative to Log4j 2, Cloud Armor helps mitigate threats for applications or services behind external HTTP(S) load balancers. You can enable Cloud Armor through Cloud Console > Network Security, or via API. Cloud Armor’s WAF rules can be configured to detect, or detect and block requests. Cloud Armor customers can now deploy a new preconfigured WAF rule that will help detect and, optionally, block commonly attempted exploits of CVE-2021-44228 and CVE-2021-45046 while you are patching your systems. To learn more about addressing the Apache Log4j 2 vulnerability with Cloud Armor, please read this blog article.
Chronicle
Threat Hunting and investigation tools can be used to look at historical data and determine if exploitation was attempted – or can be used as vehicles for monitoring active exploitation. Chronicle is Google’s threat hunting tool that provides extended event collection across cloud and on-prem (such as EDR logs, firewall logs, etc). If you are using Chronicle for log ingestion/SIEM and have historical event data stored (Chronicle retains 12 months of data by default) you can search for historical exploit attempts. Customers should search for events that contain “jndi” and a combination of strings that follow including “ldap”, “rmi”, “ldaps”, or “dns”, generated from HTTP requests, which correspond to possible Log4j 2 exploitation patterns.
For example – this syntax could be used in a Yara-L rule:
Cloud IDS Network-based Threat Detection
Cloud IDS has been updated to help detect common types of Log4j 2 exploit attempts. These new detections are on by default for any existing or newly added deployments of Cloud IDS. Positive detections will appear in Cloud IDS alert logs, visible in terse mode in the Cloud IDS UI of Cloud Console, and in verbose mode in Cloud Logging, or by API. Cloud IDS is enabled and configured through Cloud Console > Network Security, or via API. Please see this blog for more information on IDS relative to both CVEs.
Security Command Center Premium
For customers using Security Command Center (SCC) Premium, we have released a detection rule that looks for inbound exploitation attempts. Additionally, Event Threat Detection has been updated with new indicators of compromise for stage 2 exploit hosting detection as part of Malware: Bad IP and Malware: Bad Domain. These rules are enabled by default.
For customers running on GKE, the Container Threat Detection module in SCC detects execution of added binaries, libraries, and malicious bash scripting, giving defenders insight into stage 2 execution. Finally, as we’ve seen reports of attackers spinning up cryptocurrency mining, SCC provides a set of capabilities for detecting the network activity associated with mining activity.
To ensure coverage for detection, we recommend enabling Event Threat Detection and Container Threat Detection across your entire organization.
Event Threat Detection enabled by default.
In Cloud environments it’s important to remember that access to a host can mean access to identities contained on the host or owned by the host. Security Command Center Premium’s Event Threat Detection has a set of User and Entity Behavior Analytics (UEBA) capabilities to identify compromised identities in realtime.
Cloud Audit Log Admin Activity Logs are on by default and inform UEBA capabilities in SCC for threat detection. For network detection of C2 activity, Event Threat Detection does require that additional logging is enabled: ideally Cloud DNS logs, or a sampled set of VPC Flow Logs. This doc page provides additional information. To detect incoming exploitation attempts with ETD, you must have HTTP(S) Load Balancer logging enabled.
Detecting vulnerable Log4j 2 with On-Demand Scanning
The Java scanning feature of Google Cloud On-Demand Scanning can be used to help identify Linux-based container images that use an impacted version of Log4j. Identifying vulnerable images early in the CI/CD process can help prevent them from being deployed in production. To use this functionality on either a locally stored container image or one stored in Artifact Registry or Google Container Registry, follow the instructions in the scanning images for Java vulnerabilities guide. Once results are returned, you can search for the text “
” using grep, a text editor, or your own script. Scoring reported in results is based on CVSS Version 2.0. If none of the results contain the CVE, On-Demand Scanning has not identified any affected packages.
Output indicating a vulnerable image
Results returned by On-Demand Scanning are formatted to the open-source Grafeas standard, and can be parsed in the same way as vulnerability scanning in Artifact Registry and Google Container Registry. Thus, any existing tooling that consumes the Grafeas format can be used with On-Demand Scanning.
Cloud Logging
You can use the Logs Explorer to help detect potential attacks on your service exploiting the Log4j 2 vulnerability. If you are using Cloud Logging to log requests to your service, you can check httpRequest fields with user generated content for potential exploits that have string tokens like ${jndi:ldap://.
There are multiple variations on how this vulnerability is being exploited, and there are many ways to use unicode or escapes to avoid detection. This is an introduction to using a regex query to help detect some of the common exploits:
The above query matches many of the obfuscated variations of the string “${jndi:” in HTTP Load Balancer request logs. You can use similar regular expressions to scan request logs in other services by changing the resource.type. The query may take a long time to complete if you are scanning a large volume of logs. To make your queries run faster, you can make use of indexed fields like resource.type, resource.labels, or logName to narrow your query to a set of specific services or log streams.
Detecting matching log entries does not indicate that there has been a successful compromise. It may indicate that someone is probing to exploit the vulnerability within your project or workload. There could also be false positives if your application uses patterns like “${jndi:” in the http request fields.
Cloud Logging query results only include logs that have already been ingested into Cloud Logging and are also within the user specified retention limits. While most Google Cloud services have logs enabled by default, logs that were disabled or excluded will not be included in this search. If you are using an HTTP(S) Load Balancer, logging needs to be enabled for the request logs to be available in Cloud Logging. Similarly, if you have web servers like Apache or NGINX running on a VM, but have not installed the logging agent, those logs will not be accessible within Cloud Logging.
More details on how to use Cloud Logging for this kind of use case can be found here.
Apigee
Apigee provides tools that can help detect and block log4j 2 exploit attempts through your APIs. Detecting potential attacks can be done using custom reports in API Analytics. Edge API Analytics collects metadata on traffic that flows through your API proxies. The report can show patterns in your analytics records that indicate exploit attempts. Matching records don't mean there has been a successful compromise, but may mean that someone is attempting to exploit the vulnerability through your APIs. A detailed explanation on how to configure the report can be found here. Blocking log4j 2 exploit attempts through your Apigee-managed APIs can be done using Raise Fault policy and an appropriate regular expression that matches known attack patterns. The regular expression and the payload location are configurable and you will be able to tune them as more information is revealed over time. A detailed explanation on how to configure this new policy can be found here.
We will continue to actively monitor this event and will provide updates to this blog post on relevant mitigation steps and detection mechanisms. Please visit our security advisory page for updates on our Google Cloud security assessment.