Log4Shell: A retrospective

Now that the dust has settled on both the holiday season and the Log4j vulnerability that saw many of us working through it (CVE-2021-44228), it makes sense to look back and take stock of how things played out. What strategies worked in the face of one of the most notable vulnerabilities of the last decade?

To begin with, let’s briefly look at the issue itself. Log4j is a Java logging utility used by just about every Java-based product, tool, and service on the internet. If you’ve ever seen an error page on a website or mistyped your credentials, chances are you’ve generated an event processed by Log4j at some point. Injecting a specifically crafted string into an event processed by Log4j would cause Log4j to query an arbitrary external URL and load it as a Java object. An attacker could then use this vulnerability (dubbed Log4Shell) to force a victim to download, install and execute externally hosted malicious payloads with relative ease.

There were a few common practices in organizations that felt they had prepared for or responded to Log4Shell effectively.

Good asset management

Organizations with comprehensive and up-to-date asset management systems found it easier to identify vulnerable systems quickly, assess the potential “blast radius” of successful exploitation and quantify their overall risk. They also found it easier to apply emergency patching, quarantine or monitoring measures on vulnerable hosts.

Understanding the software supply chain

With a good understanding of their inclusions, libraries, dependencies and overall software supply-chain, organizations could quickly identify vulnerable third-party code. This enhanced visibility into their codebase allowed them to report the vulnerability to upstream code maintainers and peers alike.

Centralized logging

In the case of Log4Shell, centralized logging is admittedly a bit of a double-edged sword. On the one hand, having event logs shipped off-host provides a measure of assurance that a crafty attacker couldn’t remove the tell-tale signs of a successful exploit before a SOC analyst saw them. On the other hand, centralized logging inevitably provides a broader attack surface for logging-based attacks such as Log4Shell.

On balance, though, having a central location to store and view the logs of all potentially impacted devices significantly improved both the speed and ease of incident response, making the increased attack surface worth the risk.

Solid threat intelligence

Attempts to exploit Log4Shell were (at least initially) easily detectable by both automata and analysts alike. The ability to quickly acquire and distribute indicators of compromise (IOCs) was vital in the days after the vulnerability’s announcement. As attackers became more sophisticated, deploying varying methods of code obfuscation, the rapid sharing of new IOCs became even more critical. Organizations that maintained up-to-date threat intelligence came out on top regarding reaction speed and effectiveness.

Searching and alerting

Centralized logging and good threat intelligence would be nothing without the correct tooling and training to leverage them. Organizations that deployed EDR/NDR/SIEM solutions along with well-trained staff at the console were well-positioned to both detect and automate the detection of attempts to exploit Log4Shell.

Egress network filtering

Despite the far-reaching effects of Log4Shell, mitigation of the vulnerability was surprisingly easy and affordable for those adequately prepared. The most straightforward technique to ensure that an attacker would fail to exploit the vulnerability was to block HTTPS or LDAP traffic leaving the network.

Blocking the outbound request does not solve the root cause of the problem, but it does prevent attackers from installing malicious payloads on vulnerable hosts. Correctly configured, outbound network filtering has the bonus of logging blocked exploitation attempts, helping steer SOC analysts and incident responders to impacted systems.

Restrictive DNS policies

Contacting malicious external hosts is a prerequisite to a successful Log4Shell attack. By restricting network name resolution to exclusively internal resources, organizations could block some (but not all) attempts to download malicious payloads.

While not perfect – this defensive technique may be subverted by the attacker using IP addresses or hosting the malicious Java object on already-compromised internal resources – it is a cheap and surprisingly effective method of making things more difficult for attackers.

Conclusion

Log4Shell was an excellent (if terrifying) case study in the value of “defense-in-depth”. The layering of multiple less-than-perfect detection and mitigation strategies throughout the network amounted to a whole greater than the sum of its parts.

Organizations that spent the time and effort to understand their infrastructure up-front through asset management and centralized logging were better positioned to react quickly and comprehensively to a new and previously unseen vulnerability.

The ability to quickly acquire and distribute indicators of compromise internally within the organization, with peers external to the organization and, in some cases, even competitors, was vital to enabling a rapid and comprehensive global response.

Not every problem requires a new solution. Good “eyes-on” combined with some straightforward changes to existing systems such as external firewalls and name servers often gave SOC teams the air-cover needed to patch systems calmly and methodically (as opposed to pulling panicked all-nighters).

Log4Shell affected all of us to a degree, but organizations that had their ducks in a row before the vulnerability was announced could mitigate it quickly, comprehensively, and methodically.