Here’s what happened with Log4Shell while you were out

Champagne bottles stayed corked as development and security teams spent the New Year in a dystopian fight for survival with cybercriminals intent to exploit a hole in the Log4j open source library. Where do things stand now? We survey the (threat) landscape.

The New Year is traditionally a time for “ringing out the old” and “ringing in the new'' - for taking stock of what came before and what waits ahead.

Unfortunately, IT and security teams didn’t have that kind of luxury this New Year. Instead, they put the party hats and champagne aside and spent the New Year in something like a dystopian fight for survival. Their adversaries? Hacking and ransomware crews anxious to exploit Log4Shell, a “10 out of 10” remotely exploitable vulnerability in the ubiquitous Log4j open source logging library.

So how did that turn out? And what did a couple weeks away from the office buy us in cyber resilience? Here’s what things look like from where we’re standing:

Log4Shell: Here. There. Everywhere.

Three patches and almost one month after the anniversary of its discovery, the Log4Shell vulnerability is still ubiquitous across the globe. Much of that is due to the nature of Log4j2, the Java-based logging library that is the source of the security hole and a common component of thousands of consumer and enterprise applications, websites and services. Among other revelations in recent days: research suggesting that Log4j is heavily used in operational technologies (OT), which puts critical infrastructure at risk of compromise as well. In short: there’s still a lot of code out there that needs patching.

Sloppy response

Sloppy response to the Log4Shell flaw (CVE-2021-44228) has been an issue. Security experts have noted that disclosure of the vulnerability was decidedly uncoordinated - with initial reports of the flaw coming out of the Minecraft community and preempting the Apache Foundation’s official notice. The Apache Foundation's scramble to get fixes out for Log4Shell has not helped efforts to get affected organizations on top of their vulnerability response. On more than one occasion in the last month, for example, flaws tied to Log4j patches have been uncovered in patches from Apache, necessitating further patching and updates.

(Supply) chain of fools

Of course, Log4Shell’s persistence isn’t just because the Log4j library is so common nor because disclosure of the flaw went sideways. The persistence of vulnerable applications is also a byproduct of lax supply chain monitoring and security practices.

One grim detail to digest: despite all the attention that Log4Shell flaw got - which ranged from urgent alerts by the likes of CISA to mainstream media coverage - around 4 in 10 Log4j downloads in late December were still of pre 2.15.0 versions of the software, according to the folks at Sonatype. That’s right: even after all the headlines and scary press, versions of Log4j that contained the remote code execution flaw still accounted for a sizable minority of all downloads last month.

The failure of organizations to patch shouldn’t be interpreted as indifference or a “damn the torpedoes” attitude among development teams or IT administrators. It is a byproduct of a confused patch landscape and a free-wheeling application development environment. The fact is: developers today readily employ both proprietary and open source components to build new applications and services rapidly. They incorporate components like Log4j directly and indirectly, via third party platforms and packages. In the process of doing so, they may unwittingly “pull in” vulnerable components without realizing their exposure. Furthermore, static code analysis -and dynamic code analysis tools -focused on the security of code, but not of discrete software components- won’t alert development teams to these kinds of risks.

This decentralized and diverse software supply chain - heavily reliant on open source projects and code - means that Log4Shell’s tail is very long indeed: measured not in weeks but months or even years.

Nation-state and criminal networks press their advantage

With so many exposed applications out there, there is little wonder that malicious actors jumped on Log4Shell and just kept on jumping over the Christmas and New Year’s holiday.

In a recent post, Microsoft said that “exploitation attempts and testing” remained high during the final weeks of December. True, much of the Log4Shell related traffic is relatively low-risk “mass scanning” by attackers (or security firms) looking to identify vulnerable systems. Still, Microsoft’s security blog said that attackers are readily incorporating exploits of Log4Shell into existing tooling, from “coin miners to hands-on-keyboard attacks.”

And there have been a string of more substantial security incidents linked to the Log4Shell vulnerability in recent weeks, as well. They include:

• Belgium’s Ministry of Defense, which reported on December 20th that it was attacked by unknown adversaries who attempted to leverage the Log4Shell vulnerability
• ONUS, a Vietnam-based crypto currency exchange said that a payment system it relies on was also attacked by adversaries who exploited the Log4Shell flaw. Cox Media, which said it suffered a ransomware attack
• Universities, which have been targeted in attacks leveraging Log4Shell in recent days. A China-based advanced persistent threat (APT) group dubbed Aquatic Panda that is associated with both intelligence collection and industrial espionage.
• Microsoft observed hacking and ransomware groups affiliated with the governments of Iran (“Phosphorus”) and China (“Hafnium”) using and adapting Log4Shell in attacks.

Help is (kind of) on the way!

The good news is that progress is being made. The curve of updates for Log4j as development teams upgrade to a patched version of the logging library is bending in the right direction, though more gradually than most of us would like.

Security and threat intelligence firms have produced some outstanding analysis of Log4Shell and threats leveraging the flaw. Large tech firms have come to the rescue as well. Google, for example, released a free scanning tool for Log4Shell that can help identify vulnerable systems. Microsoft also has released or updated a long list of its security and management tools to help identify Log4Shell-exposed applications within Windows networks and Azure cloud environments. Outside of that, more than 20 firms have released Log4Shell scanners including Qualys, Huntress and others.

Leveraging any (or all) of these tools should be a part of your Log4Shell incident response plan. They can help your organization identify whether you are at risk of attack through exploitation of Log4Shell (or any of the subsequent Log4j flaws).

As we have said before: the visibility and tracking problem presented by Log4Shell is a great argument for organizations to start implementing software bills of materials (SBoMs) for critical applications and services. SBoMs can alert you to vulnerabilities in downstream components like Log4j and make the job of mitigating supply chain risks much more simple.

New Year’s resolution? Start building an SBoM

ReversingLabs assists companies in mitigating software supply chain attacks and vulnerabilities like Log4Shell. Our Managed Software Assurance Service provides advanced analysis of software packages to uncover a range of suspicious software behaviors, file tampering and other indicators of code compromise. Visit our website to learn more about how ReversingLabs can help you verify your software supply chain and manage software supply chain risks. And visit secure.software to see how we can help you generate and verify your SBOM.