Shai-hulud attack: Don’t let worms eat DevOps

Close to one week after the cascading Shai-hulud worm compromised hundreds of npm packages, GitHub — which is responsible for the open-source repository — announced that it would be slowly rolling out new security measures in response to the supply chain attack. 

The new policies will change how authentication and publishing works on npm in the form of local publishing that requires two-factor authentication (2FA), limited lifetime granular tokens, and the establishment of trusted publishing practices, in accordance with the Open Source Security Foundation (OpenSSF). GitHub will also support these changes by migrating users to FIDO-based 2FA, limiting how tokens can grant publishing access and more.   

After the npm account-takeover chaos of the past three weeks — from the account compromise of Qix to Shai-hulud — this update from GitHub is welcome. As I pulled my threat research team at ReversingLabs (RL) out of bed early Monday morning last week to begin mitigating the Shai-hulud worm, I had a gut reaction: Something has to change, because we can’t keep doing this every week. 

Figure 1: 5 out of the top 10 most downloaded npm packages have been compromised in the last three weeks.

RL monitors millions of open-source packages across ecosystems like npm, PyPI, RubyGems, NuGet, and VS Code. During our continuous monitoring on Monday, things got weird. Dormant npm maintainer accounts began pushing new versions of every package, all containing the same suspicious behaviors. That list included a number of npm packages that basically function as configuration files without any coded behaviors. 

The finding was a “canary in the coal mine” — an early signal that something was wrong. The RL team rushed to trace down the source of what we now know as Shai-hulud, a first-of-its-kind worm with a self-replicating malware that preys on open-source packages. The worm borrowed its name from the massive, voracious sand worms from the sci-fi classic “Dune.” 

More than 360 npm packages were compromised, including @ctrl/tinycolor (2.2M weekly downloads), ngx-bootstrap (300K), and ng2-file-upload (100K). The packages infected by Shai-hulud spawned attacks of their own by allowing the worm to infect packages linked to the compromised maintainer's account. This allowed the malicious actors behind the attack to plant malicious code, scan for developer secrets, and exploit the GitHub API and GitHub Actions to expose non-public code repositories and exfiltrate any discovered secrets to an external site controlled by the hackers.

[ Watch my Q&A about the Shai-hulud | See RL's Shai-hulud research post]

Wanted: Security barriers for OSS

Within days, most of the compromised npm packages were restored and the worm’s spread halted, while the dev and infosec communities’ attention shifted elsewhere.

But as GitHub has made clear with its new security measures, it would be a mistake to sweep the Shai-hulud outbreak under the rug. True, the outbreak was limited (npm hosts more than 3 million packages) but it proved that a self-propagating malware can automate repository compromise. In other words, unless something changes, another worm is inevitable, and the only unknown is when. The question now is: How do we protect DevOps infrastructure against such inevitable threats?

Break glass in case of worms

Software registries and open source repositories are sources of a lot of goodness. Registries like npm accelerate development and community growth, but security has lagged. PyPI, with more than 660,000 Python packages, has only two full-time security staff. RubyGems, with 175,000 packages and 236,000 users, has just one. And npm caps the number of malicious packages (3) any user can report in a single day. 

While the publishing and authentication changes being brought to npm in the coming months are a step in the right direction, what is needed is a last line security barrier for these popular platforms that will prevent threats like Shai-hulud from spreading. That barrier could be a “break glass” feature that the registry can employ when faced with an outbreak such as we observed on Monday — shutting down new account creation, package creation and updates and other modifications. PyPI has such a feature and employs it frequently to stop malicious campaigns.

Integrity and transparency are essential

Shai-hulud underscored the importance of transparency and integrity across open-source ecosystems, dependencies, and CI/CD pipelines. To respond, developers and development organizations must back efforts to strengthen both, which GitHub agrees with:

True resilience requires the active participation and vigilance of everyone in the software industry.

GitHub

In addition to ensuring that the phishing and social engineering of overworked maintainers is minimized — a risk GitHub’s new measures will help with — the industry needs to enable comprehensive software supply chain security. Until that happens, another self-replicating malware could worm its way into trusted development infrastructure.

After all, if you don’t know which packages you’re building software with — the content of your build pipeline — then someone else will figure it out and use that knowledge against you. Embracing package reputation; SBOMs; deferred package updates; and artifact provenance are no longer nice-to-haves. They’re essential to modern cyber defense as attackers look to compromise sensitive organizations by injecting malicious code upstream in their software supply chain.

Update your threat detection tooling

The final recommendation I have is for development teams to embrace a broader view of security monitoring for code. Traditional vulnerability and secrets scanning is no longer enough. A vulnerability might get exploited and give you a headache. With malware, there is no doubt. If it was deployed anywhere in your environment, you were affected. Of course, malware is not a vulnerability, but it may rely on the exploitation of a software vulnerability or other weaknesses in your defenses. And Shai-hulud used it to walk out the front door with all of your secrets — secrets that it will happily abuse to start another series of attacks. 

The software supply chain is complex. It requires augmentation of traditional security checks with more nuanced, behavioral-based detection that can spot malicious code and other anomalies. That capability is a core component of RL Spectra Assure, which gave us an early warning last Monday about the fast-spreading Shai-hulud outbreak. It is the future of defense against threats to both open source and proprietary software supply chains.

Complexity is the enemy of resilience

At the end of the day, cyber resilience won’t come from piling on complexity. That’s because attackers exploit complexity for their own means. In response, defenders must simplify their processes with a focus on supply chain integrity. In the process, they should adopt tools to detect malicious behaviors and dependencies, regardless of needed efforts to minimize maintainer account-takeovers. These software supply chain security measures won’t end attacks for good, but they will raise the bar and block many noisy, disruptive campaigns — for the benefit of everyone.