Federal security guidance: Been there, done that

The U.S. Cybersecurity and Infrastructure Security Agency (CISA) and the National Security Agency (NSA) are telling development organizations to tighten up the security of their development pipelines or face the risk of damaging software supply chain attacks.

The two federal agencies issued a Cybersecurity Information Sheet (PDF link below) late last month that provides recommendations for securing CI/CD (continuous integration/continuous delivery) systems.

The CI/CD pipeline is a distinct and separate attack surface from other segments of the software supply chain. [Malicious actors can] multiply impacts several fold by exploiting the source of software deployed to multiple operational environments.

CISA/NSA Cybersecurity Information Sheet

The Cybersecurity Information Sheet follows high-profile compromises of CI/CD platforms, including that of CI/CD vendor CircleCI in January, 2023, which led to a warning to firms to rotate any secrets that may have been stored in code hosted on the platform. It also maps risks to the MITRE ATT&CK framework, a taxonomy of adversary tactics and techniques based on real-world observations. It provides a list of common risks found in CI/CD pipelines as well as attack surfaces that could be exploited.

Here's what the new federal guidance for CI/CD security covers — and what's missing.

Get eBook: Why Traditional App Sec Testing Fails on Supply Chain Security

Multiple CI/CD attack scenarios outlined

The scenarios considered in the CISA/NSA document include attacks by malicious cyber actors who obtain credentials giving them access to critical development infrastructure such as a Git repository, a CI/CD service, or a server hosting a source-code repository. Also considered are scenarios involving a supply chain compromise of an application library, tool, or container image that is part of a CI/CD pipeline, as well as a CI/CD compromise that results in malicious code or dependencies being added to source code.

The mitigations proposed in the guidance break down into three categories, which the document terms “Active Hardening.”

1. Clean up authentication

First, the CISA and NSA propose a number of improvements to authentication and access for development environments. Those include minimizing the use of long-term credentials in favor of temporary or ephemeral alternatives. The guide also recommends implementing least-privilege access rules for users’ accounts and CI/CD environments and two-person rules for updates to core source code, meaning multiple individuals must sign off on check-ins of code.

Development organizations need to carefully manage secrets, tokens, and other credentials, as well, the agencies said — avoiding passing secrets in plaintext and making sure that secrets such as passwords and private keys are not left embedded in software.

2. Secure development environments

Within development environments, organizations need to do a better job of keeping development systems and CI/CD tools up to date with security patches and other fixes, the Cybersecurity Information Sheet document says. Organizations should “scan for vulnerabilities and keep software updated,“ as well as hardening development systems by removing unneeded applications and implementing endpoint detection and response (EDR) tools to detect malicious activity.

3. Shore up code security

Finally, the Cybersecurity Information Sheet document provides a number of recommendations for shoring up the security of developed code. They include applications of standard secure coding tools and technologies such as static and dynamic application security testing (SAST and DAST) as well as the application of software composition analysis (SCA) tools and software bills of materials (SBOMs), which map out the contents of software packages, including internally developed, third-party, and open-source components in the codebase. Vulnerability management teams working within development organizations need to correlate SBOM data to known common vulnerabilities and exposures (CVEs). And SBOMs should reflect vulnerabilities as they are found: “Once an SBOM is available for a given piece of software, it needs to be mapped onto a list of known vulnerabilities to identify the components that could pose a threat. NSA and CISA recommend connecting these two sources of information,” the CSI document reads.

On the issue of supply chain compromises, the agencies recommend a number of remediations. They include analyzing committed code using manual and automated tools to spot vulnerabilities and changes in source code and restricting the use of “untrusted libraries and tools” within development environments.

Missed opportunities on software supply chain security?

As written, the Cybersecurity Information Sheet document provides a succinct account of accepted best practices around securing development environments. But the CISA and NSA may have missed an opportunity to orient software makers, government contractors, and private-sector firms to the new kinds of threats and attacks that are becoming prevalent.

Specifically, the Cybersecurity Information Sheet fails to address the complexity of modern software supply chain attacks, which might involve compromises of core development infrastructure and see the deployment of malicious or compromised third-party and open-source dependencies.

On the issue of code scanning, for example, the Cybersecurity Information Sheet focuses on existing tools and technologies such as SAST and DAST to find evidence of code vulnerabilities. But incidents such as the 2020 hack of SolarWinds Orion and the more recent hack of voice-over-IP provider 3CX have shown that those technologies often miss more subtle manipulations by sophisticated cybercriminal and nation-state actors.

The guidance from the CISA and NSA calls out the threat posed by untrusted software libraries and tampering with committed code. However, its guidance on how to address those risks is sketchy. Software makers are urged to take a “never trust/always verify” approach toward software to limit the CI/CD attack surface, and to “only use software, tools, libraries, and artifacts from secure and trusted sources.” However, there is no guidance given on how to verify the trustworthiness and integrity of any of these elements.

“Employing software from a trusted source” minimizes threats to CI/CD pipelines, the CISA and NSA say. But how are software makers supposed to determine which sources are trusted and which aren’t? That question is left unasked, although it addresses the very challenge the Cybersecurity Information Sheet document is aiming to solve.

Instead, the CISA and NSA acknowledge that “no automated security scanning tool is perfect” and that software makers should be prepared to “manually review the code and the CI/CD pipeline” or use automated tools to find “potential security vulnerabilities in the code and track changes over time.” Development teams should try to ensure that “only approved changes are made to the code base and that any potential security vulnerabilities are addressed before they can be exploited,” the CISA and NSA say.

But recent incidents show that such manual efforts are difficult to scale and doomed to fail. The dynamic nature of CI/CD environments makes manual code reviews impractical. Sophisticated attackers take pains to mask malicious code additions, making them indistinguishable from authorized code, or ferry their attacks in third-party modules into which reviewers — human or automated — have limited visibility.

The dynamic nature of CI/CD environments makes manual code reviews impractical. Sophisticated attackers take pains to mask malicious code additions, making them indistinguishable from authorized code, or ferry their attacks in third-party modules into which reviewers — human or automated — have limited visibility.

That was what happened in the SolarWinds incident, in which malicious actors (believed to be working on behalf of Russia’s SVR intelligence service) carefully studied and then mimicked internal code to plant a malicious backdoor in an update to the SolarWinds Orion application. It is also the backstory to the hack at 3CX, whose compromised 3CXDesktopApp was analyzed by ReversingLabs researchers who flagged tampering done with d3dcompiler, a standard library used with OpenJS Electron applications such as 3CXDesktopApp and ffmpeg, another standard file that had been modified to reference malicious code appended to d3dcompiler.

Post compilation analysis: The road to software security

An approach we advocate to address this risk is what might be referred to as the "software supply chain final exam.” The final exam presumes that development teams follow all of the best practices advocated by the Cybersecurity Information Sheet: chasing down vulnerabilities in code, as well as open-source and third-party dependencies, using SAST, DAST and SCA; shoring up the security of developer accounts and development infrastructure; and so on. But it adds an integrity check of the compiled package, post-compilation but pre-deployment.

The goal is to detect the kinds of compromises that bedeviled SolarWinds and 3CX, where the standard supply chain security boxes were “checked” but development and application security teams failed to discover a larger compromise of the development pipeline. The ultimate objective is a reliable method for development organizations to realize reproducible software builds that ensure that malicious code or functionality has not been added to a compiled binary before it is shipped to end users.

It's time to get comprehensive with supply chain security

The CISA and NSA’s new document shines a light on threats to CI/CD environments. As such, it will make development organizations more attuned to supply chain threats.

However, by limiting the scope of their recommendations to what amounts to common software supply chain wisdom, the federal agencies missed an opportunity to focus attention and resources on the complexities of modern supply chain threats and attacks, which increasingly extend beyond exploitable software vulnerabilities. It’s time for the federal government — and the private sector — to broaden the scope of efforts to take such emerging threats into account.