To secure your CI/CD pipelines, round up the usual suspects

Attacks on software supply chains are receiving a lot of attention these days within the broader information security community and in the marketplace. As we noted, talks on software supply chain risks and attacks were a dominant theme at the recent Black Hat Briefings in Las Vegas.

Software supply chain attacks aren’t a new thing, but their growing prominence — as evidenced by the devastating attack on SolarWinds in 2020 — pose a challenge both to software publishers, and to their customers. (Check out our analysis of the SolarWinds compromise: SunBurst: The Next Level of Stealth.)

As attackers — including nation state actors — turn their attention to development organizations and the infrastructure and platforms that support them, would-be victims need to fine tune detection and defenses to address that new risk.

So what do those new defenses and security investments look like? Well, they may look pretty similar to the kinds of technologies and tools you’re already using, according to a Black Hat presentation by researchers from NCC Group. Here are some key takeaways from my video interview with them for our ConversingLabs podcast from Black Hat.

Three words: User. Least. Privilege.

“Generally the [CI/CD] security principles have been around for as long as security has been an issue,” said Iain Smart, the Containerization and Orchestration Practice Lead at NCC Group, in an interview. "That includes foundational security technologies and processes like role based access control (RBAC), secrets management and threat modeling,” Smart said.

Despite the popular image of supply chain attacks and attackers as ultra stealthy and sophisticated operations, Smart said his experience auditing development and IT organizations suggest that most supply chain attackers are picking low hanging fruit.

You have people running [the development] pipeline as administrator users rather than restricting their privileges; you have people allowing components to have full administrative access over the cloud rather than only what they need. Things are not as locked-down as they generally should be.

Iain Smart

In fact, many of the supply chain security assessments that NCC does may not start that way, said Viktor Gazdag, who is a Managing Security Consultant at NCC Group and a Jenkins Security MVP. “Sometimes it doesn’t even start off with the [development] pipeline. We might start off with a web application [assessment] and then we got access to a CI/CD pipeline as well."

It’s not unheard of for us to stumble onto a [development] pipeline and then go ‘Oh, this looks like a target, let’s see what we can do with it. The impact is quite bad in some cases.

Viktor Gazdag

Lots of vulnerabilities

The NCC researchers said that common vulnerabilities are also rife in development plugins and third party modules. Gazdag said that research he conducted on Jenkins plugins found lots of examples of common flaws like stored credentials, cross site scripting (XSS) flaws and server side request forgery (SSRF).

He said development organizations needed to be much more mindful of the provenance of the modules and plugins they were relying on. Those developed by large organizations like The Jenkins Project generally clear a high bar for security. However, most plugins for Jenkins and similar platforms are developed by individuals or third party firms, with little attention or effort given to code security and upkeep.

It's like open source code that you can use, but if the developer says like, ‘Okay, I'm no longer interested in maintaining the plugins,’ or ‘I don't have time to fix the vulnerability,’ then the vulnerability will be there.

Viktor Gazdag

Mind your (development) traffic

Network monitoring is also a sore point for development organizations. Smart and Gazdag said that their work with customers often revealed blind spots in network monitoring associated with development groups and infrastructure — a blind spot that enabled red teams to use build servers to pull malicious dependencies from Internet-based platforms like Jenkins into internally managed repositories.

“They weren't doing anything to stop their build servers talking out to the wider Internet,” Smart said. Something as simple as a firewall rule could stop such an attack, but the reliance on external dependencies complicates the implementation of such measures. “Generally what we recommend is pinning your version whenever possible,” Smart said.

The ephemeral nature of supply chain attacks also poses problems for organizations. Malicious code transmitted via a compromised dependency can subsequently be overwritten, leaving the downstream consumer of that malicious dependency with what changes were made or what level of access attackers had as a result of the compromise, the researchers said. That’s especially true for organizations that are leaning heavily on containerization, in which things are designed to be replaced as quickly as possible, he said.

There's every chance that a package is compromised for a very short time. You ingest that package and by the time you realize something's gone wrong, you've rebuilt your application, the public version has been patched, there's no record and you have no idea what the impact was exactly.

Iain Smart

The solution? Plan for the worst. “If you're using an external dependency, mirror it internally and make sure that you have a reproducible build,” Smart said. “That way if a package is compromised, you have a local version to do analysis on."

Hope for the best, log for the worst

Failing everything else, Gazdag and Smart said that security concerns are unlikely to dampen development organizations’ desire to "move fast and break things," so they also need to be prepared for what happens when security risks slip through the cracks. Doing security assessments and testing of every commit is great — so long as you have the staff to review and triage the output from all that testing.

Failing that, detailed logging of production applications is the best way to identify and reconstruct software supply chain compromises.

We can fix it!

If there’s a silver lining to the researchers' presentation, it’s that most of the problems g discovered and documented are easily addressed and don’t require additional investments.

Most of what we've seen have been misconfigurations rather than fundamental flaws in the software. So a developer or an admin has ticked the wrong box or has put a secret somewhere they shouldn't have, and it's just not been found until we came along and pointed it out.

Iain Smart

When those flaws were pointed out, development teams were quick to address and remediate them. “I wouldn't say there's a systemic problem in what we've been looking at, or a need for additional software,” Smart said.

Organizations just need to pay careful attention to the way you're deploying what you've got.

Check out the full ConversingLabs conversation with Smart and Gazdag.