Lessons learned from the CircleCI secrets breach

Software supply chain security has become a top priority for organizations, but new threats continue to surface that security teams need to be aware of. A harsh reality hit the industry this new year when CircleCI, a continuous integration/continuous delivery (CI/CD) and orchestration platform revealed on January 4 that it had discovered a security incident. 

The company later shared that assailants compromised a development system used by a remote CircleCI engineer. The threat actors then used this access to plant malware and steal data, which included customer environment variables, tokens and keys from CircleCI’s production systems. Also, the stolen data allowed the threat actor to access the third-party systems of several CircleCI customers. 

This CircleCI breach, which Field CISO Matt Rose noted in a blog post was a red flag for software supply chain security, is a textbook example of how secrets leaks, such as the exposure of tokens and keys, are detrimental to organizations and their customers. (Learn more in this episode of ReversingGlass: What the heck are secrets?) 

To discuss this breach and better understand its causes and industry impact, ReversingLabs hosted a recent webinar: Secrets Revealed: CircleCI’s Breach and Lessons Learned. In this conversation, Rose and Chris Wilder, Research Director at TAG Cyber, discuss what organizations should take away from this breach. They also determined what organizations should do in the wake of the breach to properly secure secrets in their software supply chains. 

Here are the key takeaways from their conversation. 

See Webinar: Secrets Revealed: CircleCI's Breach and Lessons Learned

The CircleCI breach tells a bigger story

In their conversation, Rose and Wilder noted the broad implications of the CircleCI breach, with Wilder noting, “this was a massive lapse in hygiene.” He also cited that the use of third-party code, as was the case with the CircleCI breach, is “causing a lot of chaos.” 

Wilder has a unique perspective from on the security operations side from his work at TAG Cyber, where he communicates frequently with CISOs and security teams about best cybersecurity practices and the problems they are facing. He believes that “if you have good cyber hygiene, these problems (secrets leaks and other supply chain threats) aren’t likely to come up.” 

Rose tackled the issue from the perspective of development teams, noting that software engineers are working around the clock to stay in line with production timelines, and the speed of software delivery is constantly increasing in the age of CI/CD.

It’s all about speed. Everyone wants to go faster [and that leaves] security in the backroom.

Matt Rose

That speed is why security teams need a seat at the table when it comes to production and tooling, said Wilder. He said the CircleCI breach demonstrates that DevOps and DevSecOps teams must come together to handle issues like secrets leaks. 

The need for better app sec tooling

One hindrance to securing software supply chains from secrets leaks and other threats is traditional application security (app sec) tooling.

A lot of app sec technologies are just too slow and can’t keep up with the speed of DevOps.

Matt Rose

Rose was referring to tooling such as static app sec testing (SAST) and even dynamic app sec testing (DAST), which do not meet modern security needs for software supply chains. Wilder agreed, noting that there is a “false sense of security” when security teams only use such traditional app sec tools, since these technologies don’t provide 100% coverage of software supply chain threats. 

The two said they hoped organizations will begin to embolden their teams to use the right tools, and pay attention to the factors that could cause supply chain risks like secrets leaks. When it comes to organizations charging their DevSecOps teams with the responsibility of defending against these leaks, for example, that they “need a tightly defined security policy,” Rose said. Wilder noted that areas such as incident response and inventory management should also be a part of this policy. 

For inventory tracking, Wilder stressed that software bills of materials (SBOMs) are a great start, and require “putting hygiene upfront” for software development organizations.

The creation and managing of SBOMs is “a continuous operation,” and DevSecOps teams will need to put in the work to update and analyze relevant SBOMs on a constant basis, Wilder noted. This requires modern tooling that incorporates automation and analysis of supply chain threats, including secrets leaks.

See Webinar: Secrets Revealed: CircleCI's Breach and Lessons Learned