The CircleCI secrets hack: A red flag on software supply chain risk

The security incident at CircleCI this month should be a red flag for guardians of software supply chains because it exposes security risks not typically considered by security teams.

The continuous integration, continuous delivery and orchestration platform provider revealed on January 4 that it had discovered a security incident. "At this point, we are confident that there are no unauthorized actors active in our systems; however, out of an abundance of caution, we want to ensure that all customers take certain preventative measures to protect your data as well," it stated on its website.

Those preventive measures included rotating, or changing, all secrets stored in CircleCI, and reviewing internal logs for any unauthorized access starting from December 21, 2022 through January 4, or upon completion of secrets rotation.

The advice caused consternation among some customers of the company, which operates one of the largest CI/CD platforms in the industry with about 200,000 DevOps teams using it. That's because while there are best practices for managing secrets, there's a great deal of flexibility, too. Every application is architected differently. It can make finding secrets an Easter Egg hunt.

Here's why your team should consider the CircleCI hack a wake-up call on software supply chain security.

See RL's Secrets Exposed special report series: Part I: Why secrets get leakedPart II: How hackers get accessPart III: How to mitigate risk

Who let the secrets out?

In the days following the incident, CircleCI offered additional advice about rotating secrets. For example, it recommended changing OAuth tokens, project API tokens, project environment variables, context variables, user API tokens, project SSH keys, and runner tokens.

In addition, the company suggested customers add additional layers of protection to their CI/CD pipeline configuration. For example, it recommended they use OIDC tokens wherever possible to avoid storing long-lived credentials on the platform.

It also urged users to take advantage of IP ranges to limit inbound connections to their systems to known IP addresses and use Contexts to enable the sharing of environment variables across projects, which can then be rotated automatically via API.

In addition, it promoted using runners for privileged access and additional controls to allow the connection of the CircleCI platform to a customer's compute and environments, including IP restrictions and IAM management.

OK, the fire's out, but causation is key

The process to follow when encountering a security incident is similar to the one followed by firefighters. The first thing to do is put out the fire. CircleCI did that by advising its users to rotate their secrets.

Next on a firefighter's agenda is determining the cause of the fire. That's what CircleCI is doing now. It has launched an investigation into the incident and will release a report next week. These investigations can get complicated. People think these events are as simple as A causes B, but in reality, there can be a host of interconnected events.

A network is penetrated. Privileges are escalated. Network traffic is monitored and the location of software repositories identified. A very robust process must be followed during one of these investigations to get the answers needed.

Applications today are complex. They're developed by teams all over the world. A complete investigation needs to find every instance of malicious code. If they're not found, there's a risk that a compromise will happen again.

This isn't the first time CircleCI has been hacked. In 2019, CircleCI was hit by a data breach after a third-party vendor was compromised. Hackers then compromised user data, including usernames and email addresses, usernames and email addresses associated with GitHub and Bitbucket, along with user IP addresses.

Also, at the end of 2022, the company alerted users that fake CircleCI email notifications were being used to steal GitHub credentials and two-factor authentication codes.

Looks at software supply chain risk through a new lens

What the CircleCI incident illustrates is that organizations have to not only be concerned about malware being injected into a compiled object or deliverable, but also of the tooling used to build them.

A typical software supply chain starts with the Integrated Development Environment (IDE) producing code that goes into the software repositories, which are used during Continuous Integration and distributed through Continuous Delivery to the cloud or a data center.

When assessing supply chain risk, the focus is on the artifact as it moves through the process, but risks can be lurking in the tools that make up the process itself. If all the testing is done on the artifact—whether it's software composition analysis, static application security testing, penetration testing or whatever — something will be missed unless the core competencies, like IDE and CI/CD, are examined.

If an organization looks only at the artifact and not the structure that builds it, that organization potentially has a software supply chain risk.

Software supply chain risk isn't just about the code or the compiled artifact, it's the technologies, the tooling, that actually creates the artifact itself. That's why the CircleCI hack is an eye opener to a lot of organizations out there.

Security teams can't ignore the security of an organization's CI/CD process or DevOps process tooling. They need to think about how those processes and tools may be compromised and how to protect them from this type of incident. If they are compromised, they can poison the artifact itself.

See Matt Rose's related ReversingGlass episode for a walk-through of the CircleCI hack:

See RL's Secrets Exposed special report series: Part I: Why secrets get leakedPart II: How hackers get accessPart III: How to mitigate risk