After hack, CircleCI tells devs to update secrets now

A security breach of the CircleCI development platform has exposed security tokens and other secrets used by more than a million developers, the company said in a statement on Wednesday.

CircleCI is urging its users to immediately rotate “any and all secrets stored in CircleCI,” including API tokens and secrets stored in environmental variables or contexts. CircleCI users are also urged to review internal logs for their systems for evidence of “unauthorized access” starting on December 21st, 2022 and running through January 4th, 2023.

The incident is just the latest in which popular, hosted development platforms have been targeted by malicious actors intent on gaining access to raw source code, or stealing credentials and other information that can be used in downstream attacks on development organizations and their customers.

See ReversingGlass with Matt Rose: CircleCI and Software Supply Chain Risks

CircleCI is a popular tool used by development organizations that practice continuous integration, continuous development (CI/CD). The platform is used by software developers to automate the building and testing of submitted code and to notify developers about problems with their code.

The company said it is investigating a “security incident” and that investigation is ongoing, according to the posted statement by CircleCI Rob Zuber. He did not provide any information on how or when the breach was detected, but said CircleCI will share more details with customers in “the coming days.”

In an update posted to the company's website on Thursday, CircleCI provided no new information on the circumstances or extent of the breach. However, it did assure its customers that the platform was safe to use for building code. It also provided details on the types of credentials that should be refreshed, naming OAuth tokens, Project and User API tokens, environmental variables and Project SSH keys as in need of updating.

The company also provided a free tool for customers to discover secrets in their CircleCI projects and provided additional recommendations to prevent or reduce the chances of CI/CD compromises. Among those: using Open ID Connect (OIDC) tokens with finite lifespans, instead of long-lived credentials and using IP ranges to limit inbound connections to known IP addresses.

As noted by TechCrunch, CircleCI has been the victim of attacks before. In November, the company warned its users to be on the lookout for phishing attacks in which cybercriminals impersonate CircleCI to gain access to code repositories on GitHub. The company’s customers were also affected by a 2019 breach at a third party analytics firm that CircleCI contracted with.

Malicious actors are taking greater interest in development organizations and platforms as they look for unobstructed paths into sensitive IT environments. In addition to CircleCI, a vulnerability in the TravisCI in 2021 exposed secrets on hundreds of thousands of open source projects that use the platform. A report in June found tens of thousands of user tokens were likewise exposed through the Travis CI API, which provided unfettered access to more than 700 million historical clear-text logs.

Recent months have also seen major corporations impacted by the leak of secrets and sensitive information stored in code repositories. For example, in March, 2022, Samsung and Nvidia both had hundreds of gigabytes of internal source code leaked by the Lapsus$ hacking group.

An analysis of the leaked Samsung code by the firm GitGuardian revealed that close to 7,000 secrets stored in the code were revealed in that leak. Then, in October, Toyota revealed that credentials for a database containing personal information on hundreds of thousands of customers were left exposed in an open source repository associated with a contractor who had worked on the company’s telematics application for five years before being detected.

The rapid pace of software development, a growing reliance on open source code and the ease with which code is shared and re-used facilitate compromises and can make it difficult for development organizations to understand and address the risk posed by source code leaks and exposure.

"The CircleCI hack should make us realize that it is just as important to secure the DevOps supply chain tooling as it is the software and applications they compile," said Matt Rose, a Field CISO at ReversingLabs.

As ReversingLabs noted in Flying Blind: Software Firms Struggle to Detect Supply Chain Hacks, organizations are attuned to the risk posed by vulnerable software supply chains but lack the expertise, staff and budget to address the risk. Four in 10 of those surveyed by Dimensional Research listed CI/CD toolchain exposures as posing a risk to their organization. More than 60% said threats hidden in open source repositories posed a risk.

Software development shops should take a queue from the manufacturing sector, where supply chains and manufacturing environments are tightly controlled, Rose said. "Car manufacturers make sure the cars their assembly lines manufacture are kept in a locked space but they also ensure that unauthorized personnel are not allowed to access or manipulate the assembly line itself. This is a new opportunity for hackers to disrupt the software supply chain that needs attention," he said.

Update Jan 6, 2023: This blog post has been updated to include information from CircleCI's latest statements regarding the security breach.