The Cloudflare source code breach: Lessons learned

In a post on February 1 co-written by Cloudflare CEO Matthew Prince, CTO John Graham-Cumming, and CISO Grant Bourzikas, the company revealed the details of a November 2023 malicious campaign that lasted roughly 10 days and saw malicious actors make off with 76 different source-code repositories from Cloudflare’s Atlassian Bitbucket source-code repository.

Cloudflare tied the incident to a breach at online identity provider Okta, which happened in October 2023. In that incident, malicious actors compromised the company’s internal network and downloaded a report that included names and email addresses of all clients that use its customer support system. The company said the malicious actors behind the campaign also accessed “additional reports and support cases” that contained contact information for Okta-certified users, some Okta employee information, and Okta Customer Identity Cloud (CIC) customer contacts.

The Okta breach led to an earlier attack on Cloudflare, disclosed in October, in which attackers used an authentication token retrieved from Okta to “pivot into” Cloudflare’s Okta instance. The company said it detected and blocked that attack at the time with no discernible impact on its security posture.

Here's what we know about the latest Cloadflare breach – and lessons your team can learn from the incident.

Special Report: The State of Software Supply Chain Security (SSCS) 2024Download Report: State of SSCS

Okta: The breach that keeps on giving

In the latest attack, the threat actors leveraged a set of Cloudflare credentials exposed in the Okta breach that were meant to be rotated but that the company overlooked. Those included a Moveworks service token that gave the attackers access to Cloudflare’s Atlassian system and credentials for three service accounts leaked in the Okta compromise: one used by the SaaS-based Smartsheet application with administrative access to our Atlassian Jira instance; another for a Bitbucket service account used to access Cloudflare’s source-code management system; and a final set of credentials for an AWS environment with no global access —and no customer data or other sensitive data, Cloudflare said.

With those credentials in hand, the threat actors had what they needed to establish a beachhead within Cloudflare’s environment with administrative access to Atlassian Jira. That allowed the threat actors to leverage the Atlassian Bitbucket Git archive feature to download the 76 repositories to the Atlassian server, from which they could be exfiltrated to an external source. While Cloudflare said it does not have proof that the repositories were exfiltrated, it assumes they were.

According to the post, the repositories that were downloaded contained source code related to key Cloudflare functions, including how backups work, how the company’s global network is configured and managed, and how Cloudflare manages identity and remote access. There was also code related to Cloudflare’s use of Terraform and Kubernetes. (Terraform is an infrastructure-as-code application made by HashiCorp.)

Cloudflare was quick to point out that customer data was not compromised in the attack, noting the following in its post about the incident:

We saw no evidence whatsoever that the threat actor got access to our global network, data centers, SSL keys, customer databases or configuration information. …Their access was limited to the Atlassian suite and the server on which our Atlassian runs.

Cloudflare

It's no secret that stolen code can be a problem

If Cloudflare’s actual operating environment and customer data weren’t compromised, is the theft of raw source code a concern? The answer, say security experts, is "yes."

Ashlee Benge, Director of Threat Intelligence at ReversingLabs, said leaked source-code could include exploitable software vulnerabilities, plaintext passwords, certificates used for authentication, and more.

There is a lot of stuff hidden in source code.

Ashlee Benge

Eric Milam, senior cyber security executive at ReversingLabs, said that with access to the raw source code, attackers can also gain an intimate knowledge of how a company’s systems fit together.

Removing the black box is another big advantage that may show its weakness over time.

Eric Milam

Even as Cloudflare cast doubt on the raw value of the code itself in its post, the company acknowledged the risks of exposed secrets and code flaws.

Our focus was not on someone having access to the source code, but whether that source code contained embedded secrets (such as a key or token) and vulnerabilities. [However], a small number of the repositories contained encrypted secrets, which were rotated immediately even though they were strongly encrypted themselves.

Cloudflare

In the wake of the attack, in an internal all-hands-on-deck effort dubbed Code Red (no, not that Code Red), Cloudflare directed its engineering teams to scour the stolen source-code repositories for code vulnerabilities and other avenues for a malicious actor to “mount a subsequent attack.”

That robust effort to remediate the attack may prove sufficient to ward off follow-on attacks leveraging secrets or intelligence gained from the source code. However, Cloudflare should be on guard for the myriad ways that exposed code might undermine its security. Those range from references to particular staff or developers that might lurk in source code comments, or metadata and that could provide valuable intelligence for future attacks.

I would be preparing to spearfish them after the buzz had died down about this compromise.

Ashlee Benge

Third-party risk and the scourge of leaked secrets

This latest Cloudflare incident underscores the continuing risks to organizations posed by both third-party software and leaks of development secrets alike.

Cloudflare’s zero-trust security model appears to have greatly limited the scope of the breach (at least by the company’s account). However, attackers' ability to leverage credentials stolen from a third-party identity provider (Okta) and gain persistent access to Cloudflare’s development environment underscores the acute exposure that even sophisticated firms have by way of third-party providers. A similar lesson lurked in the January 2023 disclosure by the firm CircleCI that it had been compromised.

Being prepared means being aware of what secrets and risks lurk in your code. ReversingLabs' State of Software Supply Chain Security 2024 reported the discovery of 40,000 secrets leaked in 2023 alone from four major open-source platforms: npm, PyPI, RubyGems, and NuGet. Those included embedded and encrypted private keys, web service API keys and access credentials, and plaintext credentials buried in network protocol strings.

While not all such credentials pose a security risk (placeholder and expired credentials may not pose any risk), organizations need to be aware of the security exposure that source code poses and the varieties of sensitive information that may lurk in both open-source and proprietary code repositories.