Secrets security: The why, the how – and what to do about it

For software development teams to maintain and properly set up development environments and pipelines, they need to use software secrets such as environment variables, tokens and keys in these processes.

With the speed and complexity of modern development, software teams leave these secrets exposed in both public and private software repositories. Attackers have caught on to this, and are now scouring these repositories for secrets that they can exploit.

Securing and managing secrets is yet another facet of software supply chain security task that security teams need to be aware of- and disciplined in their approach to. That's why we put together a new special report series: Secrets Exposed.

Our new special report examines how and why attackers have been successful at exploiting development secrets leaks. It also provides useful advice about how to prevent future breaches, and discusses best practices for securing and managing secrets in your code.

Here are key takeaways from the special report series.

Part I: Why secrets get leakedPart II: How hackers get accessPart III: How to mitigate risk

The problem: Modern dev practices leave secrets exposed

This latest epidemic hitting the software industry is a result of changes in the application development ecosystem in recent years. Software development has become far more complex and decentralized, relying on ever-longer supply chains that include open-source software and third-party code to get products across the release finish line. There has also been a shift in tooling, with development activity and source code control moving from vendor-owned systems within protected networks to shared, cloud-based development platforms and repositories. That has made it easy for developers to move and re-use code seamlessly between projects.

But with each advancement in modern development processes, new threats arose that are now causing major software supply chain security risks. Secrets exposed via these same platforms and repositories is one of the biggest risks out there. Attackers have turned their attention to these popular repositories, such as PyPI, npm and GitHub to look for organizations’ secrets left accidentally exposed by developers. Unfortunately, these exposed secrets are plentiful. For example, GitHub found over 1.7 million potential secrets exposed in public repositories in 2022 alone, demonstrating that secrets exposures are now an epidemic to software supply chain security.

Figure: Secrets detected in packages residing on four major code repositories, broken down by affected service. Source: ReversingLabs

And GitHub isn't alone. Secrets pertaining to a number of prominent services have been detected across several open-source repositories, including npm, PyPI and RubyGems, as can be seen in the figure above. These leaks leave the organizations that are dependent on those platforms vulnerable to software supply chain risks.

This problem of secrets exposures also isn’t limited to just open-source platforms. A major contributor to this epidemic is that developers are also exposing these secrets in their own personal environments.

Responsibility for the problem of leaked secrets is widespread, but developers share much of the blame. Gitguardian found that 85% of secrets leaks occurred in developers’ personal repositories, finding that the problem was widespread in both open and closed developer platforms.

Figure: It only takes 20 seconds or less for a bad actor to find secrets in code repositories, and 85% of secrets leaks come from developers' personal code repositories. Source: ReversingLabs & GitGuardian

Just as concerning: secrets exposures often go unnoticed for long periods of time, giving threat actors even greater opportunity to exploit organizations’ development environments. According to IBM’s 2022 Cost of Data Breach Report, the average length of time before leaked credentials were identified was close to one year (327 days). Even more worrisome, attackers can find secrets much more quickly than security teams can detect them. They do this by taking advantage of these platforms’ powerful assessing, analyzing and sharing capabilities, allowing bad actors to spot a secrets leak in a median time to discovery of just 20 seconds.

Attackers have it easy with secrets

To spot secrets leaks, bad actors often exploit lapses and breakdowns in the open source platforms that host them. For example, ReversingLabs reverse engineer Karlo Zanki spotted an unfortunate mistake by analyzing a PyPI package named dynamo_lock, revealing that it contained web service access credentials for an Amazon-owned database and Simple Queue Service (SQS) as well as Gugya’s identity management platform. Zanki said the exposure arose as a result of the developer failing to notice the inclusion of the secrets file in the release artifacts that were produced prior to publishing.

While this is just one example of a mistake that developers can make, this type of accident occurs consistently.

Figure: Number of secrets exposed in three popular code repositories to date: PyPI, npm and RubyGems. Source: ReversingLabs

Not only are attackers aware of the fact that developers are prone to making such mistakes, but they are also able to use automation to easily detect and exploit these accidents. Because secrets typically end up in the same platforms and exhibit similar characteristics, attackers can track them as they become exposed in real-time. They do this by writing scripts that leverage search engines to look for telltale patterns, such as ‘api’ or ‘log.txt.’

Securing and better managing secrets: Where to begin

Considering how prominent and serious this epidemic of secrets leaks is, developers can start to tackle this problem by making proactive changes in their build processes.

First, developers need to begin scanning application code for secrets, which will allow them to identify and track any secrets and then determine whether or not it is safe to be included in the build of a software artifact. There are a plethora of free scanning tools available to developers, but they can be noisy: warning about credentials that are actually benign and have no value to threat actors. This is why dev teams should invest in tooling that filters out this noise and pinpoints the exposures that actually pose a threat.

In the event that secrets have been exposed, development teams need to be able to assess the damage stemming from that exposure, then remediate the exposure by suspending, reissuing or, at the very least, rotating exposed secrets that attackers could take advantage of. Organizations should also take steps to determine if the exposed secrets have been exploited by malicious actors, and if so, how.

Looking beyond immediate incident response, development organizations should consider investing in a solution that will allow them to effectively review exposures and mitigate them in real-time, allowing them to minimize their software supply chain security threat. As ReversingLabs field CISO Matt Rose wrote recently, only an end-to-end security approach can deal with secrets and software supply chain security.

Pursuing a holistic approach to software supply chain security is the only way to bring the problem of secrets sprawl under control. Through defense-in-depth — addressing all the activities associated with supply chain and application security risk — you can avoid looking at the problem through a single lens and get a true picture of the risks in your supply chain.

Matt Rose

Don’t wait for a secrets breach to hit your organization

Software development teams need to reconsider their processes to include the consideration of and protection against secrets leaks. Organizations looking to better understand the epidemic of secrets leaks and how to best handle them should pay attention to this latest report for guidance.

In our Secrets Exposed special report series, security teams will also learn:

• Why organizations need to take a comprehensive approach to secrets security
• How to respond to secrets breaches when they happen
• Best practices to minimize your secrets exposure risk

Part I: Why secrets get leakedPart II: How hackers get accessPart III: How to mitigate risk