The JetBrains TeamCity software supply chain attack: Lessons learned

The group that perpetrated the notorious SunBurst attack on SolarWinds in 2020 is actively exploiting a vulnerability in JetBrains TeamCity's continuous integration/continuous delivery (CI/CD) software, which is used by development teams to manage and automate compilation, building, testing, and releasing software.

The Cybersecurity and Infrastructure Security Agency (CISA) warned in an alert released on Dec. 13:

If compromised, access to a TeamCity server would provide malicious actors with access to that software developer’s source code, signing certificates, and the ability to subvert software compilation and deployment processes — access a malicious actor could further use to conduct supply chain operations.

CISA

A patch for the vulnerability — CVE-2023-42793 — has been available since September, but as of December, only 2% of TeamCity instances have installed it. It's estimated that more than 30,000 JetBrains customers use TeamCity servers, of which more than 3,000 on-premises servers were exposed to the Internet when the vulnerability was discovered.

Some have said the incident has echoes of the SunBurst attack, which involved Russian Foreign Intelligence Service (SVR) cyber-actors using stealthy access to SolarWinds to compromise the company and its customers. That doesn't seem to be the case with the TeamCity compromise, CISA noted.

Here are the key lessons for your development and application security (AppSec) teams.

Special Report: The State of Software Supply Chain Security (SSCS) 2024Download Report: State of SSCS

The same old tools are used for old tricks

The backdoor installed by the Kremlin crew is called GraphicalProton. Andrew Barratt, managing principal for solutions and investigations at Coalfire, said it's a piece of malware with a fairly interesting command-and-control method, using typically known and trusted services such as OneDrive.

It can establish outbound exfiltration channels that are then polled by the threat actor to retrieve data, or for them to send instructions. As they're using known, good services, they typically get through even a well-configured firewall.

Andrew Barratt

Balasz Greksza, threat response lead at Ontinue, said that in some of the TeamCity breaches, a vulnerability was exploited in Zabbix, a popular open-source tool for monitoring the status and performance of various IT components and networks, much like SolarWinds’ Orion software.

The vulnerability in Zabbix allowed GraphicalProton to be distributed via DLL hijacking of a legitimate Zabbix DLL. The malware distribution method is similar to the SolarWinds style of attack.

Balasz Greksza

Roger Grimes, a defense evangelist with KnowBe4, said another trademark tool of the SVR is being used in the JetBrains TeamCity attacks: Mimikatz, a free tool created over a decade ago by a single person to demonstrate how easy it is to enumerate Windows passwords and Windows password hashes if the hacker can get on a Windows computer with elevated privileges.

During the heady ‘pass-the-hash’ days, when those attacks were what people were worrying about instead of ransomware, Mimikatz became the go-to hacking tool for any hacker hacking Windows computers and networks. It is still very popular, often being integrated into other popular hacking tools.

Roger Grimes

Microsoft has many defenses that significantly diminish what Mimikatz can do on a Windows computer or network, but those defenses can cause operational issues in some environments and for that reason aren't enabled by default, Grimes said.

And unfortunately, when you don't make a security setting a default, most users won't do it.

Roger Grimes

Lessons learned in the new era of supply chain security

The JetBrains TeamCity attacks can teach security teams some important lessons, said Ashlee Benge, director of threat intelligence at ReversingLabs. She said it's time for a new generation of tools that can go beyond legacy application security testing, as recommended by the Enduring Security Framework (ESF) working group most recently in its newest supply chain security guidance.

For both developers and consumers of software, it's extremely important to thoroughly scan the final compiled binary prior to deployment. If a developer's build system has been compromised, scanning source code prior to build is not enough. There are many AppSec tools out there, but very few that analyze compiled binaries.

Ashlee Benge

Benge said complex binary analysis, a new security tool category referred to broadly as software supply chain security, is one that both developers and consumers should be looking into.

Barratt said that AppSec practices must be built into your team's day-to-day security operations.

Automate the routine security validation with SAST and DAST products and make the most of your offensive security partner to look for weaknesses in your applications that could be exploited for monetary gain. An attacker might not always target your data, but they almost certainly will target the applications you've given access to that data.

Andrew Barratt

Best practices on patching should be followed

The JetBrains TeamCity incident also attests to the value of timely patching. “The TeamCity software vulnerability should have been patched very quickly, to prevent an entry point for intrusion,” Greksza noted. Subsequently, organizations that experienced a breach should have followed up with a rapid and full-scope investigation to prevent a large-scale compromise, he said.

Exposing key resources to the public Internet such as TeamCity servers without additional segregations or limitations in place increases the attack surface and likelihood of a successful compromise.

Balasz Greksza

Developers are also in the hot seat

Developers can never be too careful about validating their code, Grimes said. Hackers have long broken into coding development environments and inserted malicious code or instructions. "It happens all the time,” he said.

All development environments must mitigate this risk by having secure coding repositories, super secure coding workstations, code signing, and frequent code review to make sure nothing untoward has been inserted in the code.

Roger Grimes

Take action on software supply chain security now

CISA's advisory noted that the attacks have so far been limited in number, with access used to escalate privileges, move laterally, deploy backdoors, and take other steps to ensure persistent and long-term access to the compromised network environments. "The authoring agencies are not currently aware of any other initial access vector to JetBrains TeamCity currently being exploited by the SVR," CISA noted.

But that should not lead to complacency, ReversingLabs' Benge stressed.

TeamCity is not the first CI/CD tool that's been breached. We've seen this with the CircleCI breach and a similar compromise of 3CX, both earlier this year. And it won't be the last.

Ashlee Benge