Speed kills: AI coding tools revive old-school hacks

Development teams relying on AI-powered tools to write — or merely to vet — code may be opening the door to crippling cyberattacks, researchers at Kudelski Security revealed.  

Speaking on Thursday at the annual Black Hat Briefings, Nathan Hamiel, Kudelski’s senior director of research, and Nils Amiet, the company’s lead prototyping engineer, detailed their discovery of serious security flaws in AI coding tools that give malicious actors access to sensitive IT infrastructure, development secrets, and code repositories.

The adoption of AI coding tools has exploded in recent years with their promise of speeding development and automating time-consuming manual tasks such as code audits. But the tools bypass a wide range of accepted security standards — and introduce serious risks for development teams, the researchers warned. 

Here's what you need to know about the research — and what you can take from it to reduce software risk introduced by AI coding tools.

Get Report: How AI Impacts Supply Chain Security

AI coding: Speed first, security last

Hamiel said in the pair’s talk, titled “Hack to the Future: Owning AI‑Powered Tools with Old‑School Vulns,” that development speed has taken the driver's seat and put security in the back.

Hamiel and Amiet detailed their discovery of a wide range of gaping security vulnerabilities in AI-powered tools, many of which boast features that ignore long-established security best practices — or even turn them on their heads. 

Hamiel said many of the AI development tools embraced by software teams allow the “blind execution of input data” in which users feed data to an AI engine in a prompt that invokes a query and then instruct the AI engine to execute the query using the submitted data.

It’s speed above everything else.

Nathan Hamiel

That sequence is also known as remote-code execution (RCE), and is considered a five-alarm fire in cybersecurity circles. But, wrapped in the optimism of AI-powered automation, “RCE as a service” has evoked few objections from developers or software producers, Hamiel said. 

How AI coding tools introduce risk

To demonstrate the security issues created by AI-powered developer tools, Amiet described what he found out by researching CodeRabbit, an AI-powered code-review application that was launched in 2023 and is one of the most downloaded applications on the GitHub Marketplace. Developers have used it to review the content of more than 1 million GitHub code repositories. 

Amiet identified glaring security issues with CodeRabbit. For one, when installed, the tool prompts the developer to identify which code repositories it is allowed to scan, which gives CodeRabbit write access in order to run static analyzers on the code for languages including Python, JavaScript, Java, C++, Ruby, and C. 

Amiet described how he was able to use CodeRabbit to facilitate an RCE attack. By depositing a “dummy” Ruby (.rb) code file containing a malicious pull request, the researchers leveraged CodeRabbit’s automation of static analysis. The detected Ruby code file was sent to a static analyzer known as Rubocop. By modifying the Rubocop configuration file and instructing it to load a malicious Ruby extension, Amiet was able to use CodeRabbit’s access to code repositories to export any detected environmental variables it found to an external web host. 

Those variables included developer secrets such as Anthropic and OpenAI API keys, Gitlab personal access tokens, Jira secrets and PostgreSQL database host, user name and password information, among others. Included in the returned data was the GitHub application private key for CodeRabbit itself, which effectively gave the researchers write access to any of the more than 1 million GitHub repositories that CodeRabbit is used to audit. 

Since being informed of the flaw, CodeRabbit has addressed it. However, CodeRabbit is not the only problem; Hamiel and Amiet demonstrated issues in several other AI-powered tools. For example, they were able to manipulate a feature of the Qodo Merge Pro tool to get access to the tool’s AWS admin key.

It was pretty obvious that we could have basically done anything we wanted.

Nathan Hamiel

The message for development teams is clear: Use AI coding tools cautiously. Tools such as CodeRabbit have real advantages for developers, but those advantages may be outweighed by an increase in cyber risk. “Anything AI can do, your attacker can do,” Hamiel said. And while security experts have long advocated for developers to embrace the idea of “least privilege,” the AI default is often “most privilege”  -granting AI as much privilege as possible.

Code unknowns: The new normal for AI

The risks from AI also include the rise in unknowns, given that developers do not create the code and cannot say, exactly, what the AI generated code executes at run time. There is also generative AI’s habit of blending sensitive and nonsensitive data and mixing internal and external data. 

Unknowns are the new normal. Developers have outsourced the creation of functional code, which means they’re not sure what code is actually executing at run time.

Nathan Hamiel

Hamiel and Amiet said development teams too often make the mistake of thinking of generative AI as intelligent systems but AI coding tools are more like “lazy, intoxicated robots” designed to do what they’re told in the fastest, easiest way possible. And rather than transforming application security (AppSec) in a consistently positive way, gen AI and AI coding tools simply reinforce existing security practices — whether good or bad, Hamiel said.

If you don’t have mature app and product security in place, start prioritizing those efforts today. You have to understand where the data is going and why.

Nathan Hamiel

New NIST guidance identifies key AI and ML challenges. Learn why ReversingLabs Spectra Assure should be an essential part of your solution.