Why AI coding makes zero trust an AppSec requirement

Feeling comfortable about the safety of your software supply chain because your organization has invested in SBOMs, signing, and provenance? You shouldn’t.

In the AI coding era, those tenets of the traditional trust model aren’t enough to ensure software safety. That’s because that model fails to answer an important security question: What is the code capable of doing?

CodeHunter CEO Ken Ammon, writing for Help Net Security, summarized the trust vacuum.

“[T]raditional trust models are insufficient when AI generates, modifies and deploys code faster than human review can keep up. We can no longer trust software based primarily on what it is, where it came from or whether it resembles something seen before.”
—Ken Ammon

Brett Smith, a software developer at SAS, said trusted sources, signatures, SBOMs, and provenance prove identity and origin, but they don’t prove intent or safety.

“A binary can be signed by a trusted vendor and have a perfect provenance chain and yet still be malicious if the build pipeline was compromised or contains a zero-day vulnerability.”
—Brett Smith

Here’s why zero trust is now essential to application security (AppSec).

[ Join webinar: Independently Verified SBOMs: Why Trust Is No Longer Enough ]

It’s now trickier to spot malicious code

SBOMs, signing, and provenance answer important questions, but AI coding brings new questions that they don’t address, said Jeff Williams, CTO and co-founder of Contrast Security and founder of OWASP.

“An SBOM tells you what components are present. A signature establishes who signed an artifact and whether it was modified. Provenance tells you where it came from and how it was built. None of that tells you whether the developer made a mistake, whether a trusted contributor or build environment was compromised, or whether the software will behave dangerously in your environment.”
—Jeff Williams

We need to recognize that dangerous code can have a perfectly documented chain of custody. “This becomes even more obvious with agentic systems,” Williams said. “The same legitimate model, tools, and components can produce safe or dangerous behavior depending on the goal, context, data, and sequence of decisions made at runtime.”

“Authenticity is useful evidence, but it isn’t proof of safety.”
—Jeff Williams

You need to know more than just what’s inside the software; you need to understand what the software does or has the potential to do, said Denis Calderone, a principal and CTO of Suzu Labs. “The Axios attacker never modified a single line of library code. The SBOM was clean. The package was signed. The provenance was legitimate. And it still dropped a [remote-access Trojan] on every machine that pulled it,” he said.

“An SBOM is an ingredient list, not a safety test.”
—Denis Calderone

Jasmine Noel, Senior Product Marketing Manager at ReversingLabs, said applications and their dependencies often start multiple background processes to function properly. 

“The problem with that is those same background processes can mask hidden capabilities designed by malicious actors to activate only under specific circumstances,  increasing the difficulty for routine code reviews — even with AI assistance — to determine when software is doing something it is not supposed to do.”
—Jasmine Noel

Another challenge is that software (along with its hundreds of dependencies) will exhibit thousands of behaviors. So accurate behavior identification is only the first step. Intelligence on malicious behaviors, behavior prevalence, toxic combinations of changes, and tracking behavior changes across versions are also crucial solution capabilities, Noel said. 

Apply zero trust to code execution

CodeHunter’s Ammon wrote in his article that behavioral verification has to be added to software supply chain security controls. “A stronger model evaluates behavior before execution wherever possible, especially for high-risk artifacts such as third-party packages, installers, scripts, containers, CI/CD outputs and AI-generated code,” he wrote.

We have learned not to place implicit trust in networks, devices, and identities, Ammon said, and that same zero-trust approach now applies to software execution. “No artifact should be trusted solely because of its origin, signature, or reputation,” he said.

“Trust is not inherited from supply chain evidence alone. Those controls remain foundational, but they do not answer the final execution question.”
—Ken Ammon

Zero trust forces the authorization question, Suzu Labs’ Calderone said. “Right now, if code comes from a trusted source and passes a signature check, it runs,” he said. “Zero trust for code means that’s not enough.”

“You evaluate what the code actually does — does it escalate privileges, does it make unexpected network calls, does it touch credentials? — and you make it then allow it to make decisions based on behavior, not just origin.”
—Denis Calderone

SAS’s Smith said that applying zero trust to code execution makes software safer by shifting the security focus from “who the software is” to “what the software does.”

“By replacing ‘trust by identity’ with ‘verification by behavior,’ we can better ensure that no process is granted implicit trust. Enforcing a least-privilege model on what a process can do helps block malicious actions regardless of where the software came from.”
—Brett Smith

When you stop trusting signatures and provenance, you eliminate the VIP pass that attackers exploit in supply chain attacks, Smith added.

He recommends that CISOs stop treating digital signatures as safety guarantees and move toward a binary authorization policy. The strategic goal is to shift KPIs from “How fast can we detect a breach?” to “How many unauthorized execution attempts did we prevent?” he said.

Avoid creating frustration via friction

The idea of applying zero trust to code is interesting but not new, said Brian Jack, CISO of KnowBe4.

“Putting friction in place and requesting the user approve what’s happening has been around a long time and, if done well, can be a great security control enhancement.”
—Brian Jack

However, Jack noted, if done poorly, it will frustrate users with too many prompts for approval that could put the user into a default click-approve mode. “The other concern is [that] by offloading some of the responsibility to the user, how can you be sure that every user is operating off of the same baseline knowledge of what the software should and shouldn’t do?” he said.

Contrast Security’s Williams said zero trust for code is a sound principle, but it has had mixed results in the versions of it that have been tried over the years: the Java sandbox, SELinux, compartmented mode workstations, and others.

“They’ve struggled because the policies are cumbersome, difficult to maintain, and prone to breaking legitimate software. Developers need to do inherently dangerous things — access private data, modify databases, create files, launch processes, and open network connections. When security controls prevent legitimate work, they’re usually weakened or disabled.”
—Jeff Williams

A noted exception to that rule is mobile, where Apple and Google control the operating systems, app stores, and distribution model and are able to make permissions mandatory. “That kind of leverage doesn’t exist across software generally,” Williams said.

Are guardrails the answer?

Agentic AI could make traditional permission models nearly intractable, Williams added. 

“Everyone is talking about guardrails, but an agent may legitimately need to read data, write code, call APIs, send messages, execute commands, and take actions across many systems. When the potential range of behavior is almost everything, expressing everything that should be allowed and disallowed in advance becomes impossible.”
—Jeff Williams

Williams prescribed a layered strategy: 

• Use SBOMs, signatures, provenance, and pre-execution analysis as evidence.
• Use least privilege and isolation to reduce exposure.
• Use runtime visibility and enforcement to control what the software actually does.

ReversingLabs’ Noel said one of the big misconceptions folks have is that the code has to be running for behavioral analysis to happen, making it a slow and arduous process.  

“Analyzing a software binary can deliver a comprehensive SBOM, evaluate the behaviors of each embedded artifact (even images and files that are not supposed to have behaviors), and perform threat detection in a single pass that is much faster and more rigorous than traditional dynamic analysis.”
—Jasmine Noel

Gartner noted the blind spots wth traditional AppSec in a recent report, "The Gartner CISO Playbook for Commercial Software Security." Alternatively, modern startegies should be based on software supply chain security tooling that evaluates “both the software components and the security of the development practices followed to create it,” the report notes.

To carry this out, Gartner identifies required security controls and processes for effectively managing commercial software risk:

• Perform binary composition analysis to identify risks including malicious code, weak cryptography, and embedded secrets
• Analyze software component risk levels using software bills of materials (SBOMs)
• Assess secure software development practices against industry standards
• Provide self-service SBOM generation outside of formal risk assessments.

The above security measures should provide the risk scoring CISOs need “to inform procurement decisions and determine appropriate implementation controls,” Gartner noted in its report.