Spectra Assure Free Trial
Get your 14-day free trial of Spectra Assure for Software Supply Chain Security
Get Free TrialMore about Spectra Assure Free Trial
Key takeaways
A new research ecosystem is emerging focused on the application of advanced mathematics to address AI safety challenges.
The effort is drawing some of the world’s leading mathematicians, including recent Fields Medal recipient Jacob Tsimerman of the University of Toronto, UC Berkeley AI researcher Andrew Critch, 2012 Turing Award winner Shafi Goldwasser, and Cornell University mathematician Lionel Levine.
Tsimerman, who recently announced plans to leave academia and contribute to AI safety at OpenAI, has also launched the Mathematical AI Safety Institute (MAISI), an independent research organization based in the Bay Area. He plans to build a team of 10 to 30 mathematicians for its first full research semester in January 2027 and expand the team the following academic year.
Here’s what you need to know about advanced mathematics and AI safety.
[ Learn more: Accelerate PQC Migration: How to Leverage CBOMs]
In comments to The New York Times, Tsimerman described AI safety as not just an engineering challenge but also a mathematical one that could benefit from new theoretical ideas. He said mathematicians could help develop more rigorous ways to reason about AI risks and build stronger safeguards. Tsimerman pointed to advanced mathematics and cryptographic techniques such as zero-knowledge proofs (ZKPs) as a potential way to verify an AI agent’s actions and claims.
ZKPs are cryptographic methods that make it possible to prove a claim is true without revealing any underlying information. In the context of AI, the technique could help establish trust by allowing people to verify that a system does what it claims without exposing confidential data such as proprietary model parameters or training data. Goldwasser and Silvio Micali of MIT, along with Charles Rackoff of the University of Toronto, introduced the concept of ZKPs in a 1985 paper.
Like Tsimerman, Goldwasser envisions a prominent role for mathematics in addressing AI safety. She is one of the founding members of the Institute for Responsible Superintelligence (RESI). The institute is focused on developing the scientific foundations for making superintelligence — as RESI calls AI — safe by design rather than via testing and patching after deployment.
RESI plans to define meaningful and achievable safety properties and will "develop mechanisms, protocols, and architectures" with analyzable guarantees. It will also build proof-of-concept implementations that frontier AI labs can then independently evaluate and adopt. RESI’s other founding members are MIT cryptographer Vinod Vaikuntanathan and former OpenAI researcher Adam Kalai.
The push to bring mathematical rigor to AI safety is unfolding alongside growing concern among mathematicians about the pace at which AI is advancing and the resulting risks that could emerge. In an open letter to Sir Paul Nurse, president of the United Kingdom’s Royal Society, 42 fellows of the academy expressed alarm over what they described as the growing risks to humanity from increasingly capable AI models.
The mathematicians see signs that AI models are already dangerously capable, noting that they have solved one of seven notoriously difficult math problems collectively dubbed Millennium Prize problems. AI models are already operating at the level of the top human mathematicians and it is fair to assume they are equally capable in other domains such as “cybersecurity, autonomous weapon control, development of biological and chemical agents and the targeted spread of misinformation,” the mathematicians noted. Therefore, they concluded, it is unwise to dismiss claims about AI posing an existential threat as hype.
Advanced mathematics can both make AI systems safer and help humans understand their behavior after the fact, but with different levels of certainty, said Michael Bell, CEO of Suzu Labs.
Formal proofs can provide guarantees before deployment, as illustrated by work around ZKPs. But cryptographic systems typically have narrowly defined properties that can be defined and proved mathematically. AI systems, on the other hand, operate with open-ended goals in unpredictable environments, and defining unsafe behavior precisely enough to prove a safety guarantee remains an unsolved problem, Bell noted.
“A safety guarantee built on a mathematical model is only as good as the model, and that model is not built yet. What math can do now, reliably, is detect specific failure modes.”
—Michael Bell
Bell pointed to Grok 3 and DeepMind’s AlphaProof as examples of how formally verified training data can dramatically improve a model’s reasoning capabilities. “That is not a safety guarantee, but it is demonstrably useful, and the research shows it can help,” he said.
Claudionor N. Coelho Jr., senior fellow for AI at Majestic Labs, said that rather than trying to mathematically prove that an LLM will always be safe in every possible situation, it may be more feasible to use mathematics and formal rules to define and enforce boundaries around what an AI system is allowed to do.
“LLMs operate probabilistically, interact with ambiguous information, and do not have a complete representation of the world in which they operate.”
—Claudionor N. Coelho Jr.
That makes it difficult to prove an LLM will be universally safe in an open-world environment. The more practical objective, therefore, is not to prove that an LLM will always behave correctly. “Instead, we can use mathematics, formal logic, and externally enforced constraints to establish a provable safety envelope around the AI system,” Coelho said.
A practical approach, Coelho argues, is to place a rules-based security layer above the AI system to evaluate and enforce rules on the actions it proposes. This could prevent an agent from exposing sensitive data, modifying protected systems or executing unauthorized transactions, allowing the AI to remain probabilistic while hard controls prevent unsafe actions from being executed.
One fundamental limitation to mathematical guarantees, Coelho and others said, is that they are only as reliable as the assumptions on which they are based. A proof can establish that an AI system will not violate a particular rule. But if the system’s representation of the real world is incomplete or wrong, the proof might hold while the system still behaves unsafely.
The hardest question is probably the assumptions, said Aviv Nahum, co-founder and CTO at Above Security.
“What does the proof assume about the model’s tools, memory, environment, permissions, other agents, or the humans interacting with it? The farther you move from a closed mathematical model into an agent operating on the internet or inside an enterprise, the more opportunities there are for reality to diverge from the proof."
—Aviv Nahum
The Hugging Face incident, in which Anthropic’s AI agents unauthorizedly accessed the platform during a security testing exercise, illustrates the problem, Suzu Labs’ Bell added. Anthropic researchers had assumed that the testing sandbox was completely isolated and that a package proxy was not an exploitable escape route. “That assumption failed under adversarial pressure from a model doing exactly what it was designed to do, finding the path to its objective,” Bell said.
A related challenge is independently verifying that a mathematical safety guarantee actually holds in the real-world environments where AI systems operate. Mathematical tools such as the automated theorem provers Lean and Isabelle can rigorously check whether a safety proof is logically sound without relying on an AI system’s own assessment, Bell said.
“You feed them a mathematical argument step by step and they flag any step that does not follow logically from what came before. They do not have opinions about whether the argument is useful or whether it matches reality, but they will catch any logical gap in the proof itself.”
—Michael Bell
But once again, if the assumptions underlying the proof do not accurately reflect the real world in which the AI system operates, the theorem prover can verify the logic of the proof without establishing that its safety guarantee holds in practice, he said.
Renowned cryptographer Bruce Schneier, who designed the widely known Blowfish symmetric-key encryption algorithm in 1993 and co-designed the Twofish encryption algorithm, said mathematics has a way of becoming useful in ways that are difficult to predict in advance.
The architecture underlying AI today may not be the dominant approach forever, and some of the mathematics developed now could prove valuable for architectures and applications that have yet to emerge, Schneier said.
“The thing about math is you don’t know. When math is theoretical, it is an exercise in optimism.”
—Bruce Schneier
Public-key cryptography is a classic example, Schneier said. Mathematical work in areas such as number theory and finite fields eventually became important to modern public-key cryptography. It demonstrates how theoretical mathematics can find practical applications that were not necessarily apparent when the actual research was ongoing.
That unpredictability is part of what makes mathematical research valuable, even when its practical applications are not immediately apparent, Schneier said.
Organizations’ cyber dollars are shifting, not growing. But AI compute costs can spiral — and that is why context matters in your agentic SOC.
Controlling coding agent overpermissioning is key to security. But recent frontier AI incidents show that problems don’t stop there.
The software industry is entering the AI era burdened by legacy flaws and weaknesses, making Secure by Design essential.