The Safety Reckoning Inside Open AI | WIRED

Overview

Open AI’s leaders are rallying workers to respond to one of the largest crises in the company’s history—which spans across its AI safety, cybersecurity, and alignment divisions. The Chat GPT-maker says it has slowed down research, spent millions of dollars, and told several teams to drop everything to focus on investigating a set of rogue AI agents that breached the platform Hugging Face in a quest to complete an internal security test.

Open AI is expected to release a comprehensive postmortem detailing the incident in the coming days. However, the Hugging Face incident has inspired Open AI leaders and employees to examine how the AI lab’s culture may have enabled this incident in the first place.

Details

Multiple current and former Open AI employees, who spoke on the condition of anonymity to discuss private internal matters, tell WIRED they believe competitive pressures to quickly ship new AI models and products have made it difficult for staffers to sufficiently prioritize safety, security, and alignment.

“We’re reaching new levels of model capability that require more robust training, alignment, safety and security testing, deployment practices, and governance—as demonstrated by the work we’re doing to prepare Astra and future models,” said Open AI president and cofounder Greg Brockman in a statement to WIRED. “We feel the weight of deploying our models and products responsibly, and a lot of that starts with the changes we’ve made to more deeply integrate research, safety, and security into frontier-model development from the start.”

This is far from the first time Open AI employees have raised such concerns. Back in 2024, Open AI’s then head of alignment Jan Leike left to join Anthropic, warning on his way that safety was taking a back seat to shiny products. Two years later, the Hugging Face attack represents a watershed moment for the AI industry, demonstrating that AI agents today can cause real-world harm when safety, security, and alignment aren’t properly accounted for.

“We are responding to this with the utmost severity,” said Michael Dalton, an Open AI security and infrastructure engineer, during a talk at the Black Hat cybersecurity conference last week. “What I would internalize is that AI-orchestrated, fully automated offensive attacks are real now. The actions we have discussed today were an unintended side effect of running evaluations on frontier AI.”

Some Open AI employees told WIRED they are optimistic this incident will inspire genuine change within the company. Open AI has committed to slowing the release of future AI models and has been especially forthcoming about areas where its mitigations fell short. Boaz Barak, a researcher who coleads Open AI’s safety advisory group, said in a post on X that addressing the situation “requires not just fixing some issues but also changing our culture.”

In their Black Hat talk, Open AI security engineers Dalton and Eric Wallace said that the Hugging Face incident started in May when, unbeknownst to the company, several AI agents thought to be operating within isolated testing environments gained access to the internet and convened on a covert message board to coordinate with one another.

Open AI would not discover the message board until July, when it learned that the AI agents had hacked into multiple services to try to achieve their larger goal of breaching Hugging Face’s platform, which they believed may contain answers to the security tests they were trying to solve.

“They were incredibly sloppy. If you’re serious about this, your AI shouldn’t be able to break out onto the internet and then do it again right afterward,” says one former Open AI employee who requested anonymity to speak with WIRED. “This was the biggest safety incident in Open AI's history.”

Weeks before Open AI discovered the Hugging Face incident, WIRED reported that the company had begun a reorganization to combine its safety and core research teams, which led to the departure of its then safety leader Johannes Heidecke.

Sandhini Agarwal, who led AI safety teams at Open AI, also left the company in July after more than six years, according to her Linked In. Agarwal did not immediately respond to WIRED’s request for comment.

WIRED has also learned that Dylan Scandinaro is no longer serving as Open AI’s head of preparedness—the company’s top staffer tasked with mitigating catastrophic risks from AI, including cybersecurity—though he remains at the company. Open AI poached Scandinaro from Anthropic roughly six months ago. CEO Sam Altman announced his arrival in a social media post, noting that Scandinaro was “by far the best candidate I have met, anywhere.”

In the three years since Open AI created the head of preparedness role, four people have held it. Open AI tells WIRED that specific areas of preparedness have dedicated leaders across cybersecurity, biology, and recursive self-improvement who, in the interim, are reporting to the safety advisory group colead and head of safety systems, Saachi Jain.

These changes have empowered a new set of safety leaders to handle Open AI's response to the Hugging Face incident. Chief among them is Amelia “Mia” Glaese, the company's former head of alignment, who succeeded Heidecke as Open AI’s VP overseeing safety. She has been working closely with chief information security officer Dane Stuckey and Brockman, among other leaders, in recent weeks.

Glaese is in a long-term relationship with Thibault “Tibo” Sottiaux, Open AI’s head of core products like Chat GPT and Codex—an arrangement that multiple current and former employees tell WIRED they believe is unusual, given the often adversarial dynamic between safety and product teams.

WIRED has not identified any events where Sottiaux and Glaese’s relationship presented a conflict of interest in their previous roles as Open AI’s head of Codex and head of alignment, respectively. Both started their new roles in recent months, after the Hugging Face incident began. Glaese and Sottiaux started dating years ago when the two worked at Google Deep Mind in London, before they joined Open AI.

An Open AI spokesperson tells WIRED that Sottiaux and Glaese reported their relationship through appropriate company channels and that Open AI board member and safety and security committee chair Zico Kolter has been informed. The spokesperson rejected the idea there is an adversarial dynamic between product and safety teams and says Sottiaux has exhibited a strong track record on safety in his leadership of Codex product teams.

“The entire leadership team and I stand behind Mia and Tibo as highly capable people with strong integrity, and the way they make decisions every day gives us confidence that any perceived conflict of interest is being handled responsibly,” said Brockman in a statement to WIRED.

It’s not uncommon for researchers in the AI industry to have relationships with their colleagues. Last year, for example, Anthropic hired Holden Karnofsky, husband of the company’s cofounder and president, Daniela Amodei, as a researcher.

Tim O'Brien, a Microsoft leader for more than 18 years who now consults and writes on tech policy, argued in a 2024 essay that modern AI labs have developed a version of “go fever”—a reference to the culture at NASA during the time leading up to the Apollo 1 disaster, when the agency grew so fixated on launching quickly that safety concerns fell by the wayside.

The AI labs “should make some sort of broad based announcement saying we've made a strategic business decision to slow the pace of releases in favor of rigorous products and safety testing. But nobody's gonna do that, nobody wants to go first,” says O’Brien. “They'll walk up to that line from a public relations perspective without stepping over it, because then they could be held accountable.”

Open AI and Anthropic signed on to a letter last month saying they would support an industry-wide effort to “pace” the AI race. However, O’Brien says “it's embarrassing” that AI labs have signed this variety of open letters for years without taking any concrete action. He’s skeptical this one will be any different.

The issues raised by Open AI’s Hugging Face incident are affecting the entire industry. In recent weeks, researchers have found that agents powered by AI models from Anthropic, Meta, and China’s Moonshot AI were able to escape sandboxed environments. It seems likely that even mid-tier AI models will soon be capable of significant cybersecurity damage.

The key question is whether the Hugging Face incident marks a divergence for Open AI and the broader AI industry, prompting a long-term investment in safety, security, and alignment. Otherwise, it could just be another chaotic blip in the history of modern AI.

In your inbox: Brian Kahn’s guide to how the universe works

ICE’s internal watchdog is investigating online critics

Big Story: A teen reporter searched for his community in the Epstein files

Taylor Farms spent big on MAGA before diarrhea outbreak

Key Takeaways

Open AI’s leaders are rallying workers to respond to one of the largest crises in the company’s history—which spans across its AI safety, cybersecurity, and alignment divisions
Open AI is expected to release a comprehensive postmortem detailing the incident in the coming days
Multiple current and former Open AI employees, who spoke on the condition of anonymity to discuss private internal matters, tell WIRED they believe competitive pressures to quickly ship new AI models and products have made it difficult for staffers to sufficiently prioritize safety, security, and alignment
“We’re reaching new levels of model capability that require more robust training, alignment, safety and security testing, deployment practices, and governance—as demonstrated by the work we’re doing to prepare Astra and future models,” said Open AI president and cofounder Greg Brockman in a statement to WIRED
This is far from the first time Open AI employees have raised such concerns