'Today, frontier AI systems are simply not reliable enough to power fully autonomous weapons' — Anthropic CEO on why it won't agree to Pete Hegseth's scary request | Tech Radar

Overview

'Today, frontier AI systems are simply not reliable enough to power fully autonomous weapons' — Anthropic CEO on why it won't agree to Pete Hegseth's scary request

When you purchase through links on our site, we may earn an affiliate commission. Here’s how it works.

Details

"Lethal autonomous weapons threaten to become the third revolution in warfare. Once developed, they will permit armed conflict to be fought at a scale greater than ever, and at timescales faster than humans can comprehend."

That's not a quote from Anthropic CEO Dario Amodei refusing to accede to the US Department of War's request that it allow its Claude AI models for mass surveillance and perhaps more problematically "Fully autonomous weapons." Instead, it comes from a 2017 Open Letter to the UN, co-signed by, among dozens of other AI and robotics leaders, Elon Musk, asking the global organization to ban autonomous weapons.

It's a window into long-brewing concerns over the abuse and misuse of autonomous systems for warfare. It's also likely, despite Musk's closeness to the current Trump administration, that US Secretary of Defense (or War) Pete Hegseth has never read it.

‘We cannot in good conscience accede to their request’: Anthropic CEO Dario Amodei draws a line in the sand in standoff with US government

Pentagon may sever Anthropic relationship over AI safeguards - Claude maker expresses concerns over 'hard limits around fully autonomous weapons and mass domestic surveillance'

Anthropic CEO goes nuclear on Nvidia at Davos over China chip sales

Anthropic is now at risk of losing a $200M US Department of War contract, despite, as Amodei describes it, already working "proactively to deploy our models to the Department of War and the intelligence community."

Amodei is by no means anti-defense or against the use of AI by the US government. In his letter explaining Anthropic's decision, Amodei writes, "I believe deeply in the existential importance of using AI to defend the United States and other democracies, and to defeat our autocratic adversaries."

However, what Hegseth has asked is for Anthropic to countermand its own "Constitution", a set of principles and safety restrictions for the use and behavior of its AI models. The US Department of War basically wants Anthropic to remove the guardrails. Anthropic Constitution Principles, such as being "Broadly Safe" and "Broadly Ethical," are in direct conflict with Hegseth's demands that the AI be used for mass surveillance and for fully autonomous weapons.

Amodie makes it clear that his systems are not ready for any of this.

"Today, frontier AI systems are simply not reliable enough to power fully autonomous weapons," writes Amodei, adding, "Without proper oversight, fully autonomous weapons cannot be relied upon to exercise the critical judgment that our highly trained, professional troops exhibit every day."

These are not new concepts. Many in the tech industry have been pondering these issues for almost a decade (if not longer). Musk and the AI and robotics community raised the alarm in 2017 because we were already seeing AI-backed robot systems being used in questionable ways.

In 2016, a bomb disposal robot was used to kill a mass shooting suspect in Texas. Dallas PD put an explosive device on the robot's arm, guided it to where the suspect was holed up, and then they detonated the explosive device and killed the suspect.

‘We cannot in good conscience accede to their request’: Anthropic CEO Dario Amodei draws a line in the sand in standoff with US government

Pentagon may sever Anthropic relationship over AI safeguards - Claude maker expresses concerns over 'hard limits around fully autonomous weapons and mass domestic surveillance'

Anthropic CEO goes nuclear on Nvidia at Davos over China chip sales

At the time, some saw it as an inflection point, and a concerning one at that. Episodes like that may or may not have triggered that 2017 letter to the UN.

Once this Pandora's box is opened, it will be hard to close. AI and Robotics companies in 2017

Once this Pandora's box is opened, it will be hard to close.

Keep in mind that this happened before the current generative and agentic AI revolution.

Amodei knows better than most the massive leaps foundational models are taking every few months and, as he makes clear in his letter, our rules and strategies for managing AI in these circumstances have already fallen behind their capabilities.

"AI-driven mass surveillance presents serious, novel risks to our fundamental liberties. To the extent that such surveillance is currently legal, this is only because the law has not yet caught up with the rapidly growing capabilities of AI," he wrote.

Essentially, with AI, we don't know what we don't know. Hegseth's willingness to recklessly use powerful AI models in both surveillance and warfare indicates he has zero knowledge or interest in the past and even less understanding of the intricacies of these systems.

I've yet to talk to a technologist, a roboticist, or someone within the AI community who thinks letting an AI (or an AI-powered robot) control or carry a weapon is a good idea.

Hegseth isn't necessarily spelling out that scenario, but his requirement to remove the guardrails Anthropic has smartly put in place indicates to me that he doesn't really care about repercussions and AI casualties. He's focused on results, perhaps at any or all costs, including safety and liberty.

Amodei's done the right thing here, basically calling Hegseth's bluff. As the Anthropic CEO made clear, Claude AI is already being used in many Department of War systems. Pulling it out and retrofitting for another, perhaps less powerful and intelligent set of models might not be easy and probably won't have the desired outcome of a system ready to carry out Hegseth's bidding.

Clearer heads must prevail here. As the tech leaders and, yes, even Elon Musk, wrote in 2017, "Once this Pandora's box is opened, it will be hard to close."

Follow Tech Radar on Google News and add us as a preferred source to get our expert news, reviews, and opinion in your feeds. Make sure to click the Follow button!

And of course, you can also follow Tech Radar on You Tube and Tik Tok for news, reviews, unboxings in video form, and get regular updates from us on Whats App too.

A 38-year industry veteran and award-winning journalist, Lance has covered technology since PCs were the size of suitcases and “on line” meant “waiting.” He’s a former Lifewire Editor-in-Chief, Mashable Editor-in-Chief, and, before that, Editor in Chief of PCMag.com and Senior Vice President of Content for Ziff Davis, Inc. He also wrote a popular, weekly tech column for Medium called The Upgrade.

Lance Ulanoff makes frequent appearances on national, international, and local news programs including Live with Kelly and Mark, the Today Show, Good Morning America, CNBC, CNN, and the BBC.

You must confirm your public display name before commenting

1'Today, frontier AI systems are simply not reliable enough to power fully autonomous weapons' — Anthropic CEO on why it won't agree to Pete Hegseth's scary request

2'We see an opportunity to move faster with smaller, highly talented teams using AI to automate more work': Block decimates workforce in favor of AI

3 No data shared — Private Internet Access reinforces its no-log claims with independent audit and transparency report

4I've been using these super-cheap headphones for a month — I'm surprised to report they sound great

5 Samsung's newest Galaxy Buds 4 Pro are available for preorder — here's why I'm buying them at Amazon

Tech Radar is part of Future US Inc, an international media group and leading digital publisher. Visit our corporate site.

© Future US, Inc. Full 7th Floor, 130 West 42nd Street, New York, NY 10036.

Key Takeaways

• 'Today, frontier AI systems are simply not reliable enough to power fully autonomous weapons' — Anthropic CEO on why it won't agree to Pete Hegseth's scary request

• When you purchase through links on our site, we may earn an affiliate commission

• "Lethal autonomous weapons threaten to become the third revolution in warfare

• That's not a quote from Anthropic CEO Dario Amodei refusing to accede to the US Department of War's request that it allow its Claude AI models for mass surveillance and perhaps more problematically "Fully autonomous weapons

• It's a window into long-brewing concerns over the abuse and misuse of autonomous systems for warfare