'They want to build a moat': Anthropic's scary warnings about rapid AI 'self-improvement' and 'temporarily' pausing development aren’t convincing the cynics | Tech Radar

Overview

News, deals, reviews, guides and more on the newest computing gadgets

Start exploring exclusive deals, expert advice and more

Details

Unlock and manage exclusive Techradar member rewards.

Unlock instant access to exclusive member features.

Get full access to premium articles, exclusive features and a growing list of member rewards.

'They want to build a moat': Anthropic's scary warnings about rapid AI 'self-improvement' and 'temporarily' pausing development aren’t convincing the cynics

When you purchase through links on our site, we may earn an affiliate commission. Here’s how it works.

Anthropic wants to put a pause on AI development (Image credit: Getty Images/SOPA Images)

AI could soon start developing itself, the company warns

Even the giant companies at the forefront of AI are suggesting it might be time to take a step back and think about where the tech is heading: Claude developer Anthropic has posted a lengthy new blog post in which it suggests "a meaningful slowdown or pause" while we all take a breath.

Authored by Anthropic executives Marina Favaro and Jack Clark, the post centers on the idea of AI developing itself, known as 'recursive self-improvement' — at which point it might get out of our hands very quickly, as AI takes over the business of designing and developing its own models and interfaces.

"We are not there yet, and recursive self-improvement is not inevitable," explains the post. "But it could come sooner than most institutions are prepared for." By next year, an AI model like Claude could be capable of completing tasks that would take a human coder weeks, according to Anthropic's calculations.

As per Anthropic's internal testing, Claude is managing more code than ever before, and is rapidly getting better at writing code that both works and can be understood by human engineers. In fact, Claude is now catching bugs that the best programmers at Anthropic previously missed.

While humans still do better at seeing the big picture and context outside the current task, the blog post says, this is something else that AI may soon catch up with — though this part is less clear. If and when that does happen, AI could escape our control, and that's where the proposals about a freeze on AI development come in.

Anthropic drops shocking warning about near-future AI that can program and improve itself

Quote of the day by Anthropic CEO Dario Amodei: "Humanity is about to be handed almost unimaginable power, and it is deeply unclear whether [we] possess the maturity to wield it" — warnings on the looming threat of beyond-human AI

Anthropic calls for global freeze in AI development from r/technology

"If it were possible to effectively slow the development of this technology to give ourselves more time to deal with its immense implications, we think that would likely be a good thing," writes the Anthropic team. "But if a slowdown simply lets the least cautious actors catch up technologically, it could leave everyone less safe."

With that in mind, the blog post floats the idea of a "global coordination mechanism" to slow down or pause frontier AI development — which would of course require competitors like Open AI and Google to promise not to try and jump ahead in secret. During the pause, more of the future potential of AI and possible safeguards could be worked out.

Anthropic says it will "organize conversations" with governments, researchers, and AI companies to see if it's going to be possible to put the brakes on development at this point. If we don't do it now, we might lose the opportunity altogether, which is why Anthropic wants to ask these questions now.

Out in the flesh and blood world, a noticeable anti-AI sentiment is growing — at least outside of Silicon Valley and coders. Online reactions to Anthropic's post have accused the company of wanting to protect its own lead in the market and raise hype for its upcoming IPO — though many share the same safety concerns raised in the blog post.

"It's regulatory capture," says one commenter, pointing out that this is something AI companies regularly do. "They want to build a moat around their business." Another Reddit post frames the scenario in a more vivid way: "Wolf with a bloody maw and engorged belly says it's time to stop eating meat for a little bit."

Follow Tech Radar on Google News and add us as a preferred source to get our expert news, reviews, and opinion in your feeds.

➡️ Read our full guide to the best laptops

Best overall: Apple Mac Book Air 13-inch M5
Best budget: Apple Mac Book Neo
Best Windows 11 laptop Microsoft Surface Laptop 13-inch
Best thin and light: Lenovo Yoga Slim 9i
Best Ultrabook Asus Zenbook S 16

Dave is a freelance tech journalist who has been writing about gadgets, apps and the web for more than two decades. Based out of Stockport, England, on Tech Radar you'll find him covering news, features and reviews, particularly for phones, tablets and wearables. Working to ensure our breaking news coverage is the best in the business over weekends, David also has bylines at Gizmodo, T3, Pop Sci and a few other places besides, as well as being many years editing the likes of PC Explorer and The Hardware Handbook.

You must confirm your public display name before commenting

2 These are the best HP all-in-one AI-enabled inkjet printer deals I've found

3 Exclusive: Fairphone CEO teases ‘two new products’ that will ‘really push the envelope’ for sustainable tech

4 Valve's Steam Machine is coming this summer, but its price is still unknown

5A year on, my Switch 2 has more than proved its worth — but Nintendo needs to act fast if it wants to break its generational curse

Tech Radar is part of Future US Inc, an international media group and leading digital publisher. Visit our corporate site.

Key Takeaways

News, deals, reviews, guides and more on the newest computing gadgets
Start exploring exclusive deals, expert advice and more
Unlock and manage exclusive Techradar member rewards
Unlock instant access to exclusive member features
Get full access to premium articles, exclusive features and a growing list of member rewards