AI isn’t enough to protect social media communities from AI - Ars Technica

Overview

AI isn’t enough to protect social media communities from AI

Each month, we’ll examine the biggest controversial changes to tech products: what changed, why it’s happening, how users are affected, and how companies can do better.

Details

Sometimes you have to fight fire with fire. But when it comes to AI slop and hateful content threatening the safety and value of social media platforms, adding more fire—in this case, more AI—can make the problem worse.

At its best, social media can be a haven for people who want to share their experiences and knowledge. It gets closest to this ideal when users contribute authentic, valuable content, whether that’s a uniquely thoughtful blog post or a helpful video on how to build a PC. Relying primarily on AI tools to preserve that authenticity misses what makes social media worthwhile in the first place: the people behind it.

In April, a Slack channel for moderators of the r/Ask Historians Reddit community was usually busy. The channel, which automatically receives links to modmail messages, was flooded with alerts after dozens of comments and posts dating back 10 years were automatically removed from the subreddit.

“And there was nothing we or the experts [who posted the deleted content] could do about it,” Dr. Sarah Gilbert, one of the mods, told me.

This was particularly damaging to the subreddit because its users view the community as an archive of detailed responses that continue to educate people long after content is posted.

Reddit’s recently revamped AI moderation tools were apparently responsible for the removals, the moderators believe. After recovering the text of some posts, one of Ask Historians’ mods noticed that all the removed content linked to Rare Historical Photos, a historical image-sharing website. The mods think Reddit might have designated the website—and thus any post using its content for explanatory illustrations—as spam.

The deletion of the content erased valuable information that had taken time to aggregate (Gilbert tells me some people spend hours, “sometimes over the course of days,” researching and writing responses to questions submitted to the subreddit). Yet it’s possible that those erroneous removals, and others like them, have contributed to metrics intended to demonstrate how effective AI modding is on Reddit.

Reddit says that thanks to AI, it has “increased enforcement actions on hate and violent content by more than 200 percent” and that AI drives “faster, higher volume enforcement.” AI has “helped reduce exposure to potentially harmful content by more than 40 percent,” Reddit said this month. It also said that it uses large language models (LLMs) to catch “the highly subtle, coordinated patterns of fake behavior and artificial hype.”

But as the Ask Historians ordeal illustrates, more enforcement doesn’t necessarily mean better enforcement.

The growth of generative AI has created new obstacles for social media moderation. Gilbert noted, for instance, that large language models “have made spam detection a lot harder,” as they seek to mimic real human voices. “Over the last two to three months, we’ve been absolutely flooded by LLM-powered spambots,” she said.

Marketing agencies are creating social media content designed to get brands cited by generative AI chatbots. Marketers have long used inauthentic social media posts to boost visibility, but the rise of chatbots has opened a new front. Startup Reach LLM, for example, focuses specifically on marketing through chatbots. As part of that effort, company representatives have created and moderate subreddits on Reddit.

These challenges have led some social media companies to explore new AI-based moderation techniques. Reddit, for example, says its AI tools have “revoked nearly [2 million] fake votes daily” and that it uses LLMs “to catch the highly subtle, coordinated patterns of fake behavior and artificial hype that older systems once missed.”

But many social media platforms have become overly reliant on AI modding tools that have been quick to penalize users for innocuous content.

Recently, Discord admitted that its AI mod system wrongfully banned about 8,400 accounts in May to early July. The AI mistakenly labeled images containing square grids, such as chessboards or spreadsheets, as CSAM and subsequently issued a permanent ban to the uploaders. (Discord says all affected accounts have since been reinstated.)

The company said its AI moderation was not intended for use without human supervision. It claimed that a human employee is supposed to review AI-flagged content before Discord takes action, but a bug caused the AI to bypass the human step and ban accounts.

The supposed mishap highlights why human guardrails remain essential in content moderation. Without meaningful oversight, an AI-based modding system can make thousands of mistakes in a matter of weeks, with lasting consequences.

Since 2025, many Facebook and Instagram users have complained about mass bans they blame on AI moderation. The lack of human moderation has only fueled frustration among users who say they did not violate any rules, especially since there has been no way to speak with a Meta employee about what caused the ban or how to get an account reinstated. Meta has not said whether AI is behind the bans, but the company has increasingly relied on generative-AI-based moderation rather than humans in recent years—a shift that some people, including Meta employees, say is happening too quickly.

Tumblr is another social community where automated modding systems have failed. In March, Chenda Ngak, head of communications at Tumblr parent company Automattic, told The Verge that Tumblr’s automated systems wrongfully banned “sub-200” Tumblr accounts in one afternoon.

And in 2025, Tumblr users complained after the platform’s automatic content moderation systems inaccurately flagged content as “mature,” reducing its visibility. In both cases, users blamed AI. Tumblr never confirmed that AI caused these problems, but the company has said it uses “a mix of machine-learning classification and human moderation.”

AI moderation can save social media companies money and help remove harmful content faster. But until these systems can eliminate basic mistakes—like labeling a checkerboard picture as CSAM—they need human oversight.

“Back when there was more transparency in the system, we would routinely report hate and get an automated response that it wasn’t actually in violation of Reddit’s rules, prompting us to start an appeals process,” Ask Historians mod Gilbert said. “So it’s hard to trust the numbers because it’s hard to trust the ‘judgment’ of Reddit’s systems.”

False positives are a “huge problem” on Reddit, she said.

Typical social media AI-based moderating systems use machine learning classifiers to analyze posts and identify and flag content that breaks platform rules. But it’s difficult for a machine to understand the nuances of sarcasm, satire, and slang.

Further, some research (examples here, here, and here) suggests that marginalized groups can be disproportionately affected by AI moderation. Without human oversight, AI can end up penalizing the very communities most vulnerable to the hateful content the systems are designed to combat.

Gilbert, who is also the research director of Cornell’s Citizens and Technology Lab, says that “marginalized and vulnerable populations are among those who experience the highest rates of moderation, and that typically this is a result of ‘false-positives,’” often driven by instances of counter-speech, language reclamation, and “responses to hateful content.”

“False positives are an equity issue. They mean that groups that are already marginalized are further silenced and censored,” she added.

AI moderators can also make communities less effective at moderating themselves. On Reddit, for example, some subreddit moderators would prefer to ban users who use hateful or violent rhetoric. But if Reddit’s AI removes such content before a human moderator sees it, those moderators lose the ability to assess whether a ban is warranted.

In terms of giving human mods more control, Reddit this week announced expanding testing for Rules Hub, a suite of tools that lets human mods “choose which rules should be automatically enforced, decide what happens when a rule is triggered (send to queue, filter, or remove), preview the experience before enabling it, and review logs and insights.” Reddit expects Rules Hub to eventually replace the Automod tool, which relies primarily on exact keywords.

Mods I’ve spoken with have repeatedly blamed the generative AI boom for a spike in content that breaks community-specific or broader platform rules. That’s a serious problem for social media sites that rely on user contributions.

Companies will continue to try new methods of moderating more reliably and effectively, but reducing human input is a step backward. Low-effort AI-generated content is changing the challenges moderation teams face, but that makes stronger approaches more necessary, where machine-scale detection can be combined with human judgment and expertise.

Just as social media has no value without people, content moderation can’t succeed without human judgment at the forefront.

Advance Publications, which owns Ars Technica parent Condé Nast, is the largest shareholder in Reddit.

1.          This Atlantic hurricane season is looking like a dud, but there will be a price to pay
   

2.          Hank Green found the AI problem that You Tube labels can’t catch
   

3.          After jacking up prices, Disney+ and Netflix consider offering free alternatives
   

4.          Reddit signals ominous upcoming "changes” for old.reddit.com
   

5.          Google plans to kill Assistant on your phone on September 4
   

Ars Technica has been separating the signal from the noise for over 25 years. With our unique combination of technical savvy and wide-ranging interest in the technological arts and sciences, Ars is the trusted source in a sea of information. After all, you don’t need to know everything, only what’s important.

Key Takeaways

• AI isn’t enough to protect social media communities from AI

• Each month, we’ll examine the biggest controversial changes to tech products: what changed, why it’s happening, how users are affected, and how companies can do better

• Sometimes you have to fight fire with fire

• At its best, social media can be a haven for people who want to share their experiences and knowledge

• In April, a Slack channel for moderators of the r/Ask Historians Reddit community was usually busy