AI isn’t enough to protect social media communities from AI
Ars Technica · LC · trust 48/100
Consumer Checkpoint AI isn’t enough to protect social media communities from AI Why humans need to moderate humans.
38 Credit: Getty Credit: Getty Text settings Story text Size Small Standard Large Width * Standard Wide Links Standard Orange * Subscribers only Learn more Minimize to nav Consumer Checkpoint The tech products we rely on are constantly changing. Sometimes those changes bring improvements; other times, they frustrate users or make beloved products worse. Each month, we’ll examine the biggest controversial changes to tech products: what changed, why it’s happening, how users are affected, and how companies can do better.
Sometimes you have to fight fire with fire. But when it comes to AI slop and hateful content threatening the safety and value of social media platforms, adding more fire—in this case, more AI—can make the problem worse.
At its best, social media can be a haven for people who want to share their experiences and knowledge. It gets closest to this ideal when users contribute authentic, valuable content, whether that’s a uniquely thoughtful blog post or a helpful video on how to build a PC. Relying primarily on AI tools to preserve that authenticity misses what makes social media worthwhile in the first place: the people behind it.
In April, a Slack channel for moderators of the r/AskHistorians Reddit community was usually busy. The channel, which automatically receives links to modmail messages, was flooded with alerts after dozens of comments and posts dating back 10 years were automatically removed from the subreddit.
“And there was nothing we or the experts [who posted the deleted content] could do about it,” Dr. Sarah Gilbert, one of the mods, told me.
This was particularly damaging to the subreddit because its users view the community as an archive of detailed responses that continue to educate people long after content is posted.
Reddit’s recently revamped AI moderation tools were apparently responsible for the removals, the moderators believe. After recovering the text of some posts, one of AskHistorians’ mods noticed that all the removed content linked to Rare Historical Photos, a historical image-sharing website. The mods think Reddit might have designated the website—and thus any post using its content for explanatory illustrations—as spam.
Reddit has not responded to a request for comment.
The deletion of the content erased valuable information that had taken time to aggregate (Gilbert tells me some people spend hours, “sometimes over the course of days,” researching and writing responses to questions submitted to the subreddit). Yet it’s possible that those erroneous removals, and others like them, have contributed to metrics intended to demonstrate how effective AI modding is on Reddit.
Reddit says that thanks to AI, it has “increased enforcement actions on hate and violent content by more than 200 percent” and that AI drives “faster, higher volume enforcement.” AI has “helped reduce exposure to potentially harmful content by more than 40 percent,” Reddit said this month. It also said that it uses large language models (LLMs) to catch “the highly subtle, coordinated patterns of fake behavior and artificial hype.”
But as the AskHistorians ordeal illustrates, more enforcement doesn’t necessarily mean better enforcement.
The growth of generative AI has created new obstacles for social media moderation. Gilbert noted, for instance, that large language models “have made spam detection a lot harder,” as they seek to mimic real human voices. “Over the last two to three months, we’ve been absolutely flooded by LLM-powered spambots,” she said.
Marketing agencies are creating social media content designed to get brands cited by generative AI chatbots. Marketers have long used inauthentic social media posts to boost visibility, but the rise of chatbots has opened a new front. Startup ReachLLM , for example, focuses specifically on marketing through chatbots. As part of that effort, company representatives have created and moderate subreddits on Reddit.
These challenges have led some social media companies to explore new AI-based moderation techniques. Reddit, for example, says its AI tools have “revoked nearly [2 million] fake votes daily” and that it uses LLMs “to catch the highly subtle, coordinated patterns of fake behavior and artificial hype that older systems once missed.”
But many social media platforms have become overly reliant on AI modding tools that have been quick to penalize users for innocuous content.
Recently, Discord admitted that its AI mod system wrongfully banned about 8,400 accounts in May to early July. The AI mistakenly labeled images containing square grids, such as chessboards or spreadsheets, as CSAM and subsequently issued a permanent ban to the uploaders. (Discord says all affected accounts have since been reinstated.)
The company said its AI moderation was not intended for use without human supervision. It claimed that a human employee is supposed to review AI-flagged content before Discord takes action, but a bug caused the AI to bypass the human step and ban accounts.
The supposed mishap highlights why human guardrails remain essential in content moderation. Without meaningful oversight, an AI-based modding system can make thousands of mistakes in a matter of weeks, with lasting consequences.
Since 2025, many Facebook and Instagram users have complained about mass bans they blame on AI moderation. The lack of human moderation has only fueled frustration among users who say they did not violate any rules, especially since there has been no way to speak with a Meta employee about what caused the ban or how to get an account reinstated. Meta has not said whether AI is behind the bans, but the company has increasingly relied on generative-AI-based moderation rather than humans in recent years—a shift that some people, including Meta employees , say is happening too quickly .
Tumblr is another social community…
Read the original at Ars Technica →
Open in TruthVane →