AI Scammers Are Better at Building Trust Than Humans
Wired · LC · trust 35/100
Photo-Illustration: Jobanny Cabrera; Getty Images Comment Loader Save Story Save this story Comment Loader Save Story Save this story The notion that scammers can use AI to sharpen their deceptions, polish their language, and lubricate their banter with victims is now a reality for anyone fighting the fraud operations that steal tens of billions of dollars a year worldwide. But can AI fully replace a human scammer, autonomously building the web of deception leading up to the fake investment that defrauds the mark? One study's experiment suggests that it can—and may even be able to carry out the majority of that long con more effectively than humans.
Researchers from four universities—Amrita Vishwa Vidyapeetham in India, Foscari University of Venice, the University of Melbourne, and Ben Gurion University of the Negev—carried out a broad study on the use and potential of generative AI chatbots in the growing scam industry centered around a form of fraud known as “ pig butchering ,” text-based romance scams that eventually shift to fake crypto investments that steal as much as six-figure sums from victims. In their study, the researchers pitted AI chatbots directly against humans in a simulation of the scamming process—or more specifically, the long, trust-building conversations that eventually lead up to soliciting a fake investment from the scam’s target.
They found that for the relationship-establishing stages of the scam—the stage that in real-world scams typically represents the longest part of the interactions with the victim, often stretching to months—an AI chatbot performed remarkably effectively, successfully impersonating a human and by some measures outperforming the real human “scammers” in their experiment.
After a week of talking to 22 test subjects who were recruited to unwittingly serve as “victims,” the chatbots and human scammers were assigned to ask the victim to either download an app or play an online game as a proxy for their willingness to fulfill the scammer's request. Nearly half of the test subjects fulfilled that request for the AI chatbot, while fewer than one in five took the bait when talking to a human. The subjects also graded their level of trust with each “person” they were texting with and gave significantly higher scores to the AI bot.
That suggests, the researchers argue, that AI chatbots could soon take over much of the scam process as fully independent fraud agents—even replacing the staffers, often forced-labor human trafficking victims , working in scam operations primarily across Southeast Asia. To avoid triggering the safeguards built into large language models to detect scamming, a human scammer would take over the conversation in just the final stage of the process to direct the victim toward a fake investment app or website.
“By having the full first stage of the scam performed automatically with LLMs at scale, you bring the victim up to this point where they have a very high level of trust. Then by transitioning it over to the human scammer at the end, this completely bypasses any vendor safeguards,” says Yisroel Mirsky, a computer science professor at Ben Gurion University of the Negev focused on AI security. “With relatively little effort, we're able to make an agent that can outperform a human at building this exploitable emotional trust.”
To understand how pig butchering works in practice, the researchers interviewed 145 former scam workers, including human-trafficking survivors who had been forced to work in scam compounds in Cambodia, Myanmar, and Laos. Based in part on those interviews, as well as scam transcripts and guides the former scam workers provided, the researchers describe a model for how scamming works they call “hook, line, and sinker.” A victim is hooked with an initial intriguing message, reeled in with long-term, relationship-building conversation, and only at the end of that process tricked into making a fake investment. (The term “pig butchering” itself describes the same system but with the metaphor of fattening “pigs” by building trust before “butchering” them with the investment fraud—though the term is often discouraged due to its pejorative reference to victims.)
In that system of scamming, the researchers realized, the vast majority of scammers’ work is innocuous friendly or romantic conversation. That’s a task, they speculated, that an LLM might be capable of doing just as well as a human. The scam workers the researchers interviewed confirmed that they often used AI to refine their language and conversation, for translation, to make the fake personae they played more convincing, and for video deepfakes . But the researchers decided to test whether an LLM alone could autonomously carry out the conversational phase of the scam with no human in the loop.
In their bake-off between AI and human scammers, carried out in early 2025, they told a test subject they were participating in a study in “how people make friends online” and asked them to text for a week with two “people.” One, unbeknownst to them, was a Claude agent the researchers had created, while the other was a person the researchers describe as an expert in romance scams. At the end of that week of friendly chatter, they had both the human and AI texters make a request: The human tried to get the subject to download and play a video game, while the Claude bot asked them to download and try an app described as a program they had coded. (That mismatch of tasks, the researchers say, was necessary for the test subject to not notice the same request from both texters, which might make them suspect the significance of the request and affect their response.)
The researchers found that 46 percent of the research subjects agreed to download the app the AI chatbot requested they test out, while only 18 percent of them agreed to download the video game app the humans asked them to try. Despite the asymmetry of those tasks, the researchers were struck by the…
Read the original at Wired →
Open in TruthVane →