Simple OSINT techniques to spot AI-fueled disinformation, fake reviews

Error messages that ChatGPT and other AI language models generate can be used to uncover disinformation campaigns, hate speech and fake reviews via OSINT collection and analysis, says Nico Dekens, director of intelligence at ShadowDragon.

AI-generated content found via Google (Source: ShadowDragon)

Uncovering AI-generated content through OSINT

Dekens has recently tried to pinpoint specific indicators that reveal online content – news, social media posts, reviews – created with the help of AI models.

He concentrated on known error messages that ChatGPT and other AI language models occasionally deliver, such as “Violates OpenAI’s content policy” (when the input includes guidelines on topics such as hate speech, violence, and misinformation), or “I apologize, but I’m not sure what you’re asking for” (when the input is unclear or ambiguous).

“By using these error messages as a starting point, we can better understand the ways in which AI-generated content is being used to spread disinformation and false information online,” he said.

And he found a lot of them – on social media platforms, online shops, publishing platforms, job review sites, etc.

AI-generated content found via Twitter’s search function (Source: ShadowDragon)

These error messages are easy to spot and tech-savvy users will immediately recognize that the content is AI-generated.

But even if these most obvious “red flags” are removed, there may be language patterns that indicate (though not conclusively prove) that a text is AI-generated, he noted.

Using AI for malicious purposes

When the first AI language models started becoming available, we knew it was only a matter of time until they would be used for malicious or unethical ends: plagiarism, manipulating human behavior, writing malware, and so on.

Spreading disinformation is a particularly harmful activity with the ultimate goal of manipulating people through false or misleading information, and AI-based models are the perfect tool for creating convincing text, images or videos. The ability to spot these campaigns will be crucial for exposing them.

Of course, not all disinformation campaigns, hate speech or fake reviews are powered by AI tools – and those will still be difficult to spot.