An AI-generated video claiming Iran had hit a US aircraft carrier and killed people drew 643,000 views on X, a DW fact check published on 6 November found. The clip had circulated since at least May alongside a different claim.
A CNN report published on 3 November described researchers identifying more than 220 accounts on X that appeared to run financial or sex-bait scams. Earlier studies of X and Facebook, using data from 2022 to 2024, had documented synthetic media's reach and, on Facebook, its use in spam and scams.
Both Cases Use Attention to Pursue Financial Gain
Another post the fact check examined, claiming an Iranian hypersonic attack on Israeli bases, drew 2.5 million views, though it found no credible reports of such an attack. A NewsGuard analyst wrote that the account is a South Asia-based content farm, which mass-produces cheap attention-grabbing posts, and a researcher of social media monetisation wrote that financially motivated actors use news events to profit from engagement.
The CNN case ran on a bait-and-switch, the researchers said. Some AI-made anti-Muslim posts drew thousands of comments, and lead researcher Meriem Mahdhi wrote that 'The anger is the hook and the outrage gets shares and comments'. Followers who engaged were then steered towards fake investment schemes and Telegram channels. One researcher who messaged an account on Telegram was warned of a coming bank collapse and sent a link to supposedly open a new bank account.
A 2024 Facebook Study Documented the Same Mechanics
A study in Harvard Kennedy School's Misinformation Review, published in August 2024, examined 125 Facebook pages that each posted at least 50 AI-generated images and sorted them into spam, scam and other creators. Its authors say spammers and scammers appear motivated by profit or clout, though they cannot confirm the operators' aims.
Scam pages attempted to sell products that do not exist or to obtain personal details. Pages in the sample paired AI images of old people, amputees and infants with captions such as 'No one ever blessed me', and comments often suggested viewers did not realise the images were synthetic.
The authors found the feed at times recommending unlabelled AI images to people who did not follow the pages, and Meta's reports show the share of feed views from unconnected posts, meaning those outside a user's friends, followed pages and joined groups, rose from 8% in Q2 2021 to 24% in Q3 2023. The authors suspect the algorithm partly drives engagement.
The X Study Found Vast Reach and Limited Malicious Use
A separate study in the same journal analysed AI media that X users flagged through Community Notes between December 2022 and September 2023. The flagged tweets drew more than 1.5 billion views, though the authors say the sample is likely a small subset of such content.
Most of the media was non-political and appeared harmless, often humorous, while malicious deepfakes of political figures were less frequent. The authors call AI a 'threat multiplier' for existing risks, particularly in conflicts and elections, but say they do not posit a straight-line link between better AI media and a misinformation crisis.
X did not respond to CNN's request for comment, though many of the accounts were suspended after CNN asked about them. Meta told the Facebook study's authors it had reviewed the pages in their report and taken action against those engaged in inauthentic behaviour. In both cases, the companies' action became public through an outside inquiry.
