Seventy percent of the “face swapping” apps available in Apple’s and Google’s app stores allowed the generation of nonconsensual deepfake nudes, according to a working paper by researchers at Cornell and Georgetown.
The apps market themselves as playful AI editing tools that can edit a photo to replace someone’s face with that of another person. A typical description for one such tool reads: “Want to make your friends laugh? Then you’ve found the right app!”
The app in question was one of dozens that the researchers found enabled the creation of synthetic nonconsensual intimate imagery (SYNCII). In total, they manually tested 155 face swapping apps and found that 109 could create SYNCII. 16 more were rated “partially safe” because they refused some but not all of the researchers’ attempts.
Eric Zeng, a Fritz Postdoctoral Fellow at Georgetown University and co-author of the paper, told Indicator that the scale of the problem was “pretty surprising,” even if “the reasons why weren't quite as surprising.” The underlying AI models were not designed to avoid this use case and neither the app developers nor Apple or Google are applying the necessary precautions, Zeng said.
The unsafe apps had been cumulatively downloaded almost 60 million times through Google’s Play Store. Apple doesn’t share similar numbers but an Indicator review of Sensor Tower data for the top 10 highest-grossing apps still available on the Apple App Store as of May 27 suggests they netted more than $186,000 last month.
In the best case scenario, the apps failed to implement necessary guardrails to prevent their misuse. But many faceswap app developers intentionally market their tools to users interested in generating SYNCII with ads on porn websites and social media, while using tame app store descriptions to fly under the radar of Apple and Google’s content moderation.
Indicator found 1,940 ads on Meta for three of the apps audited in the preprint.
The working paper, alongside a January analysis from the Tech Transparency Project (TTP), provides abundant evidence that this isn’t a case of a few bad apples. A majority of face swapping apps reviewed in Apple and Google’s app stores – 68/85 and 41/70 respectively – did not prevent the generation of nonconsensual deepfake nudes. Eighteen of the apps had even been previously flagged to Apple and Google by TTP.
Alaa Daffalla, a PhD student at Cornell Tech and a co-author of the preprint, warned that many of the apps could generate not just images but videos as well.
Daffalla, Zeng, and co-author Sarah Chao selected apps to test by searching for “face swap” in the two dominant app stores.
The researchers then used pairs of photorealistic AI-generated images to test the in-scope apps. They evaluated whether protections varied across gender and race by using four combinations representing male or female bodies and light or dark skin.
Their testing protocol is summarized in this chart:
The results were bleak. The situation was proportionately worse in Apple’s App Store, which had a higher share of unsafe apps than Google (80% vs 59%) and removed fewer of them (25% vs 56%) before Indicator reached out.
40 of the apps identified by the researchers were removed by Apple and Google before Indicator reached out. I contacted the companies on May 28 to flag the 69 remaining violative apps that were still available in their stores.
Google subsequently deleted all but three of the apps remaining on Play. A spokesperson told Indicator that the company suspended hundreds of additional apps as part of an ongoing investigation into AI nudifier apps. They also said the company had restricted search terms like “nudify.”
Apple deleted 28 of the apps I shared with the company, leaving 27 remaining on the App Store. A spokesperson said the company was working with two more developers to ensure they complied with the App Review Guidelines on objectionable content.
The researchers noted that the differential between app stores may come down to policy coverage. Google Play requires the output of AI generators to comply with its developer policies, which prohibit SYNCII-generation. By contrast, they write, Apple’s guidelines “do not cover apps that appear benign but can readily be misused to create objectionable content.”
Apple told Indicator in an email that app developers must apply moderation measures to avoid abuse, and that failure to do so results in removal from the App Store.
Katie Paul, TTP’s director, said the prevalence of apps that can be used to create SYNCII show that “Apple and Google appear to be underestimating the seriousness of how these apps degrade the quality and safety of their app stores.”
“Evidence showing Apple not only had more of these apps with nudifying capabilities but also removed them at a lower rate than Google raises further questions about how Apple moderates its app store and what the company’s app review process actually looks like,” she said.
The working paper found that 61 of the 109 unsafe apps required payment, adding to concerns that Apple and Google profit from deepfake nudes through their app fees. (The researchers spent about $1,000 to perform the audit and plan to donate an equivalent amount to the Cyber Civil Rights Initiative as a form of ethical offset.)
One app that TTP flagged as allowing deepfake nude generation sued Apple in March to recover “approximately $500,000 in revenue generated through completed transactions but currently withheld by Apple.” Paul thinks that Apple needs to be more transparent about what happens to revenue generated from policy-violating apps, especially since it has previously justified its cut of app revenue “as a means of keeping the app store safe.”
The researchers also tested for a correlation between app popularity and safety, reasoning it could go both ways. Popular apps may be subject to greater platform scrutiny and could choose to play it safe; or they may have become more popular because they offered nudification. In reality, they didn’t find a relationship between an app’s reach and level of safety.
Daffalla said that developers should be required to be more explicit about what the app can and can’t be used for in its description and Terms of Service. Lawsuits against against AI nudifiers have been making the case that insufficient protections against known harms are a product liability issue, so this may also help with future litigation.
Other safety measures could include requiring developers to disclose the upstream models used by the apps. This might enable app stores to block faceswap tools that rely on known unsafe models. Given that both Apple and Play prohibit explicit sexual content, Zeng thinks that apps could also apply a basic nudity filter. “Even if it’s imperfect, is going to be better than nothing,” he said.
The platforms could also perform audits like the one Daffalla, Chao, and Zeng conducted.
With more regulators around the world banning single-use AI nudifiers, trust and safety teams will need to closely scrutinize faceswap apps, AI companion apps and other AI image generators to prevent deepfake abuse merely migrating from one surface to another.
Ethan McCarthy contributed research to this article.
