AI apps with hidden sexual and violent deepfake features made it into the App Store

Hundreds of AI apps with hidden sexual and violent deepfake features made it into Apple’s App Store after passing app review. The same apps were also available in the Google Play Store.

An investigation discovered more than 400 apps, apparently created by a single Chinese organization. The apps allow users to upload photos and then generate sexual and violent deepfakes which could be used to humiliate or threaten victims …

A Wired investigation discovered a network of more than 400 connected domains and apps with hidden abusive functionality.

Apps like Viefur [ostentibly used to add clothing to pet photos] often claim to be everyday services—such as language tuition or AI workout coaches. When accessed under certain conditions, however, apps in the network stealthily load a web interface allowing the generation of explicit AI sexual images and videos using photos users upload. The apps include AI scenes such as “bedroom rape” and “threatening knife,” where a man holds a knife to a woman’s neck.

Wired says that the network is “hiding in plain sight,” thanks to thousands of video ads for them published on Facebook and Instagram. Some of the apps have been around for a year and have passed review by Apple, Google, and Meta despite each having clear policies prohibiting harmful deepfake content.

Bait-and-switch tactics were used to pass app review. The apps normally show benign content, but a server check can activate the hidden features. Downloading the app after tapping an ad for the deepfake functionality on Meta platforms appears to be one of the triggers. Dozens of pornography photo and video templates can then be downloaded, with credits purchased to generate deepfakes from user-uploaded photos.

Apple removed 19 apps after Wired shared details.

We review every app before it’s published, and these apps complied with our rules when we reviewed them. After approval, their developers changed the website the app pulls its content from, replacing what we approved with pornographic content, and promoted it through pornographic ads hosted on a third-party platform.

The company recently strengthened protections against this approach. Meta says it has taken action against more than 10,000 ad accounts linked to domains associated with these apps.