Technology-Facilitated Gender-Based Violence in an Era of Generative AI

"Generative AI enables the creation of fake images, audio, text and video with amazing speed and scale."



This report, published by the United Nations Educational, Scientific and Cultural Organization (UNESCO), seeks to explore the potential impact of generative artificial intelligence (AI) on technology-facilitated gender-based violence (TFGBV). As the impact of generative AI on GBV is still too early to ascertain, this report seeks to shape a speculative - but realistic - view into what generative AI will mean for TFGBV. This perspective is required so that proactive measures can be instituted, rather than waiting for at-scale harms to occur before acting. Through a series of experiments, the author reveals how generative AI facilitates GBV and, based on the insights gained, provides guidance for a variety of stakeholders on how to address this growing threat.



This report is published under the UNESCO series World Trends in Freedom of Expression and Media Development. It is the result of an intersectoral cooperation initiated by UNESCO's Division for Gender Equality with the Section for Freedom of Expression and the Safety of Journalists and the Section for Peace and Sustainable Development. The report is aligned with UNESCO's work on raising awareness and producing solutions for GBV on the internet.



As explained in the report, generative AI, defined as "deep-learning models that create voice, text, and image - are revolutionizing the way people access information and produce, receive and interact with content. While technological innovations like ChatGPT, DALL-E and Bard offer previously unimaginable gains in productivity, they also present concerns for the overall protection and promotion of human rights and for the safety of women and girls. The arrival of generative AI introduces new, unexplored questions: what are the companies' policies and normative cultures that perpetuate technology-facilitated gender-based violence and harms? How do AI-based technologies facilitate gender-specific harassment and hate speech? What 'prompt hacks' can lead to gendered disinformation, hate speech, harassment, and attacks? What measures can companies, governments, civil society organisations and independent researchers take to anticipate and mitigate these risks?"



This report seeks to address these issues. It builds on trends seen in the past as well as evidence-based pointers for the future in the form of experiments/demonstrations (in the role of a potential malicious actor) into the risks associated with the design, deployment, and use of generative AI to facilitate GBV. It assesses the possible impact posed by generative AI that enables the creation of more realistic "synthetic" media, "hallucinations" or biases in the outputs, automated harassment campaigns, and the ability to build "synthetic histories" and compositional deepfakes. It also presents lessons learned from the experiments conducted on how gender-based cyber-harassment templates are and can be generated.



The key findings as highlighted in the report are:

1. Overall, generative AI has amplified existing methods and increased the potential avenues for TFGBV faced by many communities online.
2. While there are no global statistics on TFGBV in the era of generative AI, there are relevant global estimates from 2020 showing that 58% of young women across the world have faced some form of GBV on social media platforms.
3. A majority of the apps developed to help women be safer online place an onus on the victim to protect themselves against online harms.
4. The proliferation of generative AI brings with it new harms, including the creation of more realistic fake media, "hallucinations" or unintended biases in the outputs, automated harassment campaigns, and the ability to build "synthetic histories" - realistic false narratives. In addition, generative AI introduces the potential for unintended harms via embedded biases in the model training data.
5. Generative AI can lead to an increase in the number of attackers, the creation of sustained and automated attacks, and the generation of content such as posts, texts, and emails that are written convincingly from multiple "voices". This makes existing harms such as hate speech, cyber harassment, misinformation, and impersonation - all of which rank in the top five most common vectors of TFGBV - have a much wider reach and be more dangerous.
6. Hands-on demonstrations conducted by the author of the report show that both open and closed AI models generate cyber-harassment templates, synthesise fake pasts for people, and modify images to portray people in various non-consenting scenarios. The demonstration also highlights how these generative AI harms can be used to propagate some of the most common TFGBV harms today, such as impersonation, hacking and stalking, and cyber-harassment. Some key attack vectors:
   • TFGBV on social media commonly starts with cyber harassment (used 66% of the time as a tactic), something that can be exacerbated with the help of AI-generated harassment templates;
   • Text-to-image models can easily generate images of women in situations they did not consent to being in, thus creating a more realistic vector of image-based abuse; and
   • Creating synthetic histories is a new vector of TFGBV harm. It allows attackers intending on spreading misinformation to use text-generative AI models to come up with convincing fake reports and histories that cast the target in a bad light, with the objective of casting doubt and defaming the individual - one of the top methods of inflicting TFGBV today.

The report concludes with a combination of measures to be put in place by generative AI companies and the technology companies that platform them, by regulators and policymakers, by civil society organisations and independent researchers, as well as by users. The following is a selection of the proposed measures:

• Content distributors should, for example:
  • Conduct human rights due diligence, assessing their human rights impact, evaluating the gender-related risks and defining mitigation measures.
  • Develop better methods of reporting - including more robust reporting mechanisms that identify falsified content.
  • Examine methods of protection that do not involve removing the victim from the public sphere.
  • Create proactive solutions for identifying falsified content, including auto-checking for watermarks and improving content identification.
• Content generators should, for example:
  • Conduct human rights due diligence, assessing their human rights impact, evaluating the gender-related risks, and defining mitigation measures.
  • Develop robust methods of identifying generated media, which can help stop the flow of misinformation. Examples include adding watermarks to generated content as an easy way for people to identify it.
  • Clearly share their terms of service, guardrails, and safeguards, and monitor use for inappropriate content, including a zero-tolerance policy for abusers.
  • Encourage and support independent observatories and initiatives to monitor and address coordinated and automated harassment campaigns.
• Civil society and independent researchers should, for example:
  • Expand their toolkits to include generative AI-based harms identification and protection, and be aware of ways these harms can help TFGBV manifest.
  • Advocate for protections for the most at-risk individuals from companies and policymakers/governments.
  • Raise awareness within their own communities of the potential misuses of generative AI, and develop and disseminate media and information literacy programmes and campaigns for their representative audiences as well as policymakers, which can help civil society advocate for actions against TFGBV harms.
  • Identify patterns of abusive behaviour and when possible, address the root causes.
• Policymakers should, for example:
  • Organise digital or in-person town halls with consumers of generative AI systems to gather direct feedback and suggestions that could then be used to identify and keep up to date with ways in which TFGBV attack vectors manifest and inform the laws they create.
  • Review laws and regulations related to content generators and content distributors to be aligned with international human rights standards so as to ensure transparency, accountability, due diligence, and user empowerment.
  • Develop multi-stakeholder media and information literacy programmes and campaigns for their constituencies so individuals are not inadvertently distributing, reacting to, or interacting with harmful generated false content.
• Platform users should, for example:
  • Be vigilant for seemingly falsified information. Platform users should be aware of information that looks "fake", especially when it seems like there is an online campaign of misinformation targeting a particular person.
  • Report harmful and malicious content to platforms, which is one way users hold platforms accountable as they work to reduce TFGBV.
  • Look into tools to protect their own data. Data provenance tools identify whether users' images are in the training dataset of a generative AI model. They enable searching through large datasets and are one way users, especially women, can identify their images used by a content generation platform and request them to be taken down.
  • Take advantage of media and information literacy programmes regarding falsified online content.