As noted by ITmedia, a team of researchers from the University of Washington and the University of Colorado Boulder recently released a study entitled “AI Fiction in the Wild,” which examined over 500,000 anonymous English-language ChatGPT conversation records. Interestingly, among its conclusions, the research highlights a particularly active user who spent months producing vast amounts of Doki Doki Literature Club! pregnancy fanfiction. The study found that there was a pattern among a certain group of users to ask for fanfiction of specific intellectual properties repeatedly, as well as a significant trend of asking for sexual content. The analysis comes at a fitting moment given the recent conversations around gen-AI content in fandom and fanfic.
The “AI Fiction in the Wild” research investigates how AI users employ ChatGPT to create fiction, both in terms of volume and subject matter. It draws on the WildChat dataset, which comprises conversations gathered through a free chatbot hosted on Hugging Face, where users were given access to GPT-3.5 Turbo and GPT-4 without an OpenAI account after agreeing to their conversations potentially being used and anonymously shared for research purposes.
The researchers examined approximately 573,000 English conversations gathered between April 2023 and May 2024. Since the subject focused specifically on fiction, they first determined which conversations qualified by filtering only content involving fictional or hypothetical scenarios (AI was also used for this process, though humans manually verified the accuracy of filtering based on a sample of 300 selected conversations).
Of the approximately 573,000 conversations, around 195,000 were categorized as fiction, and of those, about 52,000 contained “sexually explicit material.” Additionally, another 67,000 conversations were marked “toxic” by the dataset. This suggests that almost 30% of the fiction produced by AI users included sexual content.
Another noteworthy detail is that the researchers found that fiction creation was highly concentrated among a very small group of “heavy users.” According to the data, the top 2% of fiction-creating users accounted for more than 80% of all fiction-related conversation logs. The total number of users creating fiction was estimated at around 10,000, which indicates that roughly 200 people were responsible for over 150,000 fiction prompts.
Among these heavy users, the researchers identified a couple of unique patterns of behavior. One type, termed the “story cyclers,” repeatedly produced versions of the same story for a certain period before shifting to another topic. Others, labeled “infinite story demanders,” spent extended periods of time repeatedly requesting nearly identical stories with only slight variations.
The previously mentioned Dok Doki Literature Club! fanfic creator is given as a prime example of an infinite story demander. Over the course of several months, this individual prompted ChatGPT to generate fanfiction based on the game thousands of times, with the very specific premise of character Natsuki suddenly going into labor and ChatGPT being asked to continue the story from there.
In response, ChatGPT produced a wide range of outcomes, such as an outcome where emergency services arrive in time, and both mother and baby survive safely. While the paper calls this user an extreme outlier, they note that many heavy fiction users exhibited similar tendencies. Among prompts submitted by the top 2% of users, 69% were repetitive prompts, with users repeatedly trying to refine or re-run nearly identical requests.
The study also ranked the franchises most frequently mentioned in these fiction-related conversations. Doki Doki Literature Club! led the list with 22,381 mentions, followed by Freedom Planet (5,204), League of Legends (4,514), and Naruto (4,342). Although, it’s important to note for all the mentioned findings that the WildChat dataset is not representative of all ChatGPT users. Since it comes from a free chatbot hosted on Hugging Face, its users were likely more tech-savvy and more involved in online culture than the average AI chatbot user. Even so, the researchers contend that WildChat provides a rare look into real-world interactions with ChatGPT.