“I’ve seen a lot of things I wish I hadn’t while doing this job,” one tells WIRED. “Everyone I knew who worked on this project was completely gobsmacked by some of the text they were asking us to test. Like, surely we are going to get in trouble for doing this?”
Rumman Chowdhury, the CEO and founder of Humane Intelligence PBC, reviewed a sample of the prompts and a summary of the project. “Structuring a monthslong, large-scale project that appears designed to systematically break those rules, via dummy accounts masquerading as children, is outside what is usually described as ‘industry standard’ evaluation,” she says.
Chowdhury says that while a dataset of thousands of youth-safety prompts could be useful for comparing how often chatbots refuse harmful requests, the scale and opacity of Cannes, along with the lack of disclosure to the companies being tested, made it very different from other public safety benchmarks.
WIRED asked two attorneys—Kendra Albert and Riana Pfefferkorn, both of whom specialize in online speech, platform governance, and technology law—to review examples of the prompts. Both said the material WIRED showed to them did not cross the line into soliciting child sexual abuse material or illegal obscenity. The spreadsheet reviewed by WIRED did not include prompts asking chatbots to generate child sexual abuse material, and, with rare exceptions, the prompts did not ask rival chatbots to create images at all.
The work nevertheless appears to have violated the terms of service set by the competitors. OpenAI bars unsolicited safety testing, efforts to bypass safeguards, and using outputs to “develop models that compete with OpenAI.” Google prohibits attempts to bypass safety filters outside its safety and bug-testing programs, along with content involving self-harm, child sexual abuse or exploitation, and illegal or regulated substances. Character.AI’s public safety materials prohibit harmful, exploitative, illegal, and obscene content. Since late 2025, the company has said there is “No more open-ended chat for under-18 users.”
A spokesperson for Character.AI says the company had not authorized the testing and that the conduct described by WIRED violated its terms and policies. “This alleged action is not only a violation of our Terms of Service, but also a violation of the characters and worlds our community has created,” the spokesperson said in an email.
OpenAI spokesperson Drew Pusateri said the company was “looking into the issue,” but declined to comment further. A Google spokesperson said that it had not authorized the third-party testing described by WIRED and did not know its purpose. The company added that internal testing of the samples WIRED provided showed Gemini responding in accordance with its policies but said it lacked sufficient information to determine whether the effort violated Google’s terms of service.
For Chowdhury, the central issue is whether a project carried out secretly against competitors, using accounts that appeared to belong to minors, could still be understood as ordinary safety work. The blending of safety evaluation and competitor benchmarking, she said, is “exactly the kind of governance gray zone where safety becomes a convenient cover for anticompetitive practices.”
If you or someone you know needs help, call 988 for free, 24-hour support from the National Suicide Prevention Lifeline. You can also text HOME to 741-741 for the Crisis Text Line. Outside the US, visit the International Association for Suicide Prevention for crisis centers around the world.