AI text detectors struggle when language models mimic an author's style

What happened

Popular AI text detectors catch plain AI-generated text with near-perfect accuracy. But when language models deliberately copy a specific author's writing style, up to one in five AI texts slips through undetected. Scientific writing is where the detectors fail the hardest. 2). The test covered three categories: genuine human writing, AI text generated from simple prompts, and AI text that deliberately mimicked a specific author's style.

Pangram uses a neural network trained on human and machine-generated text, though its founder has called the system a black box since its verdicts can't be traced. GPTZero measures how predictable word choices are and how much that varies within a text, based on the idea that language models write more uniformly than humans. ai searches for statistical patterns it learned during training on human and AI-generated text.

Why it matters

The team built a corpus of 495 human passages from 99 authors, evenly split across blogging, fiction, and scientific writing. All texts were written before ChatGPT's release in November 2022, which effectively rules out contamination by language models. 7 percent. Human texts were also classified correctly for the most part. Pangram and GPTZero didn't produce a single false alarm. 8 percent.

That result changes when language models receive writing samples from an author as reference material. 1 Pro) each received five real text passages from an author and were asked to write new text in the same style. Of the 297 passages generated this way, an average of 38 went undetected, according to Epoch AI, which works out to a false-negative rate of about 13 percent. ai missed 18 percent.

For fiction, the false-negative rate across all detectors sat at just 1 to 5 percent. scientific writing told a very different story. ai missed 29 percent. The worst individual results showed up in specific model-genre combinations within scientific writing. Pangram missed 48 percent of Gemini-generated academic passages, according to the published data. 5 texts went undetected.

What to watch

Despite these differences, all three detectors show the same pattern. They catch text from simple prompts almost every time but miss imitations far more often. Scientific writing, the genre where AI detection probably sees the most real-world use, remains the hardest to flag correctly. ai reliably classified human texts as human.

The Epoch AI study fills in the other half of that picture: a low false-alarm rate on human writing says little about how many AI texts actually slip through.