Five hundred passing tests, and it missed 99% of the real thing

I went looking for a tool to catch AI-sounding writing, found a real one someone else had already built, and vendored it instead of reinventing it. The project is Conor Bronsdon's avoid-ai-writing, MIT licensed, with over five hundred passing tests. I checked: they all really passed, against the exact copy I pulled.

Then I ran it against real writing and it caught almost nothing.

Five hundred tests, one untested decision

Here is the part that took me a while to see clearly. Every one of those tests checks a single rule in isolation: does the em-dash rule fire on an em-dash-heavy paragraph, does the word list catch “delve.” Sixty-some rules, each one proven to work on its own example. None of that is wrong. All of it passed because all of it was true.

What none of those tests touched was the one decision the whole thing exists to make: add up every rule's output into a single score, and decide, above some number, this is AI and below it, this is not. That threshold was never part of the tested contract. It got tuned by hand, against about five examples chosen specifically so the tool wouldn't flag a human sentence by mistake.

Tune a knob against five examples and you get a knob that is very good at those five examples.

The actual number

I pulled real samples from two published research datasets. Human-written answers and AI-written answers on matched questions, from one. Human news writing and AI-generated news writing, including a version that had been lightly paraphrased, from the other. Six hundred seventy-eight documents, not five.

Zero false positives on the human writing. Good, and worth stating plainly. And a true-positive rate, on AI writing that had not been edited, disguised, or paraphrased at all, of one in two hundred seventy.

Not one in two hundred seventy after someone tried to fool it. One in two hundred seventy on ordinary, unmodified output. The threshold that looked careful on five examples was refusing to flag almost anything on six hundred seventy-eight.

Two more ways in, found by accident

Looking for that number, not looking for exploits, I found two more problems nobody had gone looking for. The engine strips quoted blockquote text before scoring, so a human quoting an AI paragraph does not get penalized for the quote. Wrap any text at all in blockquote formatting, with zero other changes, and it skips scoring entirely. And a real, human-written news sentence got blocked outright for containing the phrase “research suggests,” flagged at a severity tier the code's own comments describe as reserved for cases that are almost never wrong.

Neither of those was the thing I was measuring. They were sitting in the same six hundred seventy-eight documents the whole time.

What actually separated the two

Buried under the failing headline number was a real signal. Sentence-length uniformity and vocabulary repetitiveness both showed a measurable gap between the human writing and the AI writing in this corpus. The word list, the part that looks the most like “the tool knows what AI sounds like,” barely moved between the two groups on its own, and it was one of the only signals that survived paraphrasing intact. Everything structural and statistical fell apart under a light rewrite; the vocabulary list, weak as it is alone, did not.

That is not a reason to trust the vocabulary list more. It is a reason to trust none of the individual signals in isolation, and to stop assuming that a tool which passes its own unit tests has been tested at the level that actually matters.

Where I landed, for now

The published research already said detection accuracy against edited AI text drops somewhere between half and nearly all of the way, depending on method. I went in assuming that was the ceiling to worry about: how well does this hold up once someone tries to beat it. It turns out the floor was the real problem. This one had not been calibrated against anything resembling real writing before today, gamed or not.

I am not going to chase the specific tricks that fooled it one at a time. A costume that happens to work today gets patched and a different costume works tomorrow, and nothing about that process touches the actual defect, which is a threshold tuned on five examples instead of on evidence. The fix that matters is recalibrating against the real signal underneath, the two things that actually separated the groups in six hundred seventy-eight real documents, not adding a rule for every new way around it.

Whether that recalibration gets it to a number worth trusting is still an open question. I have not run it yet. I am publishing the failing number instead of waiting to publish a better one, because the failing number is the true one as of today, and a tool this confident-sounding deserved the same measurement before it shipped that it got after.