Field notes on governed agentic AI — what was measured, what it cost, and where the evidence runs out.
I vendored a well-tested AI-writing detector instead of building my own. Its tests all passed. Then I ran it against 678 real samples from two published datasets and it caught one in 270 — not after someone tried to fool it, on ordinary unmodified output. What five hundred passing tests never checked, and the two signals that actually held up.
A note pointing at a wrong claim in another file, written carefully and accurately, turned into a false statement the moment the claim was corrected. Fixing the defect raised the number of wrong things by one. Three measurements that were true when I took them, and the type error underneath all of them.
An accurate number nearly walked me into a second automatic removal. Every input was correct and the conclusion was false — because the figure I read and the threshold I checked it against described different populations. Why this survives the review an exaggeration wouldn't.
A cleanup line that became a source-tree wipe on every successful run, and a guard that switched itself off on the exact day it started mattering. Neither changed. Neither failed at the moment it became wrong.
Three recordings of the same quiet room. The median held within 2.6 dB; the peak varied by 8.6 dB — which means a peak-triggered voice detector fires on an empty room. Scope: one room, one mic, three takes, and that limit is the point.
I got explicit permission before moving sensitive data between two systems, and still sent too much. May this move and should this much move are different questions, and the owner can only answer the first one.
An adaptive parameter retunes the transform, so the output domain moves. A fixed function never changes and its output adapts because the input did. Same phrase in a design doc, opposite consequences — plus the parser that grew rows nobody could ever satisfy.
Four commands that returned a confident wrong number instead of an error — an empty grep pattern, a redirect stub, a streamed React payload, and a caret eaten by cmd.exe. What each one actually did, and what I replaced it with.
The most-cited number about AI code quality does not appear in the report it is attributed to. I went and read the source. Here is what GitClear actually publishes, and the paper you should cite instead.
This site uses Google Analytics to see which pages get read. No ads, no selling data — just anonymous traffic counts.