How AI detectors actually work

AI detectors do not recognize AI. They measure two statistical properties of text and infer AI from them. Once you know what those properties are, both the successes and the failures make sense.

Perplexity

Perplexity is a measure of surprise. Given the words so far, how predictable was the next one? Low perplexity means the text keeps choosing the most likely continuation.

A language model generating text is, by construction, sampling from high-probability continuations. So its output tends to have low perplexity. People are less predictable — we reach for an odd word, change direction mid-sentence, use a phrase because we like it rather than because it fits the distribution.

Burstiness

Burstiness measures variation across a document. Human writing is bursty: a long, clause-heavy sentence followed by a short one. A paragraph that runs away from us followed by a tight one. Model output tends to cluster — sentences of similar length, paragraphs of similar shape.

Detectors compute how much a document varies and treat low variation as a signal.

The same problem applies. Formal academic writing is deliberately uniform. Heavy editing smooths out variation — you cut the rambling sentence and tighten the loose one, and in doing so you make your writing look more like a model’s.

What the score is

The detector combines these signals into a probability and presents it as a percentage. That percentage is the classifier’s confidence, not a proportion of your document that was generated, and not a measurement of what you actually did.

This distinction matters when a number gets treated as a finding. “87% AI” does not mean 87% of the words came from a model. It means a classifier’s output crossed a threshold.

Why detectors disagree

Different detectors use different reference models, different training data, and different thresholds. Run the same paragraph through three and you can get three different verdicts. That is not a scandal; it is what you would expect from three classifiers estimating the same unobservable thing.

It is, however, a strong argument against treating any single score as definitive.

Can this be fixed?

Not straightforwardly. The problem is that "text that is statistically regular" and "text produced by a model" are overlapping categories, not identical ones. As models get better at producing varied, surprising prose, the signal weakens further.

Watermarking — embedding a detectable signature at generation time — would be more reliable, but it only works for models that implement it and survives only until text is edited. There is no general solution on the horizon.

Which is why process evidence matters more than scores. A document’s revision history showing days of drafting is a record of what happened. A detector score is a guess about what happened.