Perplexity

Perplexity measures how predictable a text is to a language model: the average surprise per word. Low perplexity means the model finds each word easy to predict; AI-generated text typically scores lower than human writing.

The numbers

Perplexity level GPTZero associated with human authorship: "a perplexity above 85 is more likely than not from a human source": above 85 (GPTZero)
GPTZero stopped using perplexity and burstiness as its core detection method, migrating to a deep-learning architecture: autumn 2023 (GPTZero)
Perplexity and burstiness now form one of the indicators in GPTZero's upgraded model: 1 of 7 indicators (GPTZero)
False-positive reduction Pangram reports from abandoning perplexity heuristics for iterative deep learning: 2% down to 0.01% (Pangram)

Definition

Perplexity measures how predictable a text is to a language model. GPTZero describes it as "a measure of how likely an AI model would have chosen the exact same set of words as found in the document". Pangram's version: "how unexpected, or surprising, each word in a piece of text is, when looked at from the perspective of a particular language model or LLM".

Intuitively, perplexity is the model's average surprise per word, and it can be read as the effective number of choices the model was weighing at each step. A perplexity of 2 means the text reads as if the model picked between two equally likely words every time; higher values mean the text kept surprising it.

How it is calculated

A language model assigns a probability to each next word given the words before it. Perplexity inverts and averages those probabilities geometrically: multiply them together, take the reciprocal, and take the Nth root for N words. Confidently predicted words pull the score down; surprising words push it up.

The score is always relative to the model doing the scoring. Pangram notes that "closed source providers do not always serve the probabilities of each token, so you cannot even calculate perplexity for closed-source commercial models, such as ChatGPT, Gemini, and Claude". Published perplexity figures are therefore computed with an open substitute model, not with the system suspected of writing the text.

Perplexity versus burstiness

QuillBot separates the pair by level: "Perplexity measures predictability at the level of words, whereas burstiness measures predictability at the level of sentences and text structure." Pangram's framing is tighter still: burstiness is "the change in perplexity over the course of a document". Perplexity is a level; burstiness is how that level varies as the document unfolds.

In classic AI detection, low perplexity combined with low burstiness was read as the machine signature: every word predictable, and predictably so throughout. GPTZero put a working threshold on the first half of that signature, reporting that "a perplexity above 85 is more likely than not from a human source".

Why perplexity alone is a weak AI detector

Pangram's critique lists structural failures. Language models are trained to minimise perplexity on their training data, so famous texts score as machine-like: "perplexity and burstiness detectors classify the Declaration of Independence as AI-generated". Scores are model-relative, token probabilities are unavailable for closed commercial models, and "text written by English language learners is lower perplexity and lower burstiness in general", which biases detectors against non-native writers.

Statistical thresholds also cannot learn: "perplexity-based detectors cannot self-improve with data and compute scale". Pangram reports cutting false positives from 2% to 0.01% only after replacing these heuristics with deep learning, and GPTZero dropped perplexity and burstiness as its core method in autumn 2023, keeping them as one of seven indicators inside a deep-learning system.

Where perplexity fits in stylometry

ScriptGrain's analyser does not compute perplexity, and that is deliberate. The metric requires a specific language model's token probabilities, so it describes a model's reaction to your text rather than a property of the text itself; two models give two different perplexities for the same passage.

The analyser instead measures directly observable attributes in the browser, such as sentence-length variance, which captures the rhythm burstiness points at with deterministic arithmetic and no model in the loop. That is the measurement stance: describe the style, publish the method, and leave detection claims to detector vendors.

Worked example

A toy language model scores a four-word continuation of a sentence, assigning each word a probability. Text A (steadily predictable): probabilities 0.5, 0.5, 0.5, 0.5. Text B (two surprising words): probabilities 0.5, 0.5, 0.25, 0.125.

Perplexity = (1 / (p1 x p2 x p3 x p4)) ^ (1/4). Text A: product = 0.0625, reciprocal = 16, fourth root = 2.00. Text B: product = 0.0078125, reciprocal = 128, fourth root of 128 = 3.36 (to two decimal places).

Text A reads like a steady choice between 2 options per word; two surprising words push Text B's effective choice count to about 3.4. Real perplexity applies the same arithmetic over thousands of tokens, and the probabilities, hence the score, belong to the scoring model, not to the text.

Sources

More terms

ScriptGrain · Why ScriptGrain · Sample voice profiles · The Journal · Support