# Stylometry research: what it can and cannot measure · ScriptGrain > What peer-reviewed stylometry shows about measuring writing style: which features carry authorship, the accuracy shared tasks actually report, how far it falls when genre changes, and how much text it needs. Canonical: https://scriptgrain.com/reference/stylometry-research # What stylometry research actually shows Stylometry is the statistical study of writing style, and it is older than any AI writing tool: the first quantitative attempt is usually dated to Mendenhall's 1887 study of Shakespeare's plays, followed by Yule and Zipf in the first half of the twentieth century. Its most influential result remains Mosteller and Wallace's 1964 Bayesian analysis of The Federalist Papers, 146 political essays by John Jay, Alexander Hamilton and James Madison, twelve of which were claimed by both Hamilton and Madison. This page collects what the research literature reports, with a source for every figure. It is deliberately not a case for stylometry. The most useful numbers here are the ones that show where it fails: performance on the same kind of text is close to solved, and performance across different kinds of text is close to chance, and the gap between those two facts is the single most important thing to understand about the field. ## Which features carry authorship Stamatatos's 2009 survey groups stylometric features into lexical (word and sentence length, word frequencies, vocabulary richness), character (n-grams, which tolerate noise well: 'simplistic' and a typo such as 'simpilstc' still share many trigrams), syntactic, semantic, and application-specific families. Rudman estimated in 1998 that nearly 1,000 distinct style measures had been proposed by that point, which says as much about the field's early sprawl as about its progress. The features that survived are mostly the boring ones. Function words carry the most reliable signal precisely because, as the survey puts it, they 'are used in a largely unconscious manner by the authors and they are topic-independent'. Published work has used function-word sets of 303, 365 (Zhao and Zobel, 2005), 480 (Koppel and Schler, 2003) and 675 words (Argamon and colleagues, 2007); Burrows's earlier studies treated sets of at most 100 frequent words as adequate. One practical consequence is that style needs far less vocabulary than topic. The survey notes that style-based classification requires 'much lower dimensionality' than topic-based classification: a few hundred words of features, against several thousand for thematic categorisation. ## Lexical richness measures are weaker than they look Vocabulary richness functions such as the type-token ratio (V/N) and hapax legomena counts are among the oldest stylometric measures and among the least dependable on their own. The problem is arithmetic rather than linguistic: vocabulary size 'heavily depends on text-length', rising quickly at first and then more and more slowly, so the same writer scores differently on a 500-word and a 5,000-word sample. Corrections exist. Yule's K (1944) and Honore's R (1979) were both proposed to achieve stability across text lengths, and both were assessed with 'questionable results' by Tweedie and Baayen in 1998. Stamatatos's conclusion is blunt and worth quoting when anyone offers a single richness number as evidence: such measures 'are considered unreliable to be used alone'. ## How much text stylometry needs Eder's DH2010 study set out to find the minimum sample size at which attribution stops being noise. For English novels, 'samples shorter than 5000 words provide a poor guessing', and 'below the size of 3000 words, the obtained results are simply disastrous'. Across English, German, Polish and Hungarian novels the critical point fell 'between 5000 and 10000 words per sample'. Genre moved the threshold rather than removing it: about 3,500 words sufficed for English and Latin poetry, and roughly 2,500 words for Latin prose. Eder's summary is that 'using 2500-word samples will hardly provide a reliable result, to say nothing of shorter texts'. Two secondary findings matter more than they first appear. The threshold was method-independent, with the curves for different attribution methods becoming stable at effectively the same point, which means no clever algorithm buys its way out of needing text. And randomly excerpted 'bags of words' consistently outperformed contiguous passages of the same length, so how a sample is drawn changes what it is worth. ## Error rates from shared tasks, where they are comparable Most published accuracy figures are not comparable, because they use different corpora, different numbers of candidate authors and different evaluation measures. The PAN shared tasks are the exception: the same task, the same metrics, blind test sets, and multiple independent teams. Two consecutive editions happen to isolate the variable that matters most. PAN 2021 tested authorship verification within one text variety, English fanfiction, in a genuinely open-set scenario: the test pairs came from authors and fandom domains absent from the calibration data. Training material ran to 148,000 same-author and 128,000 different-author pairs across 1,600 fandoms, with 19,999 test problems. The best system reached an AUC of 0.9869 and an overall score of 0.9545. The organisers concluded that, within this text variety, open-set verification 'is not necessarily or inherently more difficult than a closed-set setup'. PAN 2022 changed one thing: the two texts in each pair came from different discourse types, drawn from essays, emails, text messages and business memos. The organisers report that 'the effectiveness of all submissions is quite low, reflecting the difficulty of the task', and record a result that deserves more attention than it gets: 'It is also surprising that a naive baseline achieved the best overall score, despite the fact that most participant models are quite sophisticated.' The best submitted system scored 0.587 overall; the naive character n-gram baseline scored 0.600. Shared task · What varied between the paired texts · Best system, overall score · Best AUC PAN 2021 · Author and fandom (same discourse type: fanfiction) · 0.9545 · 0.9869 PAN 2022 · Discourse type (essays, emails, text messages, memos) · 0.587 (a naive baseline scored 0.600) · 0.598 ## Genre dependence is the field's real limit Put those two editions side by side and the honest summary of stylometry is narrow. Holding the kind of writing constant, identifying whether two texts share an author is close to solved, even for authors never seen in training. Change the kind of writing, and the same problem collapses towards chance, to the point where sophisticated systems lost to a naive baseline. The reason is not mysterious. The features that carry authorship also respond to communicative purpose, audience and formality, which is exactly what a discourse type changes. A person writing a business memo and the same person writing a text message differ on sentence length, contraction rate, punctuation and formality by more than two different people writing the same kind of document. This is why a voice profile built from one kind of writing should be expected to score a different kind of writing as off-voice. That is the measurement behaving correctly, not failing, and it is the strongest practical reason to build a profile from writing of the kind you actually intend to produce. ## What stylometry cannot do Stylometric evidence is probabilistic, and the field has been burned before by forgetting it. The CUSUM (or QSUM) technique of Morton and Michaelson gained public attention in 1990 and 'was accepted in courts as expert evidence', before the research community 'heavily criticized it and considered it generally unreliable' (Holmes and Tweedie, 1995). Stamatatos lists the methodological failures that made that possible, and they read as a checklist for judging any stylometric claim today: texts too long and not stylistically homogeneous, usually only two or three candidate authors, evaluation corpora not controlled for topic, assessment 'mainly intuitive' and often based on visual inspection of scatterplots, and no benchmark data allowing methods to be compared at all. A modern claim should therefore state its corpus, its number of candidates, whether topic was controlled, its metric, and its sample lengths. A claim that states none of these is not a measurement, whatever numbers accompany it. ## Methodology Every figure on this page was read from the primary source listed below rather than from secondary reporting, and each is attributed in the sentence that uses it. Quotations are verbatim. Shared-task scores are quoted as the organisers published them, on the organisers' own metrics, and are not converted into a single 'accuracy' figure. The PAN overall score is a composite of AUC, c@1, F1, F0.5u and the complement of the Brier score, so it is comparable between the two editions shown and not comparable with accuracy numbers reported elsewhere. The 2021 and 2022 PAN editions are compared here because they use the same evaluation framework, which makes the drop attributable to the change in task rather than to the change in measurement. They differ in corpus, so the comparison shows the direction and rough scale of the genre effect, not a controlled effect size. Where the literature reports a range rather than a value, the range is given. Where a measure is disputed, the dispute is stated rather than resolved. ## Sources - [Stamatatos, E. (2009). A Survey of Modern Authorship Attribution Methods. JASIST 60(3), 538-556 (doi)](https://doi.org/10.1002/asi.21001) - [Stamatatos, E. A Survey of Modern Authorship Attribution Methods (author's full text)](https://icsdweb.aegean.gr/stamatatos/papers/survey.pdf) - [Eder, M. (2010). Does Size Matter? Authorship Attribution, Small Samples, Big Problem. DH2010](https://dh2010.cch.kcl.ac.uk/academic-programme/abstracts/papers/html/ab-744.html) - [Kestemont, M. et al. (2021). Overview of the Cross-Domain Authorship Verification Task at PAN 2021](https://downloads.webis.de/publications/papers/kestemont_2021.pdf) - [Stamatatos, E. et al. (2022). Overview of the Authorship Verification Task at PAN 2022](https://ceur-ws.org/Vol-3180/paper-184.pdf)