Type-token ratio (TTR)

The type-token ratio (TTR) measures vocabulary diversity by dividing the number of distinct words (types) by the total number of words (tokens) in a text. Higher values mean less repetition; scores are only comparable between equal-length texts.

The numbers

Definition

The type-token ratio, usually shortened to TTR, is the simplest widely used measure of lexical diversity: how varied a text's vocabulary is. Count the tokens, meaning every running word including repeats. Count the types, meaning the distinct words. Divide types by tokens, and multiply by 100 for a percentage. Sketch Engine's illustration is the phrase 'We met face to face': five tokens but only four types, because 'face' appears twice, giving a TTR of 80%.

A high TTR means little repetition; a low TTR means the same words are being reused heavily. In applied linguistics and stylometry it sits alongside related diversity indices such as vocd and MTLD as one signal of vocabulary range.

The length-sensitivity caveat

TTR has one serious flaw: it depends heavily on text length. The opening words of any text are almost all new, so very short samples score close to 100%. As a text grows, more of its words are repeats of words already used, and the ratio falls. Sketch Engine reports values ranging from around 100% for a single sentence down to 0.11% for a very large corpus, and notes that a larger corpus will nearly always have a lower TTR regardless of how rich its vocabulary is.

The practical rule follows directly: a TTR value cannot be interpreted on its own, and TTR should only be used to compare texts or corpora of the same size. A 200-word email and a novel can never be compared on raw TTR; the novel loses for length reasons alone.

MATTR: the moving-average fix

The moving-average type-token ratio (MATTR), introduced by Covington and McFall in the Journal of Quantitative Linguistics in 2010, removes the length problem rather than correcting for it after the fact. A window of fixed length slides through the text one word at a time, the TTR of each window is computed, and the final score is the mean of all the window values.

Because every window is the same length, the score no longer depends on how long the whole text is, so MATTR values are directly comparable across texts of different sizes. The individual window scores can also be inspected to see how diversity rises and falls within a single text.

Worked example

Take this 20-word sentence: 'The cat chased the mouse round the garden until the mouse ran into the shed and the cat gave up.'

Tokens: 20. Types: the, cat, chased, mouse, round, garden, until, ran, into, shed, and, gave, up. That is 13 distinct words ('the' appears six times, 'cat' and 'mouse' twice each; 6 + 2 + 2 + 10 single occurrences = 20). TTR = 13 / 20 = 0.65, or 65%.

Even one ordinary sentence sits well below 100% because of repeated function words. The 65% figure is only meaningful against other 20-word samples; a longer passage would score lower purely because it is longer.

Sources

More terms

ScriptGrain · Why ScriptGrain · Sample voice profiles · The Journal · Support