Did the AI tells reach medicine? 1.4 million PubMed abstracts, 2015 to 2026

SGR-024, conducted 2026-10-29. Analyst: Jack Stovell, founder, ScriptGrain.

Question: Did the words that moved in arXiv abstracts after ChatGPT (old tells such as "delve" and "pivotal", then a different set from 2025) also move in PubMed abstracts, and how does that compare with what Kobak and colleagues found for 2024?

Summary

Background. SGR-008 found that in arXiv abstracts the words once called signs of ChatGPT, such as "delve" and "pivotal", rose in 2024 and then fell, while a different set of words rose in 2025 and 2026. Kobak and colleagues had already shown excess vocabulary in 2024 PubMed abstracts, and later work estimates that a large majority of recent biomedical papers carry language-model traces. This study asks whether the arXiv pattern also holds in medicine and biology, and checks our method against theirs.

What we did. We fixed six hypotheses and two checks against Kobak et al. in public before the full measurement. We took a fixed 10% sample of PubMed records by a hash of the PubMed identifier, kept 1,383,539 English journal-article abstracts of at least 50 words entered from January 2015 to September 2026, and measured the same three word sets as SGR-008 (twelve old tells, twelve new words, five control words), the share of abstracts containing any word of each set. We then selected excess words in each year on odd months, scored them on even months, labelled each as style or content by fixed rules, and computed a lower bound on the share of abstracts affected. Confidence intervals come from resampling 13,114 journals.

What we found. Old tells: 4.73% of abstracts in 2020 to November 2022, 11.55% in 2024 and 8.22% in 2026. New words: 1.33%, 1.81% and 5.08%. The hypothesis that the old tells rose and then fell (H1) was supported; that the new words rose, doubling in at least six of eight subject groups (H2), supported; and that the new words overtook the old in 2025 or 2026 (H3), not supported. In 2026H2 (July to September 2026) the old tells were in 6.71% of abstracts and the new words in 5.61%. Unlike in arXiv, where the old tells fell below their pre-ChatGPT level, in PubMed they were still above it in 2026. So SGR-008 partly replicates in medicine. Our lower bound for 2024 from the 2024 style words was 29.83% (26.44 to 32.70), above the range we set around Kobak et al.'s 13.5% estimate (our check C1 was not supported), and the same method gives bounds of up to 8.72% in placebo years before ChatGPT.

What it means. A shift in the words used in abstracts is a shift in how abstracts read. It does not show who or what wrote them, and the lower bounds count only abstracts using the specific words we selected.

Key numbers

• Old-tell words (such as "delve" and "pivotal") were in 4.73% of PubMed abstracts entered in 2020 to November 2022, 11.55% in 2024 and 8.22% in 2026 (ScriptGrain, 2026; 1,383,539 abstracts).
• The twelve new words were in 1.33% of abstracts before ChatGPT, 1.81% in 2024 and 5.08% in 2026 (3.82 times the baseline; in arXiv the same ratio was 6.53).
• At least 30.8% of 2024 abstracts and 67.8% of 2026 abstracts contain a selected style word beyond the share the 2021 and 2022 trend projects (lower bounds; 95% intervals 26.1 to 35.2 and 63.5 to 71.3; the same method gives up to 8.72% in placebo years before ChatGPT).
• In 8 of 8 subject groups the share of abstracts with a new word in 2026 was at least twice its 2020 to 2022 level (3 of the groups are small, with wide intervals).

Sample

A fixed 10% sample of PubMed records (those whose SHA-1 of "sgr-024:" plus the PubMed identifier falls in the lowest tenth), read from the NCBI annual baseline and daily update files, latest version of each record, with 1,383,539 abstracts from 13,114 journals meeting the rules below. Entry dates run from January 2015 to September 2026.

• Entered PubMed from 1 January 2015 to 30 September 2026
• Has an abstract of at least 50 words, with section labels such as BACKGROUND removed
• Language includes English, publication type Journal Article, and the text looks English (at least 5% common English words)
• Retracted papers are kept in the main analysis and dropped in a sensitivity analysis

Method

Pre-registration: the six hypotheses, two checks against Kobak et al., word sets, thresholds and analysis were committed to the public repository before the full measurement (docs/research/sgr-024-preregistration.md). The script was tested on a 4% pilot; the pilot results are not reported.

Sampling: a fixed hash of the PubMed identifier picks 10% of records, so the sample is the same whenever it is drawn. Records come from NCBI's annual baseline and daily update files; the latest version of each record is used and deleted records are dropped.

Measures: tokens are lowercase runs of letters and hyphens of three or more characters. A set's share is the percentage of abstracts with at least one of its words. No language model is used.

Hypotheses: H1 old tells rise from 2020 to 2022 to 2024 then fall by 2026; H2 new words rise, at least doubling in six of eight subject groups; H3 the new words overtake the old in a half-year from 2025H1 to 2026H2. All are judged on 95% cluster-bootstrap intervals over journals.

Excess words: for each year, a word's expected share is its 2022 share plus the 2021 to 2022 change carried forward. Words are selected on odd entry months and scored on even months. Selected words are labelled style or content by fixed rules (Kobak et al.'s published labels first, then SGR-008's, then an AI-topic rule, then breadth across subject groups); no language model labels anything.

Subject groups: each journal is assigned a group from MeSH major topics of its records entered before December 2022.

Variables measured

• Share of abstracts containing a word of each set, by month, half-year and year
• Excess-word statistics and style or content label by word and year
• Sentence length and other measured text features
• Journal subject group

What each measure means

Old tells

Twelve words fixed in SGR-008 before any data: innovative, utilized, advancements, pivotal, facilitates, firstly, tackle, showcasing, underscores, delve, delves, intricate. The share is the percentage of abstracts containing at least one of them.

New words

Twelve words fixed in SGR-008 from a separate sample: underexplored, grounded, aligns, synthesizes, replaces, leaving, recovers, expose, structurally, reproducible, treats, vanishes. Same measure.

Control words

Five common words, however, results, using, show and method, to check that the sample did not drift.

Entry date

The date PubMed first received the record (the entrez date). Months are counted from it; publication date is a sensitivity analysis.

Excess words and the lower bound

As in Kobak et al. and SGR-008: a word is excess in a year if its share exceeds the share projected from 2021 and 2022. Words are selected on odd months and scored on even months. The lower bound is the share of abstracts containing at least one selected style word, beyond the share expected without a rise.

95% interval

From 2,000 resamples that draw whole journals with replacement (seed 20261029).

Findings

• H1 (supported): the old tells rose, then fell (2024 minus 2020 to 2022: +6.82 points (95% interval 6.36 to 7.29); 2026 minus 2024: −3.33 points (−3.76 to −2.91)): Both parts must hold: the lower end of the first interval above zero and the upper end of the second below zero. In arXiv (SGR-008) the share went from 4.54% to 11.85% to 3.03%, below its starting level; in PubMed the 2026 share was 1.74 times the 2020 to 2022 level (1.67 to 1.81).
• H2 (supported): the new words rose in most subject groups (2026 minus 2024: +3.27 points (3.08 to 3.47); 2026 minus baseline: +3.75 points (3.57 to 3.95); 8 of 8 groups at least doubled): Supported if both pooled intervals are above zero and at least six of eight subject groups have a 2026 to baseline ratio of at least 2. Three of the eight groups are small (Cancer: 12 journals; Heart, blood vessels and metabolism: 21 journals; Infection and immunity: 17 journals), so their ratios are very uncertain and the 95% intervals reach zero. The rule was set on the point ratio, as pre-registered; leaving the small groups aside, 5 of the 5 groups with 30 or more journals at least doubled, fewer than the six the rule asks for, so the group condition rests partly on the small groups.
• H3 (not supported): the new words overtook the old (No half-year from 2025H1 to 2026H2 from which the new words stay above the old): Supported if, from some half-year in 2025 or 2026, the new-word share is above the old-tell share in every later half-year. In 2026H2 (July to September 2026) the old tells were in 6.71% of abstracts and the new words in 5.61%. In arXiv the crossover was 2025H2.
• H4 (supported): sentence length did not change (Mean sentence length 22.74 words before ChatGPT, 22.25 in 2026; difference −0.49 (90% interval −0.57 to −0.42)): Supported if the 90% interval lies within one word either side of zero. A change in words without a change in sentence shape was the SGR-008 pattern.
• H5 (supported): the style words changed between 2024 and 2026 (Jaccard overlap of the two top-50 style lists: 0.299 (50 and 50 words); shared: address, challenges, comprehensive, critical, demonstrates, despite, distinct, diverse, emerged, enhance, highlight, highlighting, highlights, insights, notably, offer, offering, offers, particularly, primarily, providing, substantial, thereby): Supported if the overlap is below 0.5. In arXiv it was 0.064.
• H6 (supported): our excess words match Kobak et al.'s (45 of our top 50 raw 2024 excess words are in their published list of 900): Supported if at least 30 of 50 are. This tests whether our selection finds the same signal in a different sample.
• C1 (not supported): our 2024 lower bound sits near Kobak et al.'s (29.83% (26.44 to 32.70) against their estimate of 13.5%): Supported if the interval's lower end is above zero and the point estimate is between 6.75% and 27%. Our 2024 bound came out above the window, not below it. The placebo runs on years before ChatGPT give bounds of up to 8.72%, so part of a bound from this method can come from ordinary drift in word use rather than from anything new.
• C2 (supported): our sample matches their counts (185 of 185 word-years within 0.8 to 1.4 times their share (100%); median ratio 1.01): Compares the share of abstracts containing each of the 29 fixed words with Kobak et al.'s full-corpus counts, 2015 to 2024. Supported if at least 80% fall in the window.

What this study does not show

• That any abstract was written by AI. A word count describes vocabulary, not authorship.
• That the words moved because of AI tools. The data are observational and many other things changed after 2022.
• That the share of abstracts using a word set is the share of abstracts written with AI: the lower bound counts abstracts with specific words and misses AI-assisted text that avoids them.
• Anything about full text, non-English abstracts or journals outside PubMed.
• That voice measurement, ScriptGrain's or anyone's, can tell whether a text was written by AI. Nothing in this study tests that.

Two word sets by half-year

Lower bound on abstracts touched by style words

Pre-registered hypotheses and results

PubMed and arXiv side by side

Percentage of abstracts containing at least one word of each set. The arXiv column is SGR-008's published result; the two corpora differ in field, length and who writes in them, so the comparison is of pattern, not of level.

By subject group

Journals are grouped by the MeSH major topics of their records entered before December 2022; journals without a clear majority are in General and those with too few records are Unclassified. Ratio is the new-word share in 2026 over the share in 2020 to November 2022.

Three of the eight groups are small (Cancer: 12 journals; Heart, blood vessels and metabolism: 21 journals; Infection and immunity: 17 journals), so their ratios are very uncertain and the 95% intervals reach zero. The rule was set on the point ratio, as pre-registered; leaving the small groups aside, 5 of the 5 groups with 30 or more journals at least doubled, fewer than the six the rule asks for, so the group condition rests partly on the small groups.

Lower bounds and the placebo

Lower bound on the percentage of abstracts using at least one selected style word, beyond what the 2021 and 2022 trend projects, with 95% intervals over journals. The placebo rows apply the same method to years before ChatGPT, using a 2016 to 2017 trend; a bound near zero there is the method behaving.

Sensitivity analyses

The same set shares under different choices. Each row repeats the main analysis with one change; intervals are from 1,000 or fewer journal resamples.

With the baseline moved to 2018 and 2019 (before COVID-19), the old tells were in 4.02% of abstracts (3.89 to 4.17) and the new words in 1.42% (1.35 to 1.50), close to the main baseline. 30.6% of the sampled records entered before December 2022 were last revised after November 2022; PubMed revisions are usually to metadata, so no record was excluded for it. The within-journal row uses 636 journals.

Sentence shape and other text features

Means over abstracts for the measured features, with the 2026 minus baseline difference and its 95% interval.

How abstracts and journals changed

Limitations

• This is a 10% hash sample of PubMed, not every record. It reproduces Kobak et al.'s published word frequencies only as closely as check C2 reports, and rare words are noisy.
• PubMed abstracts are the latest version NCBI holds on the day of collection. A record revised after it was first published can show later wording in an earlier month. We report how many baseline records were revised after November 2022 in the summary.
• Entry date is when PubMed received the record, not when the paper was written; the main analysis uses it, and publication date is a sensitivity analysis.
• The lower bound counts only abstracts that contain a selected style word. AI-assisted text without them is missed, and human text with them is counted. The bound is a floor on a quantity we cannot observe, not an estimate of AI use.
• Style or content labels come from fixed rules, not a person or a model reading each word. Words that rose because a topic rose can be labelled style, and the reverse. The label source is published for every word.
• Subject groups come from the MeSH topics of each journal's earlier records and are coarse. General and Unclassified journals are outside the eight-group test.
• The old and new word sets were fixed from arXiv. They may not be the words most affected in medicine, which is what the excess-word analysis looks at.
• A change in vocabulary after 2022 is not evidence that any abstract was written by AI, and its absence would not be evidence of the reverse.

Competing interests

ScriptGrain sells writing-voice measurement and the analyst builds it; a story that medicine is full of machine-written text could suit the product. The hypotheses, word sets, thresholds and pass and fail rules were committed to the public repository before the full measurement, every hypothesis is reported whichever way it came out, and the data and scripts are public.

Reproducing this

• Pre-registration, collection script, run script and entry builder: docs/research/sgr-024-preregistration.md and scripts/experiments/sgr-024/ in the ScriptGrain repository.
• The sample is defined by a fixed hash rule on the PubMed identifier, so anyone can redraw it from NCBI's baseline and update files.

Data

• Word set shares by half-year and subject group (CSV) (csv)
• Word set counts by month and subject group (CSV) (csv)
• Word set shares by year and subject group (CSV) (csv)
• Every candidate and frozen word, share of abstracts by year, with labels (CSV) (csv)
• Excess-word statistics, selection and scoring halves (CSV) (csv)
• Journal subject groups (CSV) (csv)
• Sampling check against Kobak et al.'s counts (CSV) (csv)
• Every result, including intervals, bounds and placebo (JSON) (json)

Released under CC BY 4.0: free to reuse, including commercially, with credit to ScriptGrain and a link to this page.

Citations

• Kobak, Gonzalez-Marquez, Horvat and Lause. Delving into LLM-assisted writing in biomedical publications through excess vocabulary (Science Advances 11(27), eadt3813, 2025; arXiv 2406.07016)
• Kobak et al. llm-excess-vocab: excess word lists and yearly counts (MIT licence)
• Kousha and Thelwall. How much are LLMs changing the language of academic papers after ChatGPT? A multi-database and full text analysis (arXiv 2509.09596)
• Siler. The diffusion of large language models in published academic articles (PNAS, 2026; doi 10.1073/pnas.2605754123)
• Holzwarth, Gonzalez-Marquez and Kobak. Most biomedical publications show signs of LLM-assisted writing (arXiv 2608.10715)
• Saner et al. The evolution of ChatGPT-generated text in scientific medical publications (Saudi Journal of Anaesthesia 20, 2026; doi 10.4103/sja.sja_526_25)
• Liang et al. Quantifying large language model usage in scientific papers (Nature Human Behaviour, 2025; doi 10.1038/s41562-025-02273-8)
• NCBI. PubMed baseline and update files
• ScriptGrain SGR-008, The AI tells moved: 2.1 million arXiv abstracts, 2015 to 2026

How to cite

ScriptGrain (2026). Did the AI tells reach medicine? 1.4 million PubMed abstracts, 2015 to 2026 (Study SGR-024, conducted 29 October 2026). Dataset licensed CC BY 4.0. Derived from PubMed records (U.S. National Library of Medicine, public domain; individual abstracts remain the property of their publishers). https://scriptgrain.com/research/did-the-ai-tells-reach-medicine

About the analyst

Jack Stovell has worked in finance and data for more than twelve years, building management reporting, forecasting and profitability models for advertising agencies and tech scale-ups, from SQL and Power BI reporting to board-level analysis. Since 2016 he has run Adapt Progress Evolve, an applied AI studio, where he builds and operates AI systems and data products: ScriptGrain's measurement of writing voice across 45 attributes, UK Spend, which brings 16.7 million rows of UK council spending into one queryable dataset, and more than thirty AI agents running in production. He designs ScriptGrain's studies and is accountable for every number in them.

• jstov.uk
• LinkedIn
• GitHub

More research

• What reading age do UK brands write for? Readability of 115 brands' blogs and press releases, 2015 to 2026
• Same Brief, Four Voices
• ChatGPT vs a measured voice