ChatGPT vs a measured voice
SGR-002, conducted 2026-09-20.
Question: When GPT-5 writes from the same brief three ways (bare, with a one-page measured prompt, and through ScriptGrain's profile), how close does each get to the author's own writing?
Sample
Five posts (5,233 words) from one author's company blog, already the samples of an existing ScriptGrain profile. The two longest (1,394 and 1,392 words) were held out; a fresh study profile was extracted from the remaining three (2,447 words). Two briefs, three arms, six drafts.
- The author's own unassisted writing (adaptprogressevolve.com posts)
- Held-out pieces never shown to any arm; briefs derived from them by a separate model without quoting
- Same brief, byte for byte, to every arm; one run per arm, published unedited
Method
Study profile: api-v1 POST /v1/profiles from the three training posts (one extraction slot), status polled to complete.
Briefs: gpt-5-mini read each held-out post and wrote the brief a writer would have received (topic, angle, audience, three points, target length), instructed not to quote it.
Arm one, bare: GPT-5 (reasoning effort minimal) with the brief and a target length only.
Arm two, prompt: GPT-5 with a system prompt from ScriptGrain's free prompt builder, measured from the three training posts (numbers for sentence length and variation, contractions, punctuation, pronoun mix; judged formality, humour, argument structure; the hard rules against AI tells).
Arm three, ScriptGrain: api-v1 POST /v1/generations against the study profile, content type article, the brief's target length.
Measures per draft: voice match against the study profile (POST /v1/voice-match, feature-delta engine), pairwise match against the held-out post (POST /v1/compare-voice), AI-tell density and em dashes (the public cliché checker's scanner), share of sentences under eight words, and the AI detector's human score (POST /v1/ai-detect). Runner: scripts/experiments/run-study.mjs in the ScriptGrain repository.
Variables measured
- Voice match against the study profile (0 to 1, profile-calibrated)
- Pairwise match against the held-out post (0 to 1, pairwise-calibrated)
- AI-tell phrases and sentence shapes per 1,000 words
- Em dashes per draft
- Share of sentences under eight words
- Sentence-length variation (standard deviation over mean)
- Detector human score (0 to 1)
- Draft length against the brief's target
Findings
- The measured profile got closest to the author (0.83 vs 0.74): Mean voice match against the study profile was 0.83 for ScriptGrain's drafts, 0.74 for bare GPT-5 and 0.62 for GPT-5 with the one-page prompt. Against the held-out post itself the order was the same: 0.91, 0.82, 0.80.
- A one-page prompt was not enough for GPT-5 (0.62): The prompt asked for about 14 contractions per 1,000 words and a formality of 3.7 on a 0 to 10 scale; GPT-5 produced 0.5 contractions per 1,000 and a judged formality of 7.2 on both briefs, more formal than its bare drafts. It did obey the hard rules (zero em dashes, more short sentences, higher sentence variation), so the prompt shaped the surface and missed the register. Numbers in a prompt help; a scorer that sends the draft back is what closes the gap.
- Bare GPT-5 ignored the length and reached for em dashes (2309 words, 17.5 em dashes): Asked for about 1,393 words, bare GPT-5 wrote 2309 on average with 17.5 em dashes per draft. ScriptGrain's drafts averaged 1302 words with none; the prompt arm also removed every em dash.
- ScriptGrain's own drafts still produced 'not X but Y' (2.3 per 1,000): The checker counted more AI-tell constructions in ScriptGrain's drafts (mean 2.3 per 1,000 words; 3.8 on the first brief, three 'not X, not Y, but Z' sentences) than in bare GPT-5's (0.2). The generation prompt bans the construction and the model wrote it anyway. That is a product finding, logged for the generator: the same scanner should run on every draft before it is returned.
- Detector scores did not track voice (0.78 / 0.67 / 0.52): The AI detector read the prompt arm as most human (0.78), then ScriptGrain (0.67), then bare GPT-5 (0.52, with one draft at 0.32). Sounding human and sounding like the author are different measurements; the study reports both.
Results by arm (means over two briefs)
| Arm | Voice match vs profile | Match vs held-out piece | AI-tells per 1,000 | Em dashes per draft | Sentences under 8 words | Detector human score | Words (target ~1,393) |
|---|---|---|---|---|---|---|---|
| GPT-5, brief only | 0.74 | 0.82 | 0.2 | 17.5 | 37% | 0.52 | 2309 |
| GPT-5 with the prompt-builder prompt | 0.62 | 0.80 | 1.6 | 0.0 | 54% | 0.78 | 2294 |
| ScriptGrain (measured profile + generation) | 0.83 | 0.91 | 2.3 | 0.0 | 38% | 0.67 | 1302 |
Results by brief
| Brief | Arm | Voice match | Match vs held-out | AI-tells per 1,000 | Words |
|---|---|---|---|---|---|
| Brief 1 | GPT-5 bare | 0.74 | 0.82 | 0.0 | 2287 |
| Brief 1 | GPT-5 + prompt | 0.55 | 0.74 | 1.4 | 2205 |
| Brief 1 | ScriptGrain | 0.87 | 0.90 | 3.8 | 1312 |
| Brief 2 | GPT-5 bare | 0.75 | 0.82 | 0.4 | 2331 |
| Brief 2 | GPT-5 + prompt | 0.70 | 0.86 | 1.7 | 2383 |
| Brief 2 | ScriptGrain | 0.80 | 0.92 | 0.8 | 1292 |
What the numbers say
The measured profile moved the drafts towards the author on every voice measure, by about 0.09 on both the profile score and the held-out comparison, and it hit the length. It also carried the study's least flattering number: the checker found 'not X but Y' constructions in ScriptGrain's drafts that bare GPT-5 did not produce. The generator forbids that shape and the model wrote it anyway, which means the ban has to be checked, not only stated.
The prompt-builder arm is the surprise. The prompt carried the author's numbers and GPT-5 honoured the ones that are easy to count (no em dashes, short sentences, burstiness) while drifting away on register: fewer contractions than either other arm and a judged formality of 7.2 against a target of 3.7. A prompt can describe the target; it cannot check the draft against it. That is the difference a scorer makes, and it is why the product scores every draft rather than trusting the instruction.
Two briefs and one author make this a demonstration of direction, not a statistic. The runner is a script, the drafts are in the data file, and every number can be reproduced with the free tools on this site.
Limitations
- Two briefs, one author, one run per arm. Effect direction and rough size only; no significance test.
- The study profile was built from 2,447 words across three posts, below the 3,000 words the product recommends, so both the profile and the prompt were thinner than a real user's would be.
- No author-written custom-instructions arm; the author had not written any. It is the first addition for the next run.
- Briefs were derived from the held-out posts by a model, so they may carry more of the piece's structure than a human brief would.
- The author of the held-out posts also builds the product. Nothing was blind, and no draft was chosen or edited after the run.
- The AI-tell scanner is ScriptGrain's own instrument and the reference bands come from its reference corpus; another checker would count differently.
Reproducing this
- Data file: /research/chatgpt-vs-a-measured-voice.json (all six drafts, every measurement, the system prompt used in arm two).
- Runner: scripts/experiments/run-study.mjs (public repository).
- Free instruments: the off-voice check, the cliché checker and the prompt builder on this site produce the same numbers for any two pieces you paste.