Who Said What, and When: YouTube Auto-Captions vs Transcribe.so on Speakers and Timing (2026)
Do YouTube auto captions do speaker identification? No: their format carries no speaker labels. On 30 multi-speaker videos measured on 2026-10-26, transcribe.so attributed 91.9% of reference words to the right person (95% CI 88.7% to 94.7%) on the ten whose captions name speakers. Our weakest result: within one second we found fewer than half of the speaker changes captioners marked. Timing was close to a tie. Disclosure: we sell transcribe.so, one of the two systems compared.
The three findings, each against the creator's own caption track as the reference:
| What we measured | YouTube auto-captions | transcribe.so |
|---|---|---|
| Speaker labels on words | Not available (the caption format carries no speaker labels) | 100% of words labelled |
| Speaker-change detection, F1 at a one-second window | Not available | 43.4% (32.7% to 53.8%), 30 videos |
| Who said what: words attributed to the right speaker | Not available | 91.9% (88.7% to 94.7%), 10 videos |
| Timing: share of caption lines where this system is closer to the creator's timestamp | 40.7% | 56.2% (ties make up the rest), 56 videos |
| Timing: 90th-percentile spread around each video's typical offset | 1.15 s | 1.28 s (YouTube is tighter here) |
| Word error rate on the multi-speaker videos | 30.2% (24.8% to 35.6%) | 15% or higher (shown as a band by policy) |
Our word error rate is shown only as a band (under 5%, 5-15%, 15% or higher) because we never publish exact word-accuracy digits for our own pipeline: exact figures can fingerprint the speech-recognition vendors we use. Speaker and timing numbers are exact. YouTube's word error rate on these videos falls inside our band, so this table does not tell you which of the two made fewer word errors here. For word accuracy on single-speaker talks, see our YouTube auto-caption accuracy study, which measures YouTube alone.
Do YouTube auto-captions identify speakers?
No. YouTube's automatic captions are a stream of words with timestamps and nothing else: there is no field for a speaker, so the captions cannot tell you who is talking. That is a property of the format, not something we measured, and it is why the YouTube column reads "not available" for every speaker metric instead of zero. Some creators mark speaker changes in their own uploaded captions, with a ">>" at the start of a line, a leading dash, or a name such as "HOST:", and those creator tracks are what we used as the reference.
transcribe.so attached a speaker label to all of the words it returned on the 30 multi-speaker videos, on captions and transcripts produced on 2026-10-26. None of the 30 transcripts came back without speaker labels. A label being present does not mean it is right, which is what the next two sections measure.
How well does transcribe.so detect speaker changes?
At a one-second window, transcribe.so's speaker changes matched the creator-marked changes with an F1 of 43.4% (95% CI 32.7% to 53.8%) across 30 videos in five languages, measured on 2026-10-26: 41.9% of its changes landed on a marked change and it found 45.0% of the marked changes. With a two-second window the F1 was 55.9%, and with half a second it was 32.0%.
| Language | Videos | Marked changes | F1 at one second |
|---|---|---|---|
| English | 9 | 165 | 51.9% |
| French | 7 | 275 | 49.4% |
| German | 3 | 62 | 50.3% |
| Korean | 8 | 332 | 40.5% |
| Portuguese | 3 | 121 | 9.6% |
| All five languages | 30 | 955 | 43.4% |
This is the result where we look weakest, and three things about the reference pull the number down. Creators do not mark every change: when we read a blinded sample of caption boundaries, about one in eight unmarked boundaries looked like a change of speaker, so the references miss many real changes and our measured precision is a lower bound. Short interjections ("Yes.", "Mm-hm") are often left unmarked too. And one Portuguese video with 106 dash-marked changes, in which transcribe.so heard a single speaker throughout, accounts for most of Portuguese's low score. Even so, roughly half of the changes a careful captioner marked were not found within a second, and some of the detected changes were real errors; that is the honest reading.
How accurate is "who said what"?
On the ten videos whose captions name each speaker (nine in English, one in German), transcribe.so attributed 91.9% of the reference words to the correct speaker (95% CI 88.7% to 94.7%), on transcripts produced on 2026-10-26. Words our transcript missed count against it. Counting only the reference words our transcript aligned to, the share was 94.2%. English alone was 93.0% across nine videos; the single German video, a panel with four named speakers, was 77.0%.
A stricter measure, concatenated minimum-permutation word error rate (cpWER), scores speaker labels and word errors together. Ours was 15% or higher on these videos, shown as a band for the same reason as our word error rate; it includes every word error, so it cannot be lower than the word error rate itself.
Are YouTube or transcribe.so timestamps more accurate?
Neither, by a margin that matters. Across 56 videos, measured on 2026-10-26, transcribe.so's first word of a caption line was closer to the creator's timestamp on 56.2% of lines and YouTube's on 40.7%, but the median difference between the two was 0.03 seconds. On the single-speaker talks it was an even split (50.2% against 46.8%, median difference 0.001 seconds). On the multi-speaker videos transcribe.so was closer on 61.4% of lines, while YouTube had the tighter tail: 90% of its lines fell within 1.62 seconds of that video's typical offset against 1.75 seconds for ours. The intervals overlap on the tail measure, so we read timing as a draw with a small edge each way.
| Timing measure | Talks (29 videos) | Multi-speaker (27 videos) |
|---|---|---|
| Lines where transcribe.so is closer | 50.2% | 61.4% |
| Lines where YouTube is closer | 46.8% | 35.6% |
| Spread, YouTube (90th percentile) | 0.53 s | 1.62 s |
| Spread, transcribe.so (90th percentile) | 0.53 s | 1.75 s |
The reference here is coarse. Creator caption timestamps start a line, often a little early, and live broadcast captions can run several seconds behind the speech, so we compare each system's offset against the same reference rather than treating any one number as true timing. Nine videos whose creator timestamps matched YouTube's own word timings almost exactly (auto-synced captions) were left out of the timing comparison, as fixed in advance, because they would grade YouTube against itself.
How did we measure speaker labels and timing?
Every rule below was written down and committed before we ran transcribe.so on any video for this study (pre-registration commit 10a150cf744c4ccde1e0d5fb56d14bceb89744b6). One amendment was made afterwards, logged with its own commit: publishing these results as this companion post instead of rewriting the original study. No metric, threshold or video selection changed.
- Videos. Corpus B: public multi-speaker videos (interviews, panels, news, talk shows) between four and twenty minutes long, with a creator-uploaded caption track in the spoken language that marks speakers, and a YouTube automatic track. We searched YouTube with its own captions filter in eight languages and checked 975 videos; 368 had the right caption tracks, and most of those did not mark speakers. A video needed at least five marked speaker changes in its first ten minutes, at most two videos per channel. Spanish (two qualifying videos), Japanese and Hindi (none) fell below the three-video minimum and are excluded. The 30 videos are English (9), Korean (8), French (7), German (3) and Portuguese (3). Corpus A: the 38 talks and lectures from our word-accuracy study, used for timing only.
- Window. Every metric covers each video's first ten minutes.
- Reference speaker changes. Each video uses exactly one marker convention, chosen by reading its captions: ">>" at the start of a line (">>>", which marks a new story, does not count), a leading dash, or a "NAME:" prefix of one to three words that recurs in the video. Tags such as [Music] or [Applause] never count as a speaker. Two videos that mixed conventions and five with rolling, repeated caption lines were excluded at this step.
- Checking the extractor. Before freezing the rules we sampled 100 extracted changes and 100 caption boundaries without a marker, removed every marker, and judged from two lines of context on each side whether the speaker changed, without knowing which group a boundary came from. This was a text-only check, not a listening test, and it was judged by the same person who wrote the extraction rules, so it was blinded but not independent. 82.9% of the extracted changes read as real changes (18 were too ambiguous to call), and about one in eight unmarked boundaries also read as a change, which puts the references' estimated recall at 55.1%.
- Speaker-change detection. A transcribe.so change is the start of a segment whose speaker differs from the previous one. Marked and detected changes are paired one to one (Hungarian matching) within a window of 0.5, 1 or 2 seconds. Because creator timestamps are often offset by a fixed amount, each video's reference times are first shifted by that video's median offset between the creator's line starts and our word starts; we report unshifted results too (F1 39.0% at one second), and ten videos whose offset exceeded two seconds are flagged in the data.
- Who said what. On videos with "NAME:" labels, both transcripts are aligned word by word to the reference, each transcribe.so speaker is matched one to one to a reference speaker by word overlap, and we count reference words given the right speaker. A transcript with no speaker labels would have counted as one speaker (none occurred).
- Timing. For each caption line we take the first word each system got exactly right and compare its start time with the line's start time, for both systems against the same reference. Hindi is excluded from timing, as fixed in advance. Japanese is aligned by character.
- Word error rate. Same normalization as the original study (lowercasing, punctuation and bracketed tags removed, fillers removed, numbers not normalized). YouTube's figure is exact; ours is banded.
- Statistics. 95% intervals come from resampling whole videos within each language, 2,000 resamples, fixed seed.
- The transcribe.so runs. All 68 jobs ran on 2026-10-26 through our public API on a Pro-plan test account, with no settings changed; every job was served by our standard pipeline.
What are the limitations?
The biggest limitation is the reference. Creator captions mark speakers inconsistently, so speaker-change scores here are a comparison with a careful but incomplete human captioner, not with ground truth. The samples are small: the who-said-what figure rests on ten videos, nine of them in English and five of those C-SPAN-style interviews labelled "HOST:" and "GUEST:", and each language has only three to nine videos.
- Broadcast skew. Videos that mark speakers in captions are mostly news, public-affairs and TV shows. Several English references are live-stenography captions with typos and missing words, which raises every word error rate measured against them, YouTube's and ours alike.
- Coarse timing. Creator timestamps mark the start of a caption line, sometimes seconds after the speech in live captions, so timing results are relative: they say which system is closer to the same reference, not how close either is to the true moment a word was spoken.
- Not every language. Spanish, Japanese and Hindi had too few videos with speaker-marked captions and are not in the results.
- One point in time. Both systems change. These are results for captions and transcripts produced on 2026-10-26.
Where can I get the data?
The per-language, per-metric results are a CSV: youtube-caption-study-2026-10-v2.csv. The full results file, with the pre-registered rules, the extractor check, per-video scores and the amendment log, is here, and the 30 multi-speaker videos are listed by YouTube ID with their marker convention in the video list. Caption text is not republished. Our own word error rates appear in these files as bands only.
If you need a transcript that says who spoke and when, our free YouTube transcript tool transcribes the audio and labels speakers, which YouTube's own captions do not.
Frequently asked questions
Do YouTube auto-generated captions show who is speaking?
No. YouTube's automatic captions carry no speaker labels: the format has words and timestamps only. Some creators add ">>" or names to captions they upload themselves, but the automatic track never has them.
How accurate is transcribe.so at attributing words to speakers?
On ten videos with named speakers in their captions, measured on 2026-10-26, transcribe.so attributed 91.9% of reference words to the right speaker (95% CI 88.7% to 94.7%). English alone was 93.0%; one German panel was 77.0%.
How well does transcribe.so detect when the speaker changes?
Moderately. At a one-second window its F1 against creator-marked changes was 43.4% across 30 videos, rising to 55.9% at two seconds, measured on 2026-10-26. Creators leave many changes unmarked, so the true precision is higher than measured, but many marked changes were also missed.
Are YouTube's caption timestamps accurate?
About as accurate as ours. Against creator timestamps, transcribe.so was closer on 56.2% of caption lines and YouTube on 40.7%, but the median gap between them was 0.03 seconds, and YouTube had the tighter spread on multi-speaker videos.
Why are transcribe.so's word error rates shown as bands?
Exact word-accuracy figures for our pipeline can identify the speech-recognition vendors behind it, so we publish word error rate and cpWER only as under 5%, 5-15% or 15% or higher. Speaker and timing numbers are exact. On these videos YouTube's word error rate (30.2%) falls inside our band, so the bands do not show which system made fewer word errors.