parakeet-ctc-0.6b-all

parakeet-ctc-0.6b-all

nvidia/parakeet-ctc-0.6b fine-tuned on every Speech Accessibility Project recording available to us -- SAPC1 train, SAPC1 dev and SAPC2 train, deduplicated -- for recognition of dysarthric and otherwise disordered speech.

All three rows are the same evaluation on the same 17,582 utterances, greedy CTC decoding, no language model, both sides through the same normaliser. Against the stock model that is a 63% cut in word errors; against the SAPC2-only model, 1.27 points, which is what the extra SAPC1 data buys.

Read the number carefully

This figure is not comparable to the ones on the sibling model cards, and the reason is worth understanding before you use any of them together.

This model trains on SAPC1 dev in full. SAPC1 dev and SAPC2 dev share 28,253 recordings and 76 of SAPC2 dev's 124 speakers, so neither sapc1_dev nor the whole of sapc2_dev is a fair test set for this model. What remains fair is the 48 SAPC2 dev speakers who appear nowhere in training, and that is the set scored above: sapc2_dev filtered to those speakers, 17,582 of 47,929 records, 35.6 h.

That set is harder than SAPC2 dev as a whole, because the exclusion is not random. Every one of SAPC2 dev's 35 Parkinson's speakers also appears in SAPC1 dev, so the held-out set contains no Parkinson's speech at all -- and Parkinson's is by some distance the easiest cohort in this corpus, averaging 7.5% WER against 20-22% for Down Syndrome and Cerebral Palsy on the sibling model. So parakeet-ctc-0.6b-sapc2's published 11.51% on all of SAPC2 dev and this model's 12.08% here are not evidence that this model is worse. Scored head to head on this set, as above, it is better by 1.27 points.

Output text convention

This model writes numbers as words and emits upper-case, unpunctuated text.

audio:  "lower the temperature three degrees"
output: LOWER THE TEMPERATURE THREE DEGREES     # not "... 3 DEGREES"

The tokeniser has 1,025 tokens and no digit characters, so numerals cannot be produced. Training transcripts are verbalised before tokenisation: "2" becomes TWO, "$45" becomes FORTY FIVE DOLLARS, "1949" becomes NINETEEN FORTY NINE. If you score this model, apply the same verbalisation to your references, or numerals will dominate your error count.

Transcript markup

SAPC2 wraps the canonical interview prompt, and PII redactions, in square brackets. The prompt is shown to the speaker rather than read to them, so none of it is in the audio, and left in the labels it trains the model to transcribe a question nobody asked. Bracketed spans and the #ts / #dis provenance tags are stripped before normalisation. It affects 12.4% of SAPC2 records and essentially none of SAPC1's.

Training data

SAPC1 train + SAPC1 dev + SAPC2 train, deduplicated by audio filename, then filtered to 0.5-30 s.

SAPC2 re-releases most of SAPC1 rather than extending it: 182,575 of SAPC1's 218,900 train recordings reappear in SAPC2 under the same filename with identical duration and contributor. Concatenating the manifests would train on those twice and silently double the weight of the 483 speakers in both, so identity is taken from the audio filename and each recording is kept once. Where a recording appears in both releases the SAPC1 copy is kept, being the one without the bracketed markup; this is immaterial in practice, since after normalisation 182,574 of the 182,575 duplicated pairs are character-identical, the single exception being a typo SAPC2 corrected.

Held-out evaluation set, by aetiology:

Training hyperparameters

Inherited unchanged from parakeet-ctc-0.6b-sapc1, whose learning rate and effective batch came from a 20-cell grid; they were not re-searched here.

Per-device batch size is not a free choice. Parakeet normalises over it in 24 BatchNorm1d layers, so it behaves as a hyperparameter in its own right. SyncBatchNorm all-reduces the statistics across both processes, so 2 x 16 normalises over all 32 as a single device holding 32 would.

Dev WER was still falling on the final epoch, so this recipe has not saturated:

The trainer's metric reads slightly optimistic against a standalone pass: 11.69% there against the 12.08% reported above.

Results by speaker

The 12.08% aggregate is pooled over words and so is dominated by the speakers who talk most. Averaged with equal weight per speaker it is 14.96%:

20 of the 48 held-out speakers are under 10% WER; 4 are above 30%.

Any single number for this model is a poor description of what a given speaker should expect.

Usage

import torch
from transformers import AutoModelForCTC, AutoProcessor

model_id = "dys-asr/parakeet-ctc-0.6b-all"
processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForCTC.from_pretrained(model_id).eval()

# audio: 16 kHz mono float32
inputs = processor(audio, sampling_rate=16_000, return_tensors="pt")
with torch.inference_mode():
    logits = model(**inputs).logits
print(processor.batch_decode(logits.argmax(dim=-1)))

Greedy CTC decoding, no language model.

Limitations

• No fair comparison against the sibling models on their own test sets. Having trained on SAPC1 dev, this model has seen both. Only the 48-speaker held-out set above is fair, and it excludes Parkinson's speech entirely.
• Numerals are written as words, not digits.
• No punctuation or casing.
• Accuracy is very uneven across speakers, 0.8% to 71.6% WER, tracking aetiology and severity.
• Not trained to convergence. Dev WER was still improving at epoch 10.
• Hyperparameters were inherited, not re-tuned for a corpus 1.8x the size of the one they were selected on.
• English only, 16 kHz mono.
• Single training seed. No variance estimate on the reported figures.