parakeet-ctc-0.6b-sapc2

parakeet-ctc-0.6b-sapc2

nvidia/parakeet-ctc-0.6b fine-tuned on the Speech Accessibility Project corpus 2 (SAPC2) train split for recognition of dysarthric and otherwise disordered speech.

On the SAPC2 dev split it removes about 53% of the generic model's word errors:

Both rows are the same evaluation: all 47,929 dev utterances (106.6 h, 451,437 reference words), greedy CTC decoding, no language model, no duration cap on scoring, both sides through the same normaliser.

An earlier release of this model reported 13.46% WER, and that figure should be discarded. It was scored against references that still contained the bracketed interview prompt, which the audio does not contain, and that release had been trained on the same inflated labels -- so it was rewarded for reproducing a question nobody asked. Scored against corrected references it measures 22.29%, and on Codabench Track 1 Test1 it scored 32.08 WER while the stock nvidia/parakeet-ctc-0.6b it was fine-tuned from scored 17.45. This release fixes the labels. See Transcript markup.

This is not comparable to parakeet-ctc-0.6b-sapc1's 10.32% WER. That figure is on the SAPC1 dev split, a different and easier set of speakers. See Relationship to the SAPC1 model.

Output text convention

This model writes numbers as words and emits upper-case, unpunctuated text.

audio:  "lower the temperature three degrees"
output: LOWER THE TEMPERATURE THREE DEGREES     # not "... 3 DEGREES"

The tokeniser has 1,025 tokens and no digit characters, so numerals cannot be produced. Training transcripts are therefore verbalised before tokenisation: "2" becomes TWO, "$45" becomes FORTY FIVE DOLLARS, "1949" becomes NINETEEN FORTY NINE, and dash-joined digit groups are read digit by digit (053-621 becomes OH FIVE THREE DASH SIX TWO ONE).

This matters for scoring. An earlier model in this series was trained on labels that kept numerals, which the tokeniser mapped to <unk>; it emitted <unk> wherever a number belonged and could never produce one. Fixing that was worth 3.29 WER points. If you score this model, apply the same verbalisation to your references, or numerals will dominate your error count.

Transcript markup

SAPC2 wraps the canonical interview prompt, and PII redactions, in square brackets. The prompt was shown to the speaker, not read to them, so none of it is in the audio. Left in the labels it trains the model to transcribe a question nobody asked. It affects 41,553 train records -- 230.3 h, 31% of the corpus by duration -- and inflates the labels 11.6% in words.

The first release of this model was trained that way. Bracketed spans and the #ts / #dis provenance tags are now stripped before normalisation, which is what this release retrains on.

The evidence is a cross-check of the two corpus releases rather than inspection: SAPC2 re-releases 182,575 of SAPC1's train recordings and 28,253 of its dev recordings under the same filenames and identical durations, and for 3,589 of the 3,601 re-released dev records carrying a bracket, SAPC2's text with the bracket removed is character-identical to SAPC1's. The clinching case reads [Talk about some of your hobbies.] while the speaker audibly says "Tell us about some of your hobbies" -- the bracket does not match the audio it precedes, because it is the prompt as written rather than as read. SAPC1 has 15 bracketed records out of 218,898 and is unaffected.

Training data

SAPC2 train split, filtered to 0.5-30 s to match the SAPC1 recipe: 327,168 of 336,075 utterances, dropping 8,903 over 30 s and 4 that normalise to empty.

Speakers by aetiology:

No speaker appears in both splits (verified: zero speaker-id overlap). SAPC2 is noticeably better balanced than SAPC1, whose dev split is 48% Parkinson's against 28% here, so the aggregate is less dominated by one aetiology.

What the 30 s cap costs

The first release trained on the corpus uncapped, on the argument that a 30 s cap discards 121.5 h — 16.4% of the audio — in only 2.6% of the utterances, because SAPC2's long tail is read passages rather than short prompts. Durations are heavily skewed: median 5.6 s, 99th percentile 47 s, maximum 180 s. Gradient checkpointing made the uncapped run possible, since a full batch of 32 at 181 s needs 48.8 GiB against the 59 GiB that 60 s reaches without it.

This release caps at 30 s anyway, so that the SAPC1 and SAPC2 models differ in their corpus and not in their recipe.

Two things therefore changed at once, and the improvement over the first release is not attributed to either alone: the labels were corrected, and 16.4% of the audio was dropped. The label fix is very likely the larger part -- CER falls from 17.30% to 6.85% against corrected references, which is the signature of removing text the model should never have emitted rather than of a better acoustic model -- but the two have not been separated by an ablation.

Training hyperparameters

The learning rate and effective batch size are inherited from the 20-cell grid tuned for parakeet-ctc-0.6b-sapc1; they were not re-searched on SAPC2.

Per-device batch size is not a free choice. Parakeet's Conformer normalises over the per-device batch in 24 BatchNorm1d layers, so it behaves as a hyperparameter in its own right: holding the learning rate and effective batch fixed and changing only per-device batch was worth about a WER point. The SAPC1 model got its statistics from a single GPU holding all 32 examples. Splitting across two GPUs would ordinarily drop that to 16 and change the model, so SyncBatchNorm is applied to all 24 layers, which all-reduces the statistics and restores them to the same 32 examples. The run logs this directly:

sync_batch_norm converted=24 processes=2 per_device_batch=16 statistics_over=32

Dev WER was still falling on the final epoch (11.43% to 11.33% on the trainer's own metric), so this recipe has not saturated and more epochs would likely gain further.

The trainer's metric reads slightly optimistic against a standalone pass: it reports 11.33% where the full evaluation above gives 11.51%.

Results by speaker

The 11.51% aggregate is pooled over words, so it is dominated by the speakers who talk most. Scored per speaker and averaged with equal weight the figure is 13.84%, and the spread behind it is very wide:

65 of 124 dev speakers are under 10% WER; 12 are above 30%. Broken down by aetiology, averaging over speakers:

Any single number for this model is therefore a poor description of what a given speaker should expect.

Relationship to the SAPC1 model

SAPC2 re-releases and extends SAPC1, so the two models are related, and the overlaps matter if you intend to compare them.

The last two rows are the ones that make a comparison legitimate: neither model was trained on any SAPC2 dev speaker, so SAPC2 dev is a fair test set for both. The first row is why they are not independent models.

Do not compare this model's 11.51% against the SAPC1 model's published 10.32%. Those are different test sets. Score both on the same manifest.

Usage

import torch
from transformers import AutoModelForCTC, AutoProcessor

model_id = "dys-asr/parakeet-ctc-0.6b-sapc2"
processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForCTC.from_pretrained(model_id).eval()

# audio: 16 kHz mono float32
inputs = processor(audio, sampling_rate=16_000, return_tensors="pt")
with torch.inference_mode():
    logits = model(**inputs).logits
print(processor.batch_decode(logits.argmax(dim=-1)))

Greedy CTC decoding, no language model.

Limitations

• Numerals are written as words, not digits. See the convention above.
• No punctuation or casing. Output is upper-case and unpunctuated.
• Accuracy is very uneven across speakers, from 1.2% to 71.5% WER, and tracks aetiology and severity: Down Syndrome and Cerebral Palsy speakers average roughly three times the error of Parkinson's speakers.
• Not trained to convergence. Dev WER was still improving at epoch 10.
• Hyperparameters were inherited, not re-tuned for this corpus, which is 1.5x the size of the one they were selected on after the 30 s cap.
• The gain over the first release mixes two changes, corrected labels and the 30 s cap, which have not been separated.
• English only, 16 kHz mono.
• Single training seed. No variance estimate on the reported figures.