parakeet-rnnt-0.6b-v2split-sampler

Parakeet RNN-T 0.6B — etiology-weighted sampling

Identical to dys-asr/parakeet-rnnt-0.6b-v2split-uniform in every respect but one: each training utterance is drawn with probability proportional to its etiology's share of a target distribution, rather than uniformly. The two runs exist to measure that one change.

It did not work. On the held-out split the weighted sampler is 0.13 CER points worse than uniform. This model is published because a negative result that is not published gets repeated.

The sampler

$$p_d \propto q_d \left(\frac{E_d}{\bar E}\right)^{0.5}, \qquad w_i = \frac{p_{d(i)}}{N_{d(i)}}$$

• $q_d$ — the etiology's share of the private test set by character mass (its speaker count times its characters per speaker). The metric pools characters, so a speaker prior would not align with its denominator; measured on this corpus, characters per speaker range from 11,222 for Stroke to 32,338 for Parkinson's, a factor of 2.9.
• $E_d$ — the median per-speaker CER the uniform run reaches on the same held-out split. Median rather than mean because every etiology's speaker CERs are strongly right-skewed: Down Syndrome averages 12.92% against a median of 8.45% because one of its seventeen speakers sits at 57.71%, and a mean would aim the sampler at a handful of outliers instead of at the group.
• $\beta = 0.5$ damps the difficulty term, so an etiology twice as hard is sampled $\sqrt2$ times more rather than twice as much.
• Dividing by $N_d$ turns a group share into a per-utterance weight. Without it a group would be up-weighted twice, once by its share and again by however many utterances it happens to hold.

Drawn with replacement, one draw per epoch, the same draw on every rank so that accelerate's sharding stays consistent. The epoch length is unchanged, so the learning-rate schedule is identical to the uniform run's.

No floor was applied, deliberately.

Results

Held-out split, 38,141 utterances after filtering:

These are dev numbers on a locally held-out split and are not comparable to the CERs this family reports from the challenge's sequestered test set.

Why it probably lost

The competition scores pooled CER — total character errors over total reference characters — while the difficulty term pulls the sampler towards equalising error across etiologies. Moving budget away from ALS, where the model is already at 1.37% median CER, costs more than it buys: ALS is 36.6% of the test set by character mass, so a small regression there outweighs a larger proportional gain on Cerebral Palsy.

Parkinson's took the sharpest cut, to 0.26x, and it is the largest group in the corpus at 146,653 utterances. This run saw roughly a quarter of the Parkinson's exposure the control did, and its dev CER was still falling at epoch 10 (5.13% -> 4.92%) where the control had flattened (4.84% -> 4.84%). Longer training might close the gap, but that would no longer be the same comparison.

One seed. The gap is about seven times the sampling standard error on the full split, so it is unlikely to be noise, but a single run cannot separate the sampler from the seed.

Training data

441,928 survive the 0.5-45 s and 200-label-token filters. Nothing comes from outside the challenge corpora, so this is a constrained-track model.

SAPC1's test1 — a 50-speaker official split that SAPC2 re-released inside its own training data — is carved back out and added to SAPC2 dev, because SAPC2's own dev split contains no Parkinson's speakers at all and a per-etiology measurement on it would be blind to the largest group in the corpus. Speaker, file-path and basename overlap with the training manifests are all zero.

Training procedure

Ten epochs on sixteen GH200s, effective batch 32 at per-device 2, AdamW at 1e-4 with a tri-stage schedule (10% warmup, 40% hold), weight decay 0.01, layerdrop 0.05, gradient clip 1.0, bf16, seed 42, 138,110 optimizer steps. Augmentation is speed perturbation over 0.8-1.2, SpecAugment (5% of the time axis in spans of 10 frames, 40% of the mel axis in spans of 27 bins) and SpecCutout (two 20x20 rectangles). The published weights are epoch 10, the best of ten by dev CER.

Usage

import soundfile as sf
import torch
from transformers import AutoProcessor, ParakeetForRNNT

model_id = "dys-asr/parakeet-rnnt-0.6b-v2split-sampler"
device = "cuda" if torch.cuda.is_available() else "cpu"

processor = AutoProcessor.from_pretrained(model_id)
model = ParakeetForRNNT.from_pretrained(model_id).to(device).eval()

audio, rate = sf.read("utterance.wav", dtype="float32")
inputs = processor(audio, sampling_rate=rate, return_tensors="pt").to(device)
with torch.inference_mode():
    generated = model.generate(**inputs)
print(processor.batch_decode(generated, skip_special_tokens=True)[0])

Requires transformers>=5.9. Audio must be 16 kHz mono.

Intended use and limitations

A competition entry for the Speech Accessibility Project Challenge, and research on atypical-speech recognition.

• Worse than its uniform control on held-out data. Use the control unless you are studying the sampler itself.
• Single run, single seed. No variance estimate.
• Requires transformers>=5.9. Not loadable by the 4.x line.
• Slow relative to CTC. Decoding is autoregressive.
• Lower-case, unpunctuated, numerals written as words.
• English only, 16 kHz mono.
• Not a clinical tool. Nothing here supports inference about any diagnosis.

License and attribution

The Speech Accessibility Project corpora are governed by their own data use agreement and are not redistributed here; reproducing this training set requires authorized access. Fun-CosyVoice3's terms apply to the synthesis component.