Downloads · 30 days
43
46% of all-time downloads
dys-asr/parakeet-rnnt-0.6b-all-syn-chunk-cutout-swa
parakeet-rnnt-0.6b-all-syn-chunk-cutout-swa is a automatic speech recognition model from dys-asr. Use it when you need speech turned into text. It is set up for transformers. The card lists the license as other.
Stochastic weight averaging on top of dys-asr/parakeet-rnnt-0.6b-all-syn-chunk-cutout. Training continued for five more epochs with a triangular cyclic learning rate — one cycle per epoch, from 5e-6 up to 3e-5 and bac…
Downloads · 30 days
43
46% of all-time downloads
All-time downloads
93
Public
Parameters
618M
2.5 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors2.5 GB · 100%
From the Hugging Face model README
Stochastic weight averaging on top of
dys-asr/parakeet-rnnt-0.6b-all-syn-chunk-cutout.
Training continued for five more epochs with a triangular cyclic learning
rate — one cycle per epoch, from 5e-6 up to 3e-5 and back — and the weights
were exported at the bottom of each cycle. Those five snapshots are averaged
with equal weights.
The point of the cycle is that every snapshot is taken at the same low learning rate, so they are comparable points rather than arbitrary ones, and the average cancels the noise each of them carries.
Scored on the held-out dev split, 4,000 utterances, every row measured the same way in the same job:
| checkpoint | CER | WER |
|---|---|---|
| this model (five cycles averaged) | 5.964% | 9.706% |
| best single snapshot (cycle 1) | 6.036% | 9.889% |
| the checkpoint it started from | 6.121% | 9.939% |
| the same average with BatchNorm recomputed | 6.154% | 9.993% |
No individual snapshot beat the starting checkpoint — they range from 6.01% to 6.21% on the training-time metric. The gain is entirely from averaging, which is what weight averaging is supposed to do and is the only reason this model exists.
Textbook SWA resets the BatchNorm running statistics and recomputes them, on the
argument that averaged weights are not any of the models that were averaged, so
their stored activation statistics no longer describe them. That was tried here:
24 BatchNorm layers reset, statistics re-accumulated over 2,000 training batches
with momentum=None. It made the model worse — 6.154% against 5.964% — so
the published weights carry the averaged statistics instead.
Why is a guess, not a result: the five snapshots sit at the bottoms of five cycles and are close together, so averaging their statistics is a small approximation, while recomputing them under layerdrop and full augmentation estimates a distribution that inference never sees. Nothing here tests that explanation.
Epoch-end exports from cycles 1 through 5, uniform weights, 987 tensors. Float
tensors are averaged in float64 and cast back; num_batches_tracked is carried
across as a maximum rather than averaged.
Cycle 5 was produced by a fresh restart. The run that would have produced it died 75 updates from the end of the epoch, and the retry began that cycle again from the cycle-4 weights, so it carries no optimizer momentum from its predecessor while cycles 2 to 4 do. Cycle 1 had none either. At 5e-6 into a converged model the first steps are small, but this cycle was not made under identical conditions to its neighbours.
The same corpus as
dys-asr/parakeet-rnnt-0.6b-all-syn-chunk-cutout,
unchanged, so the two are directly comparable.
| source | records |
|---|---|
| SAPC-1 train and dev | 250,014 |
| SAPC-2 train | 153,500 |
| synthetic CosyVoice3 speech | 55,988 |
| chunks force-aligned out of over-length recordings | 8,933 |
| total before filtering | 468,435 |
463,177 records survive the 0.5-45 s and 200-label-token filters. Nothing comes from outside the challenge corpora, so this is a constrained-track model.
The held-out dev split is genuinely held out. sapc2_dev_heldout.jsonl is
17,582 clips and its intersection with the training manifests is empty; that was
checked by comparing file paths, not assumed.
Ten epochs on sixteen GH200s, effective batch 32 at per-device 2, AdamW at 1e-4 with a tri-stage schedule (10% warmup, 40% hold), weight decay 0.01, layerdrop 0.05, gradient clip 1.0, bf16, seed 42, 144,750 optimizer steps. Augmentation is online speed perturbation over 0.8-1.2, SpecAugment (5% of the time axis in spans of 10 frames, 40% of the mel axis in spans of 27 bins) and SpecCutout (two 20x20 rectangles).
Five further epochs over the same corpus, --lr-scheduler triangular with
--learning-rate 3e-5 and --lr-min 5e-6, the cycle period pinned to one epoch
(14,475 optimizer updates under this world size). Everything else — corpus,
augmentation, batch, layerdrop — is the ten-epoch recipe above, unchanged.
import soundfile as sf
import torch
from transformers import AutoProcessor, ParakeetForRNNT
model_id = "dys-asr/parakeet-rnnt-0.6b-all-syn-chunk-cutout-swa"
device = "cuda" if torch.cuda.is_available() else "cpu"
processor = AutoProcessor.from_pretrained(model_id)
model = ParakeetForRNNT.from_pretrained(model_id).to(device).eval()
audio, rate = sf.read("utterance.wav", dtype="float32")
inputs = processor(audio, sampling_rate=rate, return_tensors="pt").to(device)
with torch.inference_mode():
generated = model.generate(**inputs)
print(processor.batch_decode(generated, skip_special_tokens=True)[0])
Requires transformers>=5.9. Audio must be 16 kHz mono.
A competition entry for the Speech Accessibility Project Challenge, and research on atypical-speech recognition.
transformers>=5.9. Not loadable by the 4.x line.The Speech Accessibility Project corpora are governed by their own data use agreement and are not redistributed here; reproducing this training set requires authorized access. Fun-CosyVoice3's terms apply to the synthesis component.