nemotron-speech-streaming-hr-0.6b

Nemotron Speech Streaming HR 0.6B

Fine-tuned Croatian automatic speech recognition model based on nvidia/nemotron-speech-streaming-en-0.6b. It is a cache-aware streaming FastConformer-RNNT model: it transcribes audio incrementally with a fixed left context and a selectable lookahead, giving a smooth accuracy/latency trade-off from ~1 s lookahead down to zero-lookahead (instant) streaming.

The model was fine-tuned on a large corpus of Croatian speech using a multi-phase strategy that balances in-domain accuracy with broader speaker and topic generalization.

Intended Use

Designed for:

• Real-time Croatian speech transcription in streaming and low-latency applications
• Formal Croatian speech (parliamentary, broadcast, dictation)
• Clean to moderately noisy audio (SNR ≥ 10 dB)
• 16 kHz mono audio input

Not recommended for:

• Heavily dialectal Croatian
• Number-heavy or date-critical content — number transcription is unreliable (see Number transcription errors)
• Audio below SNR ~10 dB — accuracy degrades
• Utterances longer than 15 s without prior segmentation

Model Details

Evaluation

Methodology

All streaming numbers below use the model's native cache-aware inference path: the left context is held at the trained value (70 frames, ~5.6 s) and only the lookahead (right context) is varied across the four values the model was trained on. Operating points are labeled by total buffer — the new audio consumed per step — following the convention used in NVIDIA's Nemotron ASR cards; total buffer = lookahead + one 80 ms frame. The four points are 1120 / 560 / 160 / 80 ms total buffer, corresponding to lookaheads of 1040 / 480 / 80 / 0 ms. This is the configuration in which cache-aware models are designed to match offline decoding. WER/CER are computed after light normalization (lowercase, punctuation stripped, whitespace collapsed) using jiwer, with greedy decoding on an NVIDIA A40.

Streaming context. This model uses FastEmit regularization (λ=0.005), which biases toward emitting tokens earlier to reduce latency. Its main side-effect is a higher deletion rate for short function words (i, je, u, na, od). CER therefore tracks true transcription quality more closely than WER; comparisons to offline models are best made on CER.

The three datasets probe different conditions:

Headline results (1120 ms buffer)

The spread reflects content difficulty, not decoding behavior: FLEURS is dominated by foreign proper nouns and numerals (its WER is high while CER stays ~10%, the signature of whole-word errors on rare tokens), whereas conversational Croatian sits around 16–24%.

Recommended operating point. For most applications, use the 560 ms buffer (att_context_size = [70, 6]): it is within 0.15–0.56 pt WER of the best (largest) buffer across the three datasets while running at ~17× real-time with a roughly half-second response floor. Move up to the 1120 ms buffer if you have latency headroom and want peak accuracy, or down to 80–160 ms for near-instant captioning at ~1–2 pts higher WER. First-token latency ≈ lookahead + ~30 ms compute (e.g. ~510 ms at the 560 ms buffer).

Accuracy vs latency (all four lookaheads)

Accuracy is nearly flat across the full latency range — the model degrades gracefully as lookahead shrinks, with no instability at any setting.

ParlaSpeech-HR (500, held-out)

VoxPopuli-hr (664)

FLEURS-hr (914)

Going from the largest buffer (1120 ms) to the smallest (80 ms, zero lookahead) costs only 1.3 pts WER on ParlaSpeech, 2.3 on VoxPopuli, 1.5 on FLEURS. Stability (fraction of partial-hypothesis words retained in the final output) actually rises as the buffer shrinks, because less future context means fewer late revisions — a useful property for live captioning.

Streaming ≈ offline

At the largest buffer the streaming model reaches parity with full-file (offline) decoding. On FLEURS, offline WER is 28.41% vs streaming-at-1120 ms-buffer 28.50% — a +0.09 pt difference. Cache-aware streaming here is effectively free: you get incremental output at no accuracy cost relative to batch transcription.

Latency

Benchmarked on NVIDIA A40, greedy decoding, ParlaSpeech.

Per-step compute latency is flat (~30–34 ms) because it is dominated by fixed encoder cost, not buffer length. What changes is throughput: a larger buffer means fewer, larger forward passes, so RTFx climbs to 33.8× at the 1120 ms buffer. Even the 80 ms buffer (zero lookahead) runs at 2.6× real-time, so every operating point is real-time-capable on this hardware. The response latency floor (how long before a token can be emitted) is set by the lookahead: 1040 / 480 / 80 / 0 ms.

Deployment

The measured 5.1 GB peak is higher than the ≈2.5 GB nominal fp32 weight size (622M params × 4 bytes); the remainder is additional runtime allocation that NeMo holds at load (retained buffers, workspace, and framework overhead) rather than model weights. Streaming activations and encoder cache add only about 0.1 GB on top, so the footprint is essentially flat with utterance length — a 6 GB GPU is sufficient for single-stream inference. Casting the weights to BF16 would roughly halve the weight portion (≈2.5 GB → ≈1.2 GB) but leaves the non-weight runtime allocation largely unchanged, so the total footprint would drop modestly rather than by half. The numbers above reflect the default fp32 configuration in which the accuracy results were measured. Throughput on the A40 ranges 2.6–33.8× real-time depending on buffer size.

Transcription Examples

Real reference/hypothesis pairs from the 1120 ms-buffer runs (normalized text), illustrating typical behavior:

Clean parliamentary speech — exact match

REF: Idemo dalje, druga replika poštovani kolega Karlić, izvolite.
HYP: idemo dalje druga replika poštovani kolega karlić izvolite

Minor error — word-boundary slip (10% WER)

REF: Ovo je udar na male iznajmljivače, na kičmu hrvatskog turizma.
HYP: ovo je udar na izmale iznajmljivače na kičmu hrvatskog turizma

(na male → izmale, a boundary merge; every other word correct.)

Number handling — written-form drift

REF: ...prije otprilike 10.000 godina u planinama zagros u iranu
HYP: ...prije otprilike deset tisuća godina u planinama zagroz u iranu

(The quantity is correct but rendered as words deset tisuća instead of 10.000, so WER penalizes it — see Number transcription errors.)

Foreign names + numerals — the hardest case (38% WER)

REF: bus scotturb 403 redovito vozi za sintru a staje u cabo da roci
HYP: bus kotor 43 redovito vozi za sintru a staje u kabodiroki

(scotturb 403 → kotor 43, cabo da roca → kabodiroki: foreign proper nouns and a dropped digit — the dominant FLEURS failure mode.)

Error Analysis

Recomputed from full per-sample alignments at the 1120 ms-buffer operating point: 914 (FLEURS), 500 (ParlaSpeech), 664 (VoxPopuli) utterances, word-level Levenshtein alignment on normalized text.

WER distribution

The distributions are well-behaved — the corpus WER is representative of the typical case, not driven by a heavy tail. On ParlaSpeech nearly 13% of utterances are transcribed perfectly and under 1% exceed 50% WER.

Error type breakdown

Substitution classification

Each substitution is bucketed into one of five types.

Inflectional morphology accounts for 25–35% of substitutions across all datasets — the correct stem with the wrong grammatical ending (e.g. koje → koji, neuter vs masculine; istraživanja → istraživanje, genitive vs nominative). These are semantically low-cost (the referent stays recognizable) but WER penalizes them like completely wrong words. For a heavily inflected language this is an inherent gap between WER and usable transcription quality.

Truncation on ParlaSpeech (16.5%) is largely colloquial short forms (kada → kad, sada → sad) plus a few reference-punctuation effects.

Most-deleted words

The same monosyllabic function words dominate deletions everywhere, and are the main reason WER sits above CER.

• FLEURS: i ×61 · je ×43 · u ×41 · s ×24 · od ×17 · na ×15 · za ×11
• ParlaSpeech: i ×65 · je ×56 · u ×41 · o ×18 · da ×15 · se ×10
• VoxPopuli: i ×119 · je ×54 · u ×45 · gospodine ×29 · za ×19 · na ×17

gospodine appears in VoxPopuli deletions because speakers routinely open by addressing the chair and the model sometimes drops the salutation — an in-domain parliamentary phenomenon. The English tokens two (×19) and thousand (×16) in VoxPopuli deletions are a reference artifact: some VoxPopuli-hr references contain English words (see data-quality note).

Most common substitution pairs

• FLEURS: s→sa ×14 · iz→i ×10 · s→iz ×9 · o→u ×7 · koje→koji ×6 · južnoafričkim→afričkim ×5
• ParlaSpeech: kada→kad ×10 · koje→koji ×5 · kn→kuna ×4 · sada→sad ×4 · što→šta ×4
• VoxPopuli: to→što ×36 · s→sa ×29 · takoer→također ×15 · izvjee→izvješće ×14 · izmeu→između ×14

Short prepositions (s / sa / iz / o / u) are disproportionately confused: 1–2 character words, phonetically similar, under early-emission pressure. On VoxPopuli several top "substitutions" (takoer→također, izvjee→izvješće, izmeu→između) are the model producing the correct diacritic form against a diacritic-stripped reference — an artifact, not an error (see below).

Word-length error rate (FLEURS)

Error rate climbs steeply beyond ~8 characters. Croatian forms long inflected compounds (e.g. južnoafričkim) that stack multiple morphological layers and fall outside common training n-grams; SentencePiece may also segment them differently at inference than in training.

Number transcription errors

Numeric content is unreliable (~6% of FLEURS substitutions involve digits) and should not be trusted without post-processing. Observed patterns, with real examples from the runs:

• Trailing-zero drop: 120 → 12, 403 → 43, 39000 → 309
• Year transposition / mangling: 2017 → 17, 1537 → 307
• Written ↔ digit oscillation: number words rendered as digits and vice-versa

Year transpositions are the most operationally dangerous — plausible and hard to catch downstream. For any date/quantity-sensitive application, add a number-normalization post-processor.

Diacritic recall

Fraction of reference diacritics preserved in the hypothesis.

ParlaSpeech recall is highest (92–97%) — formal register with consistent diacritic usage in training. ć is the weakest on FLEURS (82%), consistent with its acoustic similarity to č in connected speech. † VoxPopuli recall is artificially depressed by the reference issue below.

Hypothesis repetition loops

Consecutive word repetitions (e.g. igra igra igra) — a known RNNT failure mode — are rare on the native cache-aware path: 2 / 914 (FLEURS), 2 / 500 (ParlaSpeech), 0 / 664 (VoxPopuli). The worst single case is FLEURS id=1976 (igra repeated on the cricket name "Sachin Tendulkar"). A deduplication pass (collapse runs of ≥3 identical tokens) removes the few remaining cases at no cost on clean audio.

Data-quality note on VoxPopuli-hr

A substantial share of VoxPopuli-hr references are romanized ASCII that omit Croatian diacritics (takoer for također, gospoo for gospođo) and some contain English tokens (two, thousand). Where the model emits correct standard Croatian, alignment against these references scores it as an error. VoxPopuli numbers here are therefore a pessimistic estimate; ParlaSpeech and FLEURS references are orthographically standard.

Training Data

Total ~1,871 h pre-filtering (effective hours lower after 0.3–15.0 s duration filtering).

Evaluation integrity. The model was trained on ParlaSpeech-HR. The 500 ParlaSpeech utterances used for evaluation above were held out from the training set (sampled with seed 42 from utterances not seen in training, clips ≥ 4 s), so the ParlaSpeech figure is a held-out in-domain result rather than a train-set score. FLEURS (full 914 test set) and VoxPopuli are reported on their standard test splits.

Training Procedure

Three-phase strategy on an NVIDIA A100-SXM4-80GB.

Phase 1 — Decoder & Joint only (encoder frozen)

Adapt decoder and joint to Croatian while keeping the encoder stable.

Phase 2 — Full fine-tuning, differential learning rates

Unlock the encoder at a small LR. After Phase 2, in-domain (ParlaSpeech) WER reached ~14.5% but out-of-domain FLEURS WER was ~35% — clear domain overfitting.

Phase 2.5 — Diversity fine-tuning

Reduce domain overfitting without sacrificing stability; brought FLEURS WER from ~35% down to the reported 28.50%.

Data augmentation & preprocessing

• Speed perturbation: 0.9×–1.1×
• SpecAugment: 3 freq masks (27 bins), 12 time masks (0.08 s)
• Text normalization: strip trailing punctuation
• Duration filtering: 0.3–15.0 s
• Tarred dataset (128 shards); bucketed batching (buckets [96, 72, 48, 32, 16])

Usage

Loading

import nemo.collections.asr as nemo_asr

model = nemo_asr.models.ASRModel.restore_from("nemotron-speech-streaming-hr-0.6b.nemo")
model.eval()

Compatibility: load with a recent NeMo (the 26.02 NGC container or NeMo main). ASRModel.restore_from auto-detects the model class. On very new PyTorch you may need weights_only=False when loading checkpoints.

Offline / batch inference

transcriptions = model.transcribe(["audio.wav"])
print(transcriptions[0])

Streaming inference (native cache-aware)

Select an operating point via the model's trained att_context_size — fixed left context (70 frames), lookahead in frames:

# lookahead frames -> total buffer:  13 -> 1120 ms (best WER)   6 -> 560 ms
#                                      1 ->  160 ms              0 ->   80 ms (instant)
LEFT, LOOKAHEAD = 70, 13
model.encoder.set_default_att_context_size([LEFT, LOOKAHEAD])
model.encoder.setup_streaming_params()

Then drive it with CacheAwareStreamingAudioBuffer + model.conformer_stream_step, carrying the cache state between chunks. A complete reference implementation is NeMo's speech_to_text_cache_aware_streaming_infer.py.

Prefer the larger buffer (1120 ms) when latency budget allows; use the 80–160 ms buffers for near-instant streaming at ~1–2 pts higher WER. Accuracy is flat across the range, so choose purely on your latency requirement.

Known Failure Modes

• Short isolated utterances. Single words / very short phrases at low lookahead can yield empty output — a structural property of streaming RNNT with limited context ("Hvala." → (empty)).
• Numeric content. Trailing-zero drops (120 → 12) and year transpositions (2017 → 17) are the most dangerous; use number post-processing for date/quantity-critical use.
• Long compound words. 10+ character words show 35–58% error rates on FLEURS; domain-adapted fine-tuning helps for technical vocabulary.
• Morphological case. Correct stems with wrong endings (istraživanja → istraživanje) score as full-word errors but carry low semantic cost; downstream lemmatization mitigates.
• Repetition loops. Rare on the cache-aware path (≤2 per ~900 utterances); a ≥3-token dedup pass removes them.

Related models

Since this model was trained, NVIDIA has released nvidia/nemotron-3.5-asr-streaming-0.6b — a multilingual (40 language-locale) cache-aware streaming model that adds native Croatian support via language-ID prompt conditioning, along with built-in punctuation and capitalization. It shares the same Cache-Aware FastConformer-RNNT architecture and the same total-buffer operating points used here, and is a strong general-purpose alternative for Croatian streaming ASR. This model remains a lightweight, Croatian-specialized fine-tune focused on the parliamentary/formal domain; a direct head-to-head on the datasets above is a natural point of comparison.

Limitations

• Optimized for 16 kHz mono audio.
• Best on clean speech; robustness degrades below ~10 dB SNR.
• Primary domain is parliamentary Croatian; strong dialectal speech may vary.
• Max recommended utterance length 15 s; segment longer audio first.
• Number transcription is unreliable without post-processing.

Training Infrastructure

• GPU: NVIDIA A100-SXM4-80GB
• Loss: RNNT (warprnnt_numba), FastEmit λ=0.005
• Batch size: 32 (per device), gradient clip 1.0
• Experiment tracking: Weights & Biases

Citation

@misc{nemotron-speech-streaming-hr-0.6b,
  title        = {Nemotron Speech Streaming HR 0.6B},
  author       = {Ivan Luksic},
  year         = {2026},
  howpublished = {Hugging Face},
  url          = {https://huggingface.co/iluksic/nemotron-speech-streaming-hr-0.6b},
  note         = {Fine-tuned from nvidia/nemotron-speech-streaming-en-0.6b on Croatian speech}
}

License

This model is a derivative of nvidia/nemotron-speech-streaming-en-0.6b and is released under the NVIDIA Open Model License, inheriting the base model's license terms. Please review the base model card and the license text for the full conditions before use or redistribution.