nemotron35-he-archive-2026-07-04

benderrodriguez/nemotron35-he-archive-2026-07-04

nemotron35-he-archive-2026-07-04 is a machine learning model from benderrodriguez. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for nemo.

Complete archive of the Hebrew fine-tune of nvidia/nemotron-3.5-asr-streaming-0.6b on a vast.ai H100. Repo of record (scripts, RECIPES.md, STATUS.md): git nt-3.5-asr-streaming-ft. Repo was private, made public 2026-07…

Downloads · 30 days

1% of all-time downloads

All-time downloads

1.5K

Public

Repo size

212 GB

Likes

Public

Hugging Face

Repo makeup

Click a slice to open those files.

.ckpt161 GB · 76%

At a glance

Library: nemo
Access: Public
Created: Jul 4, 2026
Updated: Jul 5, 2026
SHA: c57e4f53

Model card

From the Hugging Face model README

Hebrew Nemotron 3.5 ASR streaming fine-tune — full archive (final, 2026-07-05)

Complete archive of the Hebrew fine-tune of nvidia/nemotron-3.5-asr-streaming-0.6b on a vast.ai H100. Repo of record (scripts, RECIPES.md, STATUS.md): git nt-3.5-asr-streaming-ft. Repo was private, made public 2026-07-05 (private-storage quota).

Final model

checkpoints/2026-07-05_03-18-46/nemotron35_he_mix_ct5--val_wer=0.2085-epoch=198.ckpt (= nemo_models/ckpt_mix_final.nemo, weights-only 2.4G; the leg .nemo backup in the same dir is identical). Mixed run: committees 0.5 / CT-v5 0.5 (Lhotse input_cfg), 200k steps total, init from ckpt150k (committees-only run 2). Final global-best val_wer 0.20851@199k (CT-v5 dev, canonical Hebrew).

Resume training

Resume point: checkpoints/2026-07-05_03-18-46/...epoch=198-last.ckpt (fp32 + Adam moments + trainer state, 7.7G) via +exp_manager.resume_from_checkpoint=... — see RECIPES.md §4. Training data: committees tars in HF dataset benderrodriguez/knesset-committees-prepped; CT-v5 tars NOT archived (gated source) — re-prep with scripts/prepare_ct5_train.py + scripts_box/prep_ct5_box.sh.

Final eval (canonical WER, streaming att_context [56,3], target_lang=he-IL)

benchmark	base	com.@20k	@40k	@50k	@150k	mix@49k	@79k	@100k	mix final @199k
CT-v5 test (2000)	0.3570	0.3058	0.2792	0.2671	0.2733	—	0.1947	0.1906	0.1806
committees dev (946)	0.4620	0.3391	0.3196	0.3135	0.2959	—	0.2871	0.2918	0.2826
eval-d1 (17 long-form)	0.1892	0.1911	0.1713	0.1700	0.1543	0.1394	0.1393	0.1367	0.1374
eval-whatsapp (54)	0.3273	—	—	—	0.2658	0.2062	0.2011	0.1994	0.1998

All numbers rescored 2026-07-05 from the archived prediction manifests in outputs/ with scripts/score_canonical.py (identical test subsets throughout; earlier notes mislabeled ckpt50k results as "base").

Scoring: scripts/score_canonical.py; predictions in outputs/eval_*.

val_wer milestones (mix run, CT-v5 dev)

0.2283@37k → 0.22017@53k → 0.21645@73k → 0.2132@104k → 0.21206@128k → 0.21092@133k → 0.20949@139k → 0.20886@194k → 0.20851@199k (final).

Layout

checkpoints/<run|leg>/ — best + -last Lightning .ckpt per leg (7.7G each)
nemo_models/ — deployable .nemo milestones (ckpt20k/40k/50k/150k, ckpt_mix49k/79k/100k, ckpt_mix_final)
nemo_experiments_meta/ — full exp tree w/o ckpt/nemo: TensorBoard events, hparams.yaml, cmd-args for every leg
logs/ — all console logs (train legs incl. crash-preserved, evals, uploads)
outputs/ — eval prediction manifests for every eval run
scripts_box/ — box-only scripts not in git
MANIFEST_all_ckpts_on_box.txt — ls -laR at first-archive time

Crash history (mix run, 9 recoveries)

[email protected], [email protected], [email protected] (BD 100→70), ENOSPC@106k+107k, numba@131k, [email protected] (BD→60), [email protected], numba@199k. RNNT grad spikes ~26G; numba = cuLaunchKernel CUDA_ERROR_INVALID_VALUE. Preserved logs: logs/train_mix_*.log.

Library: nemo
Created: Jul 4, 2026
Updated: Jul 5, 2026