canary-1b-v2-vllm

canary-1b-v2-vllm

A repackaged build of nvidia/canary-1b-v2 that loads directly with vLLM 0.19.0's cohere_asr architecture, so you can serve Canary behind the OpenAI-compatible /v1/audio/transcriptions endpoint without running NeMo.

Thank you

This repository is only possible because of two pieces of open work:

• NVIDIA open-sourced both the Canary-1B-v2 model weights and the training recipe (NeMo Conformer + Transformer AED, dw-striding subsampling, relative-positional attention). All architecture knowledge and every weight in this repo comes from their release.
• Cohere Labs contributed a clean, standalone implementation of the architecture to both transformers (CohereAsrForConditionalGeneration) and vllm (cohere_asr). That implementation happens to be a near-bitwise port of the NeMo AED-Conformer inference path, which is what makes this repackaging a mechanical rename rather than a reimplementation.

We're very grateful to both teams — this repo is essentially a bridge between their work.

What this repo is

• Raw Canary-1B-v2 weights (Conformer encoder + 8-layer Transformer decoder, ~1 B params) converted to safetensors and downcast to bfloat16.
• A config.json that mirrors NeMo's model_config.yaml layout (nested encoder, transf_decoder, head, preprocessor dicts plus the three top-level fields — sample_rate, max_audio_clip_s, overlap_chunk_second — that vLLM's CohereAsr reads).
• The 16,384-token SentencePiece tokenizer shipped with the original NeMo checkpoint, packaged as a HF fast tokenizer.

What this repo is NOT

• Not a transformers-loadable repo. Transformers 5.5's CohereAsrForConditionalGeneration expects renamed HF keys and a flat config; the weights here are raw NeMo keys. Loading with AutoModelForSpeechSeq2Seq.from_pretrained will fail with missing / unexpected weights.
• Not an original model. All behaviour, accuracy, licence, and known limitations are inherited from nvidia/canary-1b-v2.

Serving with vLLM

uv venv --python 3.12 --seed
source .venv/bin/activate

uv pip install -U "vllm==0.19.0" --torch-backend=auto
uv pip install "vllm[audio]==0.19.0" librosa
uv pip install -U transformers
uv pip install "numpy<2.3" ninja  # numba/librosa + flashinfer JIT deps

vllm serve reson8/canary-1b-v2-vllm --trust-remote-code

Then hit the OpenAI-compatible transcription endpoint:

curl -X POST http://localhost:8000/v1/audio/transcriptions \
  -F "file=@your_audio.wav" \
  -F "model=reson8/canary-1b-v2-vllm" \
  -F "language=nl"

language accepts any of the 25 ISO codes Canary-1B-v2 was trained on. vLLM uses Canary's built-in audio chunking (35 s window, 5 s overlap), so long-form audio works out of the box.

Required versions

• vllm==0.19.0 (the cohere_asr registration landed in that release)
• transformers>=5.5 (needed for the CohereAsrConfig class that vLLM's auto-config resolution uses)
• numpy<2.3 — numba (pulled in by librosa) won't load on newer numpy

Common pitfalls

• Numba needs NumPy 2.2 or less — pin numpy<2.3.
• FileNotFoundError: 'ninja' — pip install ninja; flashinfer compiles kernels on first run.
• Stale flashinfer JIT cache from another venv — rm -rf ~/.cache/flashinfer.

Licence

cc-by-4.0, inherited from the upstream NVIDIA release. See nvidia/canary-1b-v2 for the authoritative licence text and acceptable-use notes.

Citation

If you use this repackaging, please cite the original Canary-1B-v2 paper and the Cohere Labs cohere_asr implementation:

• Canary-1B-v2 Technical Report
• CohereLabs/cohere-transcribe-03-2026

— and, of course, NVIDIA NeMo for the underlying training framework and model release.