cohere-transcribe-03-2026-NVFP4

cohere-transcribe-03-2026 NVFP4 W4A16

This is an NVFP4 W4A16 (weights FP4, activations BF16) quantized version of CohereLabs/cohere-transcribe-03-2026, a 2B parameter Conformer-based encoder-decoder ASR model supporting 14 languages.

Quantization Details

Usage

With vLLM

vllm serve jeffpeng3/cohere-transcribe-03-2026-NVFP4 \
  --quantization modelopt \
  --dtype bfloat16

With Transformers

import torch
from transformers import AutoProcessor, AutoModelForSpeechSeq2Seq

model_id = "jeffpeng3/cohere-transcribe-03-2026-NVFP4"

processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForSpeechSeq2Seq.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True,
)

# Transcribe an audio file
from transformers.audio_utils import load_audio

audio = load_audio("path/to/audio.wav", sampling_rate=16000)
inputs = processor(audio, sampling_rate=16000, return_tensors="pt", language="en").to(model.device)
inputs = inputs.to(dtype=model.dtype)

outputs = model.generate(**inputs, max_new_tokens=256)
text = processor.decode(outputs[0], skip_special_tokens=True)
print(text)

Supported Languages

• European: English, French, German, Italian, Spanish, Portuguese, Greek, Dutch, Polish
• APAC: Chinese (Mandarin), Japanese, Korean, Vietnamese
• MENA: Arabic

Model Architecture

• Encoder: Conformer (48 layers, d_model=1280)
• Decoder: Transformer (8 layers, hidden_size=1024)
• Parameters: 2B
• Tokenizer: SentencePiece (vocab_size=16384)

Output Structure

cohere-transcribe-03-2026-NVFP4/
├── config.json                  # Model config + quantization metadata
├── hf_quant_config.json         # ModelOpt quantization config
├── model.safetensors            # NVFP4 quantized weights (1.5 GB)
├── modeling_cohere_asr.py       # Custom model code
├── processing_cohere_asr.py     # Custom processor code
├── tokenizer.json / .model      # SentencePiece tokenizer
└── preprocessor_config.json     # Feature extractor config

vLLM Serving

For online inference with vLLM:

vllm serve jeffpeng3/cohere-transcribe-03-2026-NVFP4 \
  --quantization modelopt \
  --dtype bfloat16 \
  --max-model-len 1024

License

Apache 2.0