cohere-transcribe-03-2026-int8

cohere-transcribe-03-2026 — int8 (CPU, pre-quantized)

A dynamic int8-quantized build of CohereLabs/cohere-transcribe-03-2026 for faster, lighter CPU inference, plus a small loader so you never have to load the fp32 weights at runtime.

⚠️ You still need access to the GATED base model. This repo ships only the quantized weights (cohere_int8_state.pt). The config, custom model code and processor are pulled from CohereLabs/cohere-transcribe-03-2026, which is gated — request access there and be logged in. License/usage follow the base model.

Requirements

pip install -r requirements.txt
# transformers MUST be 5.3.0 — the model's custom code targets 5.3.0.dev0.
# transformers 5.4.0+ loads the weights cleanly but produces multilingual GARBAGE.

Usage

from load_cohere_int8 import load_int8_model
import soundfile as sf, numpy as np

model, processor = load_int8_model()          # downloads cohere_int8_state.pt from this repo
wav, sr = sf.read("audio_16k_mono.wav")       # 16 kHz mono float
wav = np.asarray(wav if wav.ndim == 1 else wav.mean(1), dtype=np.float32)

out = model.transcribe(processor=processor, language="en",
                       audio_arrays=[wav], sample_rates=[sr], punctuation=True)
print(out[0])

language is required (no auto-detect). Supported: en, fr, de, es, it, pt, nl, pl, el, ar, ja, zh, vi, ko.

How it was made

from transformers import AutoModelForSpeechSeq2Seq
import torch
m  = AutoModelForSpeechSeq2Seq.from_pretrained("CohereLabs/cohere-transcribe-03-2026",
        dtype=torch.float32, low_cpu_mem_usage=True, trust_remote_code=True).eval()
mq = torch.ao.quantization.quantize_dynamic(m, {torch.nn.Linear}, dtype=torch.qint8)
torch.save(mq.state_dict(), "cohere_int8_state.pt")   # ~2.7 GB

The loader (load_cohere_int8.py) then rebuilds the architecture on the meta device (zero weight memory), swaps each nn.Linear for an empty dynamic-int8 Linear, and load_state_dict(..., assign=True) — so fp32 is never allocated.

Caveats

• CPU-only / heavy. Even at ~6 GB this is a 2B model; it's slower than Whisper on small devices. Great accuracy, but for tiny edge boards prefer faster-whisper.
• Hallucinates on silence/noise — gate your audio with a VAD; non-speech in → fixed gibberish out.
• bfloat16 is NOT used — it is ~14× slower on a typical CPU (emulated).

Attribution

Base model © Cohere Labs, CohereLabs/cohere-transcribe-03-2026. This is an unofficial quantized redistribution of the weights for convenience; all rights, license terms and gating of the base model apply.