cohere-transcribe-onnx-int8

Cohere Transcribe - ONNX

ONNX conversion of CohereLabs/cohere-transcribe-03-2026, a 2B parameter Conformer-based speech recognition model supporting 14 languages.

Runs locally with ONNX Runtime in Python, C#, C++, Java, JavaScript, or any ONNX-compatible runtime. No Python, no cloud APIs, no GPU required.

What's Included

INT8 dynamically quantized ONNX model (2.75 GB total):

Plus:

• Export and quantization scripts (reproduce FP32 or re-quantize from the original model)
• C# inference example
• Complete porting guide with all concepts explained

Quick Start (Python)

pip install onnxruntime numpy soundfile librosa

import onnxruntime as ort
import numpy as np
import librosa

# Load audio (must be 16kHz mono)
audio, sr = librosa.load("your_audio.wav", sr=16000, mono=True)

# Load models
enc = ort.InferenceSession("cohere-encoder.int8.onnx")
dec = ort.InferenceSession("cohere-decoder.int8.onnx")

# Load tokens
tokens = {}
with open("tokens.txt", "r", encoding="utf-8") as f:
    for line in f:
        parts = line.strip().rsplit(" ", 1)
        if len(parts) == 2:
            tokens[int(parts[1])] = parts[0]
token_to_id = {v: k for k, v in tokens.items()}

# Build prompt (change <|en|> to <|de|>, <|fr|>, etc.)
prompt_ids = [token_to_id[t] for t in [
    "<|startofcontext|>", "<|startoftranscript|>", "<|emo:undefined|>",
    "<|en|>", "<|en|>", "<|pnc|>", "<|noitn|>", "<|notimestamp|>", "<|nodiarize|>"
]]

# Run encoder (raw audio in, encoder representations out)
cross_k, cross_v = enc.run(None, {"audio": audio.reshape(1, -1).astype(np.float32)})

# Run decoder (autoregressive greedy decoding)
N_LAYERS, HEADS, HEAD_DIM, MAX_CTX = 8, 8, 128, 1024
self_k = np.zeros((N_LAYERS, 1, HEADS, MAX_CTX, HEAD_DIM), dtype=np.float32)
self_v = np.zeros((N_LAYERS, 1, HEADS, MAX_CTX, HEAD_DIM), dtype=np.float32)
eos_id = token_to_id["<|endoftext|>"]

generated = list(prompt_ids)
current = np.array([prompt_ids], dtype=np.int64)
offset = np.array(0, dtype=np.int64)

for _ in range(256):
    logits, self_k, self_v = dec.run(None, {
        "tokens": current, "in_n_layer_self_k_cache": self_k,
        "in_n_layer_self_v_cache": self_v, "n_layer_cross_k": cross_k,
        "n_layer_cross_v": cross_v, "offset": offset,
    })
    next_id = int(np.argmax(logits[0, -1, :]))
    if next_id == eos_id:
        break
    generated.append(next_id)
    offset = np.array(int(offset) + current.shape[1], dtype=np.int64)
    current = np.array([[next_id]], dtype=np.int64)

# Decode to text
text = "".join(
    tokens.get(t, "").replace("\u2581", " ")
    for t in generated[len(prompt_ids):]
    if not tokens.get(t, "").startswith("<|")
).strip()
print(text)

Quick Start (C#)

dotnet add package Microsoft.ML.OnnxRuntime

See CohereTranscribe.cs for a complete example. For GPU acceleration on Windows:

dotnet add package Microsoft.ML.OnnxRuntime.DirectML

Supported Languages

Change language by swapping <|en|> in the prompt to the desired language code.

Performance (INT8)

All benchmarks run on a Windows 11 machine with CPU only (no dedicated GPU), 24 GB shared RAM, using the INT8 variant. No special optimization or batching -- just sequential chunk-by-chunk processing.

What the Numbers Mean

• 5.6x realtime means 1 second of audio is transcribed in ~0.18 seconds
• 3.7-3.8x realtime on longer files (the slight slowdown vs short clips is because longer audio produces more decoder tokens per chunk)
• Encoder takes ~3.5-7s per 30s chunk (the heavy lifting -- 48 Conformer layers)
• Decoder takes ~1.5-3s per chunk depending on how many words are spoken

Memory Usage

Peak RAM stays constant regardless of audio length -- only one chunk is in memory at a time. A 5-minute file and a 5-hour file use the same amount of RAM.

GPU Acceleration

These benchmarks are CPU-only. For GPU acceleration on Windows, use DirectML (works with any DirectX 12 GPU -- AMD, NVIDIA, Intel). Expected 3-5x additional speedup, bringing throughput to 15-25x realtime.

Long Audio

The encoder handles up to ~35 seconds per call. For longer audio, split into overlapping chunks:

• Chunk size: 30 seconds
• Overlap: 5 seconds
• Stride: 25 seconds

Transcribe each chunk, join results with spaces.

How This Was Made

The original PyTorch model was converted to ONNX with several engineering adaptations:

1. Feature extraction baked in -- The encoder takes raw 16kHz audio, not mel spectrograms. STFT is implemented via Conv1d DFT filters (since torch.stft doesn't export to ONNX).

2. Cross-attention K/V pre-computed -- The encoder pre-computes Key/Value projections for all 8 decoder layers, matching the Whisper-style encoder-decoder ONNX pattern.

3. Selective quantization -- INT8 quantization protects convolutional and batch normalization layers in the audio frontend for accuracy preservation.

See PORTING_GUIDE.md for the complete technical writeup with every concept explained.

ONNX Tensor Contract

Encoder

Input:   audio              (batch, samples)       float32, raw 16kHz mono
Output:  n_layer_cross_k    (8, batch, T', 1024)   float32
         n_layer_cross_v    (8, batch, T', 1024)   float32

Decoder

Input:   tokens                    (batch, n_tokens)              int64
         in_n_layer_self_k_cache   (8, batch, 8, 1024, 128)      float32
         in_n_layer_self_v_cache   (8, batch, 8, 1024, 128)      float32
         n_layer_cross_k           (8, batch, T', 1024)           float32
         n_layer_cross_v           (8, batch, T', 1024)           float32
         offset                    ()                              int64 scalar
Output:  logits                    (batch, n_tokens, 16384)       float32
         out_n_layer_self_k_cache  (8, batch, 8, 1024, 128)      float32
         out_n_layer_self_v_cache  (8, batch, 8, 1024, 128)      float32

Attribution

Original model: CohereLabs/cohere-transcribe-03-2026 by Cohere and Cohere Labs.

ONNX conversion, quantization, and documentation: community contribution.

License: Apache 2.0