cohere-transcribe-coreml-int8

Cohere Transcribe 2B — CoreML INT8 (W8A16) for Apple Silicon & iOS

This repository provides W8A16 Linear Symmetric Quantized CoreML (.mlpackage) models for Cohere Transcribe 2B (c4ai-transcribe-2b), optimized for high-performance speech-to-text on Apple Silicon (M1/M2/M3/M4/M5) and mobile Apple devices.

Quantized using Apple's official coremltools.optimize.coreml pipeline with OpLinearQuantizerConfig(mode='linear_symmetric', dtype='int8').

---

Performance Highlights (Apple M5 Silicon)

Benchmarked on the full 15.08-minute (904.93s) JFK Inaugural Address:

---

Architecture Partitioning

• Conformer Encoder (cohere_encoder.mlpackage): Runs natively on the Apple Neural Engine (ANE), processing 20s of audio in ~190 ms while consuming 0% of the GPU.
• Transformer Prefill (cohere_decoder_prefill.mlpackage): INT8 weights (145 MB), computes prompt representation in ~30 ms on GPU.
• Autoregressive Decoder (cohere_decoder_decode.mlpackage): INT8 weights (129 MB) utilizing in-place Metal buffer mutation via MLState. Autoregressive single-step token latency drops from $1.68\text{ ms} \rightarrow 0.84\text{ ms}$ weight memory read time.

---

Quick Setup with Voxtype

To install and compile these INT8 CoreML models into Voxtype:

cd ~/voxtype
python3 scripts/setup_cohere_coreml.py --repo-id ctyau/cohere-transcribe-coreml-int8

This will automatically:

1. Download the .mlpackage bundles.
2. Compile them into native .mlmodelc bundles for your specific Mac.
3. Install them into ~/Library/Application Support/voxtype/models/cohere-transcribe-fp16/compiled_coreml.

---

Model Card & License

• Base Model: CohereForAI/c4ai-transcribe-2b
• License: CC-BY-NC-4.0 (inherits base model license)