whisper-large-v3-turbo-a16w8

Whisper Large v3 Turbo A16W8: Optimized for SiMa.ai Modalix

Compatibility: This model is compatible with Neat/LLiMa v0.6.1. For Neat/LLiMa v0.5.0, download simaai/whisper-large-v3-turbo-a16w8 at tag v0.5.0 instead.

hf download simaai/whisper-large-v3-turbo-a16w8 --revision v0.5.0 \
  --local-dir whisper-large-v3-turbo-a16w8-v0.5.0

Overview

This repository contains a precompiled and quantized version of OpenAI Whisper Large v3 Turbo, optimized for execution on the SiMa.ai Modalix platform. Large v3 Turbo retains the 32-layer Large v3 encoder and uses a pruned 4-layer decoder for faster generation.

This repository is not a standard Transformers checkpoint. The elf_files/ and devkit/ directories contain compiled artifacts consumed by the SiMa.ai Neat runtime on Modalix.

Prerequisites

To use this model, you need:

1. A SiMa.ai Modalix device.
2. The SiMa.ai Neat Runtime installed or updated on Modalix.
3. The Hugging Face CLI, optionally, for downloading the model on a host before copying it to Modalix.

Follow the SiMa.ai Neat getting started guide to install or update the Neat Runtime.

Installation and Deployment

Download the Model

On Modalix, download the precompiled model with the Hugging Face CLI:

hf download simaai/whisper-large-v3-turbo-a16w8 \
  --local-dir /media/nvme/llima/models/whisper-large-v3-turbo-a16w8

The model is installed under:

/media/nvme/llima/models/whisper-large-v3-turbo-a16w8

Alternatively, download it on a host and copy it to Modalix:

hf download simaai/whisper-large-v3-turbo-a16w8 \
  --local-dir whisper-large-v3-turbo-a16w8

scp -r whisper-large-v3-turbo-a16w8 \
  sima@<modalix-ip>:/media/nvme/llima/models/

Replace <modalix-ip> with the IP address or hostname of the Modalix device.

Usage

This model is intended for applications built with the SiMa.ai Neat GenAI APIs. Use ASRModel for direct, in-process transcription or GenAIServer when clients access the model over HTTP.

Direct Application Integration

The Neat ASRModel API provides the lowest-overhead path for speech transcription inside an application:

#include "neat/genai.h"

#include <iostream>

int main() {
    simaai::neat::genai::ASRModel model(
        "/media/nvme/llima/models/whisper-large-v3-turbo-a16w8");

    simaai::neat::genai::GenerationRequest request;
    request.audio_file = "speech.wav";
    request.language = "en";

    auto result = model.run(request);
    std::cout << result.text << "\n";
}

For the complete C++ and Python workflows, including audio tensors and streaming, see GenAI Model.

Serve with the Neat GenAI Server

Use GenAIServer when a browser, service, or remote client needs to access the model through an HTTP API. The packaged Neat tutorial can serve this ASR model by itself:

./lib/sima-neat/tutorials/tutorial_021_serve_genai_models \
  --asr /media/nvme/llima/models/whisper-large-v3-turbo-a16w8

Verify that the model was registered:

curl http://<modalix-ip>:9998/v1/models

Send an audio transcription request with the tutorial client:

python3 share/sima-neat/tutorials/021_serve_genai_models/request_audio_transcription.py \
  --server-ip <modalix-ip> \
  --model asr \
  speech.wav

See Serve GenAI Models for the complete C++, Python, and HTTP workflow.

Recompile

The repository includes the Bash wrapper and Python compiler utility used for the existing Whisper compilation path. Run them from a Model Compiler environment:

hf download openai/whisper-large-v3-turbo \
  --local-dir models--openai--whisper-large-v3-turbo

bash compile.sh

Set NUM_PROCESSES to change the default of 16 compiler workers. Additional arguments are forwarded to gen_models--openai--whisper.py.

Artifact Layout

devkit/                         Runtime configuration, tokenizer, and embeddings
elf_files/                      Compiled Modalix accelerator programs
compile.sh                      Compilation entry point
gen_models--openai--whisper.py  Whisper compilation utility

The compilation scripts are included for provenance and reproducibility. They are not required when using the precompiled artifacts.

Evaluation

No accuracy or performance results specific to this A16W8 compiled artifact are currently reported. Results on the source model card describe the unquantized upstream checkpoint and should not be treated as measurements of this build.

Limitations

• These artifacts run on SiMa.ai Modalix and cannot be loaded with transformers.AutoModel.
• The runtime processes an audio window of up to 30 seconds. Segment longer recordings before inference.
• This build produces text without word- or segment-level timestamps.
• Speaker diarization is not provided.
• Quantization can cause minor differences from the full-precision source model.
• The pruned Turbo decoder can have minor quality degradation compared with Whisper Large v3.
• Whisper can hallucinate text during silence or noisy audio and performs unevenly across languages, accents, and domains. Evaluate the model on representative data before production deployment.

Resources

• SiMa.ai Neat
• GenAI Model API
• Serve GenAI Models
• GenAI with LLiMa
• OpenAI Whisper Large v3 Turbo
• Whisper paper

Citation

@misc{radford2022whisper,
  doi       = {10.48550/ARXIV.2212.04356},
  url       = {https://arxiv.org/abs/2212.04356},
  author    = {Radford, Alec and Kim, Jong Wook and Xu, Tao and
               Brockman, Greg and McLeavey, Christine and Sutskever, Ilya},
  title     = {Robust Speech Recognition via Large-Scale Weak Supervision},
  publisher = {arXiv},
  year      = {2022}
}