whisper-small-a16w8

Whisper Small A16W8: Optimized for SiMa.ai Modalix

Compatibility: This model is only compatible with Neat/LLiMa v0.5.0. For older Neat/LLiMa versions, download simaai/whisper-small-a16w8 at tag v0.4.0 instead.

hf download simaai/whisper-small-a16w8 --revision v0.4.0 \
  --local-dir whisper-small-a16w8-v0.4.0

Overview

This repository contains a precompiled and quantized version of OpenAI Whisper Small, optimized for execution on the SiMa.ai Modalix platform.

This repository is not a standard Transformers checkpoint. The elf_files/ and devkit/ directories contain compiled artifacts consumed by the SiMa.ai Neat runtime on Modalix.

Prerequisites

To use this model, you need:

1. A SiMa.ai Modalix device.
2. The SiMa.ai Neat Runtime installed or updated on Modalix.
3. The Hugging Face CLI, optionally, for downloading the model on a host before copying it to Modalix.

Follow the SiMa.ai Neat getting started guide to install or update the Neat Runtime.

Installation and Deployment

Download the Model

On Modalix, download the precompiled model with the LLiMa model manager:

llima pull whisper-small-a16w8

The model is installed under:

/media/nvme/llima/models/whisper-small-a16w8

Alternatively, download it on a host and copy it to Modalix:

hf download simaai/whisper-small-a16w8 \
  --local-dir whisper-small-a16w8

scp -r whisper-small-a16w8 \
  sima@<modalix-ip>:/media/nvme/llima/models/

Replace <modalix-ip> with the IP address or hostname of the Modalix device.

Usage

This model is intended for applications built with the SiMa.ai Neat GenAI APIs. Use ASRModel for direct, in-process transcription or GenAIServer when clients access the model over HTTP.

Direct Application Integration

The Neat ASRModel API provides the lowest-overhead path for speech transcription inside an application:

#include "neat/genai.h"

#include <iostream>

int main() {
    simaai::neat::genai::ASRModel model(
        "/media/nvme/llima/models/whisper-small-a16w8");

    simaai::neat::genai::GenerationRequest request;
    request.audio_file = "speech.wav";
    request.language = "en";

    auto result = model.run(request);
    std::cout << result.text << "\n";
}

For the complete C++ and Python workflows, including audio tensors and streaming, see GenAI Model.

Serve with the Neat GenAI Server

Use GenAIServer when a browser, service, or remote client needs to access the model through an HTTP API. The packaged Neat tutorial can serve this ASR model by itself:

./lib/sima-neat/tutorials/tutorial_021_serve_genai_models \
  --asr /media/nvme/llima/models/whisper-small-a16w8

Verify that the model was registered:

curl http://<modalix-ip>:9998/v1/models

Send an audio transcription request with the tutorial client:

python3 share/sima-neat/tutorials/021_serve_genai_models/request_audio_transcription.py \
  --server-ip <modalix-ip> \
  --model asr \
  speech.wav

See Serve GenAI Models for the complete C++, Python, and HTTP workflow.

Artifact Layout

devkit/                         Runtime configuration, tokenizer, and embeddings
elf_files/                      Compiled Modalix accelerator programs
compile.sh                      Compilation entry point
gen_models--openai--whisper.py  Whisper compilation utility

The compilation scripts are included for provenance and reproducibility. They are not required when using the precompiled artifacts.

Evaluation

No accuracy or performance results specific to this A16W8 compiled artifact are currently reported. Results on the source model card describe the unquantized upstream checkpoint and should not be treated as measurements of this build.

Limitations

• These artifacts run on SiMa.ai Modalix and cannot be loaded with transformers.AutoModel.
• The runtime processes an audio window of up to 30 seconds. Segment longer recordings before inference.
• This build produces text without word- or segment-level timestamps.
• Speaker diarization is not provided.
• Quantization can cause minor differences from the full-precision source model.
• Whisper can hallucinate text during silence or noisy audio and performs unevenly across languages, accents, and domains. Evaluate the model on representative data before production deployment.

Resources

• SiMa.ai Neat
• GenAI Model API
• Serve GenAI Models
• GenAI with LLiMa
• OpenAI Whisper Small
• Whisper paper

Citation

@misc{radford2022whisper,
  doi       = {10.48550/ARXIV.2212.04356},
  url       = {https://arxiv.org/abs/2212.04356},
  author    = {Radford, Alec and Kim, Jong Wook and Xu, Tao and
               Brockman, Greg and McLeavey, Christine and Sutskever, Ilya},
  title     = {Robust Speech Recognition via Large-Scale Weak Supervision},
  publisher = {arXiv},
  year      = {2022}
}