whisper-small-ternary

whisper-small-ternary

192 of this model's weight matrices contain only −1, 0 and +1. Normalised WER goes down against stock whisper-small. Punctuation and multilinguality go away.

Two caveats belong here, not further down.

This is a comparison against stock whisper-small. A matched fp16 control — same code path, same data, same schedule, quantisation off — is still ≈0.37 pp better than the ternary branch, and both curves were still descending when the runs ended. The model beats the teacher it was distilled from; that is not the same claim as "ternarisation improved it".

The WER is normalised — case and punctuation are stripped before scoring, so a model can score better while producing worse text for a human. That is what happened here.

Bits, precisely

Of the 140 MiB, 53 MiB is the ternary matrices and 87 MiB is everything else — the tied embedding head plus convolutions, norms and biases stay fp16.

At runtime it is a hybrid by design: the encoder is expanded Q2_0 → Q8_0 on load (exact — ternary codes and the same fp16 scale are representable in Q8_0) because the encoder is compute-bound, while the decoder stays 2-bit. So the encoder is mathematically ternary and physically 8-bit while running.

Code, training recipe and every measurement file: github.com/AubakirovArman/whisper-ternary

---

Files

The dense file is not a different model. It is the ternary codes multiplied by their learned scales, written out as ordinary weights so you can reproduce the WER numbers with transformers and confirm they match.

Run on CPU

Q2_0 exists in ggml but had never been exercised end to end, so whisper.cpp needs a patch — it is four small changes and it is in the repo:

git clone https://github.com/ggml-org/whisper.cpp && cd whisper.cpp
git apply ../whisper-ternary/deploy/whisper_cpp_q2_0.patch
cmake -B build && cmake --build build -j

./build/bin/whisper-cli -m ggml-small-wal-ternary-q2_0.bin -f audio.wav -ng

As a service:

./build/bin/whisper-server -m ggml-small-wal-ternary-q2_0.bin \
    --host 127.0.0.1 --port 8178 -t 16 -ng -ml 100000
curl 127.0.0.1:8178/inference -F [email protected] -F response_format=json

Pass -ml 100000. Without it the server forces 60-character segments and splits words in half (fl uttered).

Verify in PyTorch

from transformers import WhisperForConditionalGeneration, WhisperProcessor

model = WhisperForConditionalGeneration.from_pretrained("armanibadboy/whisper-small-ternary")
proc  = WhisperProcessor.from_pretrained("armanibadboy/whisper-small-ternary")

Every quantised matrix holds at most 3 distinct values per group of 128 — check it yourself:

import torch
W = model.model.encoder.layers[0].fc1.weight
g = W[:, :128]
print(torch.unique(g / g.abs()[g != 0].min()).round())   # -> tensor([-1., 0., 1.])

Speed

Same weights, 66 s of audio, 16 threads, batch 1, Xeon:

Against an H200 running the same weights in PyTorch, a single decoder step is faster on the CPU — 6.0 ms against 8.3 ms — because at batch 1 the GPU is launch-bound (439 kernel launches per decoder pass, ~15 µs each) rather than compute-bound. But end to end the H200 still wins by 2.3× (1.17 s against 2.7 s for the same 66 s of audio): the encoder is a dense matmul over 1500 frames and there the GPU is 77× ahead. Note this compares runtimes as well as hardware — hand-written C/AVX2 against PyTorch.

Limitations

Stated plainly, because a quantisation result without a failure list has not been examined hard enough.

• No punctuation. 0.0–0.7 % of utterances contain a comma depending on mode; stock whisper-small manages 63.3 %. Capitals appear on 2 % against 98 %. In timestamp mode a sentence-final period survives on 15 % of utterances — residue from the original model, not learned punctuation. Measured on 300 utterances.
• English only. Russian degrades by 12.6×. An fp16 control trained identically degrades the same way, so the cause is monolingual fine-tuning, not ternarisation.
• Out of domain, English is roughly twice as bad as on LibriSpeech.
• The tied embedding head stays fp16 — 18 % of parameters, 86 of the 140 MiB. A one-shot ternary projection of it, with no dedicated QAT, takes WER from 2.71 % to 1152 %. That shows this projection fails on that tensor, not that the head is incompressible; a learned Q4 head would plausibly bring the file to ~78 MiB.
• Load takes 950 ms against 447 ms for fp16 — the encoder expansion at load is an unresolved 826 ms. Three hypotheses were tested and all three were wrong.
• Trained and evaluated on read speech (LibriSpeech). Conversational and noisy audio were not measured.

What ternarisation cost

Beating stock whisper-small answers "is this model good", not "what did quantisation cost" — this model was also fine-tuned. So an fp16 control was trained on the same code path, same data, same schedule, quantisation off:

• the gap holds at ≈0.37 pp absolute on test-clean and does not narrow;
• neither branch reached a floor — both still falling nearly linearly when the runs ended (−0.173 and −0.186 pp per 10k steps);
• so "ternary gets to the same place, just later" is unproven. It needs a run to plateau, which was not done.

Training

Codes are initialised by an activation-weighted projection, not by rounding: per group of 128 inputs we solve for the codes and scale minimising the error weighted by how large the activations multiplying them actually are.

One trap worth naming, specific to this design: in classical QAT the high-precision master weights are genuinely the source the deployed checkpoint is rebuilt from. Here they are not — the group scales are separately learned parameters, so re-projecting from the latent tensor recomputes scales that were never trained, and the latent tensor is only a gradient surrogate free to drift outside its quantisation cells. Re-projecting from it produced 1000.56 % WER. The deployment representation is the trained (codes, scales) pair, from the start.

Evaluation protocol

batch_size     = 16
max_new_tokens = 440
greedy, num_beams = 1

Batch size shifts WER by 0.013 pp on identical code (3.4298 % / 3.4224 % / 3.4168 % at batch 16 / 8 / 1) — systematic implementation sensitivity, not random noise. Batch size is therefore pinned and differences under 0.02 pp are not treated as signal. No confidence intervals are reported yet; the test-other gap in particular has not been shown to exceed its uncertainty.

test-clean and test-other are selection sets, not sealed tests. They took no part in gradient training, but were measured 21 times across four runs, and this checkpoint was chosen over its own step-7500 sibling by looking at them. A publication-grade claim needs a fresh one-shot set; that has not been done.

The CPU path was verified independently: all 2620 utterances of test-other through patched whisper.cpp give 7.3754 %, against 7.3817 % in the PyTorch harness — within the noise floor.

Citation

@software{aubakirov2026ternarywhisper,
  author = {Aubakirov, Arman},
  title  = {Ternary Whisper: whisper-small at 2.125 bits per weight},
  year   = {2026},
  url    = {https://github.com/AubakirovArman/whisper-ternary}
}

MIT.