whisper-large-v2-fp16

whisper-large-v2-fp16

openai/whisper-large-v2 stored in fp16 instead of fp32, ready to serve with vLLM. The repo holds both the converted weights and the scripts that produced and verified them.

The checkpoint is bit-exact with the upstream fp32 release — see Why this is lossless.

Why this is lossless

OpenAI released Whisper large-v2 in fp16; the Hugging Face float32 checkpoint is an exact upcast of those fp16 weights. Every one of the 1,543,304,960 values in the upstream file has its low 13 mantissa bits set to zero, so rounding back to fp16 loses nothing:

• max absolute weight error: 0.0
• max relative weight error: 0.0
• values overflowing the fp16 range (|x| > 65504): 0 (largest weight magnitude is 4.64)
• values underflowing to zero: 0
• logits on identical input: identical to the fp32 model, same argmax tokens

Full numbers are in fp16_conversion_report.json. Halving the file removes storage and load time, not accuracy — but note this holds for this model because upstream was fp16 to begin with; do not assume it for other checkpoints.

Serving with vLLM

vLLM reads torch_dtype from config.json, so --dtype auto loads fp16 directly instead of upcasting to fp32.

vllm serve HTXDSAI/whisper-large-v2-fp16 --dtype auto

Then use the OpenAI-compatible transcription endpoint:

curl http://localhost:8000/v1/audio/transcriptions \
  -F "model=HTXDSAI/whisper-large-v2-fp16" \
  -F "[email protected]" \
  -F "language=en"

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
with open("audio.ogg", "rb") as f:
    print(client.audio.transcriptions.create(model="HTXDSAI/whisper-large-v2-fp16", file=f).text)

Whisper is a transcription-only model in current vLLM: serve it and call /v1/audio/transcriptions (or /v1/audio/translations) rather than the text completions API.

This checkpoint was smoke-tested end-to-end on vLLM 0.16.0 (CUDA 12.8, RTX 4090): the engine reported dtype=torch.float16 straight from config.json, loaded the weights in 0.30 s, and transcribed the mary_had_lamb sample correctly.

Using with transformers

import torch
from transformers import WhisperForConditionalGeneration, WhisperProcessor

model = WhisperForConditionalGeneration.from_pretrained(
    "HTXDSAI/whisper-large-v2-fp16", dtype=torch.float16
).to("cuda")
processor = WhisperProcessor.from_pretrained("HTXDSAI/whisper-large-v2-fp16")

fp16 needs a GPU — CPU inference in fp16 is unsupported or emulated in PyTorch. Load with dtype=torch.float32 to run on CPU.

Reproducing the conversion

The scripts use uv. uv run creates the environment from pyproject.toml/uv.lock on first use, pulling CPU-only torch (~250 MB) since the cast needs no GPU.

# fp32 from the hub -> fp16 in this directory (~10 s once the source is cached)
uv run scripts/convert_to_fp16.py

# structure, dtype, key-set and bit-exactness checks against the fp32 source
uv run scripts/verify_fp16.py

# ...plus a forward pass on both checkpoints, comparing logits
uv run --group verify scripts/verify_fp16.py --logits-parity

Useful flags:

What the conversion does

scripts/convert_to_fp16.py streams the source model.safetensors tensor by tensor and:

1. audits each float tensor against the fp16 range and aborts if any weight would become inf (a single overflowed weight yields NaN logits at inference time);
2. casts float tensors to fp16, passing integer/bool tensors through untouched;
3. copies every non-weight file byte-for-byte, so the tokenizer, feature extractor and generation config are exactly upstream's;
4. edits only config.json, setting torch_dtype to float16;
5. asserts every file vLLM needs is present, then writes fp16_conversion_report.json.

Files

License and attribution

Weights are openai/whisper-large-v2, Apache-2.0, unmodified apart from the dtype cast. Model behaviour, languages, training data and limitations are documented in the upstream model card.