Downloads · 30 days
6
12% of all-time downloads
rudrakshrakeshzodage/distil-whisper-large-v3-pytorch-q4
distil-whisper-large-v3-pytorch-q4 is a automatic speech recognition model from rudrakshrakeshzodage. Use it when you need speech turned into text. It is set up for transformers. The card lists the license as apache-2.0.
This model is a quantized/optimized version of the baseline Distil-Whisper (distil-large-v3) model. It is part of a benchmarked suite of quantized models evaluated on local hardware.
Downloads · 30 days
6
12% of all-time downloads
All-time downloads
49
Public
Parameters
756M
504 MB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors502 MB · 99%
How the weights are stored.
U8682M · 90%
From the Hugging Face model README
This model is a quantized/optimized version of the baseline Distil-Whisper (distil-large-v3) model. It is part of a benchmarked suite of quantized models evaluated on local hardware.
[!NOTE] This model is part of a suite of optimized/quantized versions of the base model. Other variants in this direction:
- PyTorch Q4 (GPU Quantized - NF4):
rudrakshrakeshzodage/distil-whisper-large-v3-pytorch-q4<span style="color: #3b82f6;">(Current)</span>- CTranslate2 INT8 (CPU Quantized):
rudrakshrakeshzodage/distil-whisper-large-v3-ct2-int8
Base model: distil-whisper/distil-large-v3
Below is the comparative performance table generated empirically using 8 CPU threads / cores:
| Backend | Precision | Device | Model Size (GB) | Mean Latency (s) | Throughput (Words/s) | RTF | WER (%) | Peak RAM (GB) | Peak VRAM (GB) |
|---|---|---|---|---|---|---|---|---|---|
| PYTORCH | FP32 | CUDA | 3.024 | 0.427 | 44.81 | 0.064 | 5.29% | 3.62 | 2.98 |
| PYTORCH | FP16 | CUDA | 1.512 | 0.189 | 100.25 | 0.028 | 5.29% | 2.12 | 1.46 |
| PYTORCH | INT8 | CUDA | 0.756 | 0.293 | 64.00 | 0.042 | 5.29% | 1.92 | 0.85 |
| PYTORCH | Q4 | CUDA | 0.378 | 0.377 | 50.63 | 0.056 | 5.29% | 1.93 | 0.60 |
| GGML | FP16 | CPU | 1.409 | 17.269 | 1.14 | 2.633 | 5.29% | 4.14 | 0.00 |
| GGML | INT8 | CPU | 1.409 | 9.271 | 2.18 | 1.441 | 5.29% | 1.50 | 0.00 |
| GGML | Q5 | CPU | 1.409 | 16.735 | 1.18 | 2.566 | 5.29% | 2.28 | 0.00 |



import torch
from transformers import AutoModelForSpeechSeq2Seq, AutoProcessor, BitsAndBytesConfig
model_id = "rudrakshrakeshzodage/distil-whisper-large-v3-pytorch-q4"
quant_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type="nf4",
bnb_4bit_use_double_quant=True,
bnb_4bit_compute_dtype=torch.float16
)
model = AutoModelForSpeechSeq2Seq.from_pretrained(
model_id,
quantization_config=quant_config,
device_map="cuda"
)
processor = AutoProcessor.from_pretrained(model_id)
Quantization research and benchmarking by RudrakshRakeshZodage.