Downloads · 30 days
14
4% of all-time downloads
pavanperi/sarvam-30b-awq-w4a16-broad-5120
sarvam-30b-awq-w4a16-broad-5120 is a text generation model from pavanperi. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as apache-2.0.
<img src="https://atos.net/content/assets/global-images/atos-logo-blue-2023.svg" alt="Atos Logo" width="200" /
Downloads · 30 days
14
4% of all-time downloads
All-time downloads
324
Public
Parameters
6.1B
19.8 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors19.8 GB · 100%
How the weights are stored.
I3230B · 93%
From the Hugging Face model README
In your fresh python environment, use :
pip install vllm==0.15.0
python3 hotpatch_vllm.py
python3 patch_compressed.py --moe-kernel marlin
Then serve the model with :
vllm serve --config vllm_config.yaml
We are not using A100 GPU as we heard that Marlin could have problems with the A100 architecture. Contact us if that is the case.
The key idea is to use real activations from a calibration dataset to quantize the targeted weight tensors in a smart way.
We tried many other compression techniques :
In practice, AWQ does the following.
W4A16 means that weights are quantized to 4 bits, while activations remain at 16-bit precision. That matters because it gives a good tradeoff for LLM inference:
[!WARNING] On the model card on HuggingFace, it shows 6B parameters, but it is not the case, no parameter was cut-off, they were just quantized to int4 at most
MoE models are not just bigger dense transformers. They have token-dependent specialist paths.
That creates two important constraints.
AWQ is useful here because it uses observed activations to preserve the most sensitive directions better than naive low-bit rounding.
After different trial-and-error processes, we concluded that we should limit ourselves to quantizing Linear layers, which means AWQ is allowed to quantize linear modules across the model rather than only the routed experts.
It includes
attention.query_key_value and attention.denseBut we did not compress
lm_head because it directly controls the final token logitsThe calibration dataset is not training data, and it is not fine-tuning. Its purpose is to show AWQ realistic activation patterns so weight quantization can be adapted to the way the model is actually used.
For Sarvam, calibration quality matters a lot because:
If the calibration set is too narrow, the quantizer protects the wrong activation ranges and the model can become less coherent, less stable across languages, or more brittle on certain tasks.
The original master design used a 512-sample benchmark-aware multilingual calibration set, but we expanded it to 5,120 samples due to significant performance degradation. While 512 samples is generally considered the optimal size for a standard dense model, Mixture-of-Experts (MoE) architectures require a much larger calibration set to properly align all experts. Although an even larger dataset would likely be preferable, memory constraints on our hardware limited us to 5,120 samples
| Task Category | Language / Source Split | Count | Direct Repository Link |
|---|---|---|---|
| Math | English (en): 430 samples<br><br> <br><br>French (fr): 430 samples<br><br> <br><br>Indic (10 scripts): 420 samples | 1280 | juletxara/mgsm |
| Medical | English (en): 430 samples<br><br> <br><br>French (fr): 430 samples<br><br> <br><br>Indic scripts: 420 samples | 1280 | Medical-Corpus |
| Writing | English (en): 430 samples<br><br> <br><br>French (fr): 430 samples<br><br> <br><br>Indic scripts: 420 samples | 1280 | wikimedia/wikipedia |
| Questions | English (en): 420 samples<br><br> <br><br>French (fr): 420 samples<br><br> <br><br>Indic scripts: 440 samples | 1280 | aya dataset |
| TOTAL | 25% English / 25% French / 50% Indic | 5120 |
On our hardware, the model achieved over 50% energy savings and reduced memory usage (under 20GB vs. 120GB VRAM), while retaining more than 81% of the base model’s capabilities based on our internal benchmarks. Energy savings are highly dependent on the underlying kernel, which is more or less optimized for W4A16 compression and not compatible with all hardware types.