Downloads · 30 days
14
27% of all-time downloads
mastersubhajit/sarvam-30b-AWQ-GPTQ-Hybrid
sarvam-30b-AWQ-GPTQ-Hybrid is a machine learning model from mastersubhajit. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as other.
This model is a quantized version of the Sarvam-30B multilingual Mixture-of-Experts (MoE) large language model. It was compressed using a novel hybrid AWQ–GPTQ (W4A16) quantization technique at the National Electronic…
Downloads · 30 days
14
27% of all-time downloads
All-time downloads
51
Public
Parameters
6.3B
20 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors20 GB · 100%
How the weights are stored.
I3229.9B · 93%
From the Hugging Face model README
This model is a quantized version of the Sarvam-30B multilingual Mixture-of-Experts (MoE) large language model. It was compressed using a novel hybrid AWQ–GPTQ (W4A16) quantization technique at the National Electronics and Computer Technology Center (NECTEC), Thailand, in collaboration with the Asian Institute of Technology (AIT).
The primary goal of this model is to substantially reduce the GPU memory and storage footprint required to deploy the 32-billion parameter MoE model, while rigorously preserving its multilingual reasoning, instruction-following, and mathematical capabilities.
mastersubhajit)sarvam_moe (Mixture-of-Experts)llm-compressor.Unlike standard dense models, the Sarvam-30B architecture employs a sparse MoE structure that requires specialized handling to preserve routing integrity and output quality.
The model applies a sequential llm-compressor recipe:
To maintain stability and reasoning performance, not all layers were converted to 4-bit. Sensitive components were explicitly retained in BF16:
mlp.gate): Retained in BF16 to ensure that the expert selection logic and internal computational paths remain intact.The model was calibrated on a task-diverse mixed corpus of 1,024 samples (max sequence length of 4,096 tokens). To solve the issue of incomplete expert calibration in MoE models, hidden states were forwarded through every routed expert during calibration to ensure comprehensive activation coverage. The calibration corpus included:
The hybrid model was evaluated alongside the original BF16 model and a standard AWQ 4-bit variant using a strict conditional log-probability scoring method (to eliminate generation/sampling variance) on the LANTA HPC system.
| Benchmark | Configuration | Hybrid W4A16 Score (%) | BF16 Reference Score (%) |
|---|---|---|---|
| MMLU | 57 English subjects | 42.83 | 43.40 |
| GPQA Diamond | English | 28.65 | 29.67 |
| MMLU-Indic | Hindi, 5-shot | 50.76 | 52.10 |
| MMLU-Indic | English, 5-shot | 61.97 | 63.80 |
| ARC-Indic | Hindi, 25-shot | 75.88 | 76.21 |
| ARC-Indic | English, 25-shot | 82.40 | 83.49 |
| BBH | 3-shot exact match | 57.90 | 63.22 |
| GSM8K | 5-shot direct answering | 66.73 | 72.45 |
| GSM8K | 8-shot chain-of-thought | 77.56 | 72.71* |
| ARC-Challenge | Multiple-choice accuracy | 26.81 | 29.11 |
(Note: The chain-of-thought GSM8K result for the quantized models exceeded the BF16 baseline under this specific evaluation protocol, likely due to prompt sensitivity or evaluation variance rather than an intrinsic improvement in mathematical reasoning.)
This model is intended for deployment using an OpenAI-compatible server (such as vLLM) and is fully compatible with batched evaluation and streaming generation.
During testing on the LANTA HPC infrastructure, the model successfully demonstrated its ability to:
This model is highly suitable for research, institutional environments, and consumer deployments where GPU memory is constrained, but the powerful multilingual and reasoning capabilities of the 30B Sarvam MoE architecture are required.
Developed by Subhajit Ghosh at the National Electronics and Computer Technology Center (NECTEC), Thailand.