Downloads · 30 days
17
13% of all-time downloads
TevunahAi/Qwen3-30B-A3B-Instruct-UltraHybrid-GPTQ-v2
Qwen3-30B-A3B-Instruct-UltraHybrid-GPTQ-v2 is a machine learning model from TevunahAi. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as apache-2.0.
- Base Model: Qwen3-30B-A3B-Instruct (30B parameters, 128 experts, 8 active per token) - Quantization: TevunahAi Ultra-Hybrid GPTQ + EoRA (Router-Optimized) - Compression: 60GB → 18-20GB (~70% reduction) - Quality: 99…
Downloads · 30 days
17
13% of all-time downloads
All-time downloads
135
Public
Parameters
30.5B
17.1 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors17.1 GB · 100%
How the weights are stored.
I3229.8B · 98%
From the Hugging Face model README
Router-Optimized Mixed-Precision + EoRA:
The router selects 8 of 128 experts per token - a critical decision:
Attention mechanisms are the "brain" of the model:
MoE experts benefit from aggressive compression:
EoRA is applied exclusively to attention layers for intelligent quantization error recovery:
Why only attention?
When loading this model, you may see warnings about ~60 unused expert weights. This is intentional and normal:
This is structured pruning + quantization - going beyond simple bit reduction to intelligently optimize the architecture itself.
Here's a real generation from this model explaining its own architecture:
Prompt: "Explain to me what autoregressive AI models with mixture of experts are?"
Response: The model generates a comprehensive, well-structured explanation covering:
Quality indicators:
This demonstrates the model maintains 99%+ quality even with 70% compression and EoRA-enhanced INT8 attention.
pip install gptqmodel torch transformers accelerate
# Install PyTorch with CUDA
pip install torch --index-url https://download.pytorch.org/whl/cu121
# Install gptqmodel with Marlin kernels
pip install gptqmodel --no-build-isolation
Note: Marlin kernels require:
from gptqmodel import GPTQModel
from transformers import AutoTokenizer
# Load model with Marlin acceleration
model = GPTQModel.from_quantized(
"TevunahAi/Qwen3-30B-A3B-Instruct-UltraHybrid-GPTQ-v2",
device="cuda:0",
trust_remote_code=True,
use_marlin=True, # Enable Marlin kernels for 2-4x speedup
)
tokenizer = AutoTokenizer.from_pretrained(
"TevunahAi/Qwen3-30B-A3B-Instruct-UltraHybrid-GPTQ-v2"
)
# Chat format
messages = [
{"role": "user", "content": "Explain quantum computing to a 10 year old."}
]
text = tokenizer.apply_chat_template(
messages,
tokenize=False,
add_generation_prompt=True
)
inputs = tokenizer(text, return_tensors="pt").to("cuda")
# Generate response
outputs = model.generate(
**inputs,
max_new_tokens=512,
temperature=0.7,
top_p=0.9,
repetition_penalty=1.1,
pad_token_id=tokenizer.eos_token_id
)
response = tokenizer.decode(
outputs[0][inputs['input_ids'].shape[1]:],
skip_special_tokens=True
)
print(response)
temperature=0.7
top_p=0.9
top_k=40
repetition_penalty=1.1
temperature=1.2
top_p=0.95
repetition_penalty=1.15
do_sample=False
temperature=0.0
temperature=0.3
top_p=0.85
top_k=20
| Component | Precision | Parameters | EoRA | Reasoning |
|---|---|---|---|---|
| Router | FP16 | ~5M | No | Critical path - expert selection |
| Attention | INT8 | ~8B | Yes (Rank-128) | Quality preservation + error correction |
| Experts | INT4 | ~20B | No | Aggressive compression (sparse) |
| EoRA Adapters | FP16 | ~300M | N/A | Attention error correction only |
You may see these warnings - they are normal and safe to ignore:
WARNING: Some weights were not used when initializing Qwen3MoeForCausalLM
Why: Router-based pruning removed ~60 low-activation experts. This is intentional optimization.
WARNING: The class 'Qwen3MoeForCausalLM' is not registered
Why: Custom model architecture. Set trust_remote_code=True to resolve.
WARNING: model.rotary_emb.inv_freq was not used
Why: Qwen3 uses different positional encoding. No impact on generation.
# Enable CPU offloading
model = GPTQModel.from_quantized(
"TevunahAi/Qwen3-30B-A3B-Instruct-UltraHybrid-GPTQ-v2",
device_map="auto", # Automatic device placement
max_memory={0: "18GB", "cpu": "32GB"}
)
# Ensure Marlin kernels are enabled
python -c "from gptqmodel.nn_modules.qlinear.qlinear_marlin import QuantLinear; print('Marlin available!')"
# Check GPU utilization
nvidia-smi dmon -s u
# Reinstall with correct dependencies
pip uninstall gptqmodel -y
pip install gptqmodel --no-build-isolation
| Method | Tokens/Sec | Speedup |
|---|---|---|
| FP16 | 8-12 | 1.0x |
| Standard GPTQ | 12-18 | 1.5x |
| Marlin GPTQ | 20-40 | 3-4x |
This model inherits the license from the base model:
This quantization uses EoRA (Error-correcting Low-Rank Adaptation) developed by NVIDIA Research for improved quality retention through eigenspace low-rank approximation without requiring additional training.
Paper: EoRA: Training-free Compensation for Compressed LLM with Eigenspace Low-Rank Approximation
GitHub: https://github.com/NVlabs/EoRA
Authors: Shih-Yang Liu, Maksim Khadkevich, Nai Chit Fung, Charbel Sakr, Chao-Han Huck Yang, Chien-Yi Wang, Saurav Muralidharan, Hongxu Yin, Kwang-Ting Cheng, Jan Kautz, et al.
Implementation: This model applies EoRA adapters (rank-128) to attention layers only (Q, K, V, O projections across all 48 transformer layers = 192 adapters), recovering 1-2% quality vs standard GPTQ while maintaining efficient inference.
Citation:
@article{liu2024eora,
title={EoRA: Training-free compensation for compressed LLM with eigenspace low-rank approximation},
author={Liu, Shih-Yang and Khadkevich, Maksim and Fung, Nai Chit and Sakr, Charbel and Yang, Chao-Han Huck and Wang, Chien-Yi and Muralidharan, Saurav and Yin, Hongxu and Cheng, Kwang-Ting and Kautz, Jan and others},
journal={arXiv preprint arXiv:2410.21271},
year={2024}
}
If you use this model in your research or applications, please cite:
@software{tevunahai_qwen3_30b_ultrahybrid_2024,
author = {TevunahAi},
title = {Qwen3-30B-A3B-Instruct Ultra-Hybrid GPTQ v2 + EoRA},
year = {2024},
publisher = {HuggingFace},
url = {https://huggingface.co/TevunahAi/Qwen3-30B-A3B-Instruct-UltraHybrid-GPTQ-v2}
}
Quantized by TevunahAi
Professional AI Model Quantization - Where Precision Meets Performance
For questions, issues, or custom quantization requests, please open an issue or contact us directly.