Downloads · 30 days
31
0% of all-time downloads
TevunahAi/Nemotron-3-Nano-30B-A3B-GPTQ
Nemotron-3-Nano-30B-A3B-GPTQ is a text generation model from TevunahAi. Use it when you need the model to write or continue text. It is set up for gptqmodel. The card lists the license as other.
NVIDIA Nemotron-3-Nano uses one of the most sophisticated architectures available:
Downloads · 30 days
31
0% of all-time downloads
All-time downloads
6.9K
Public
Parameters
6.2B
19.8 GB on disk
Likes
2
Public
Click a slice to open those files.
.safetensors19.7 GB · 100%
How the weights are stored.
I323.7B · 60%
From the Hugging Face model README
| Property | Value |
|---|---|
| Base Model | NVIDIA Nemotron-3-Nano-30B-A3B-BF16 |
| Architecture | HYBRID (Mamba-2 + MoE + GQA) |
| Parameters | 30B total, ~3.5B active |
| Context Length | 128K (up to 1M supported) |
| Quantization | TevunahAi Ultra-Hybrid GPTQ |
| Original Size | 60.23 GB (BF16) |
| Quantized Size | 18.39 GB |
| Compression | 68.73% reduction |
NVIDIA Nemotron-3-Nano uses one of the most sophisticated architectures available:
23 Mamba-2 Layers: State Space Models with selective state spaces
23 MoE Layers: Mixture-of-Experts
6 GQA Attention Layers: Grouped Query Attention
TevunahAi Ultra-Hybrid Mixed-Precision:
| Component | Precision | Rationale |
|---|---|---|
| Router/Gate | FP32 | Maximum routing precision - critical for MoE |
| GQA Attention (q/k/v/o_proj) | INT8 | Quality preservation for attention |
| Mamba-2 Projections (in/out_proj) | INT4 | SSM layers |
| MoE Experts (128/layer) | INT4 | Aggressive compression |
| Shared Experts | INT4 | Standard compression |
| Embeddings & LM Head | BF16 | Preserved for accuracy |
| Benchmark | Score |
|---|---|
| AIME25 (with tools) | 99.2% |
| AIME25 (no tools) | 89.1% |
| LiveCodeBench | 68.3% |
| SWE-Bench | 38.8% |
| RULER-100@256k | 92.9% |
| RULER-100@1M | 86.3% |
Formal benchmarks pending - inference quality verified manually.
import gptqmodel.models.loader as gptq_loader
# Bypass version check for NemotronH (required for transformers > 4.48)
_original_check_versions = gptq_loader.check_versions
def patched_check_versions(model_class, require_pkgs_version):
if 'NemotronH' in str(model_class):
return
return _original_check_versions(model_class, require_pkgs_version)
gptq_loader.check_versions = patched_check_versions
from gptqmodel import GPTQModel
from transformers import AutoTokenizer
model = GPTQModel.load(
"TevunahAi/Nemotron-3-Nano-30B-A3B-GPTQ",
trust_remote_code=True
)
tokenizer = AutoTokenizer.from_pretrained(
"TevunahAi/Nemotron-3-Nano-30B-A3B-GPTQ",
trust_remote_code=True
)
# Generate
prompt = "The capital of France is"
output = model.generate(
**tokenizer(prompt, return_tensors='pt').to('cuda'),
max_new_tokens=100
)
print(tokenizer.decode(output[0]))
messages = [{"role": "user", "content": "Solve: What is 23 * 47?"}]
tokenized = tokenizer.apply_chat_template(
messages,
tokenize=True,
add_generation_prompt=True,
return_tensors="pt"
).to('cuda')
outputs = model.generate(
tokenized,
max_new_tokens=1024,
temperature=1.0, # Recommended for reasoning
top_p=1.0
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
tokenized = tokenizer.apply_chat_template(
messages,
tokenize=True,
enable_thinking=False, # Disable reasoning traces
add_generation_prompt=True,
return_tensors="pt"
).to('cuda')
outputs = model.generate(
tokenized,
max_new_tokens=32,
do_sample=False, # Greedy for non-reasoning
num_beams=1
)
pip install gptqmodel transformers>=4.48
pip install -U "vllm>=0.12.0"
vllm serve TevunahAi/Nemotron-3-Nano-30B-A3B-GPTQ \
--max-num-seqs 8 \
--tensor-parallel-size 1 \
--max-model-len 32768 \
--trust-remote-code
AutoModelForCausalLM.from_pretrained() may encounter compatibility issues with optimum/transformers GPTQ integration. Use GPTQModel as shown above.fix_mistral_regex=True| Specification | Value |
|---|---|
| Method | GPTQ with Ultra-Hybrid configuration |
| Quantizer | GPTQModel 5.6.12 |
| Time | 268.2 minutes (4.5 hours) |
| Calibration Samples | 2048 (8x industry standard) |
| Sequence Length | 4096 tokens |
| Bits Per Weight | 4.29 BPW estimated |
| desc_act | True (activation ordering) |
| sym | True (symmetric quantization) |
| group_size | 128 |
Ideal for:
| Specification | Value |
|---|---|
| Model Family | NVIDIA Nemotron-3 |
| Variant | Nano-30B-A3B (Hybrid) |
| Total Parameters | 30B |
| Active Parameters | ~3.5B |
| Total Layers | 52 |
| Mamba-2 Layers | 23 |
| MoE Layers | 23 |
| Attention Layers | 6 |
| Experts per MoE Layer | 128 (+1 shared) |
| Top-K Routing | 6 |
| Hidden Size | 2688 |
| Intermediate Size | 1856 |
| Context Length | 128K (default), 1M (max) |
| Vocab Size | 131072 |
| Supported Languages | EN, DE, ES, FR, IT, JA |
NVIDIA Open Model License
https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-license/
@software{nemotron_nano_gptq_2024,
title = {NVIDIA Nemotron-3-Nano-30B-A3B - TevunahAi Ultra-Hybrid GPTQ},
author = {TevunahAi},
year = {2024},
note = {First GPTQ quantization of hybrid Mamba-2 + MoE + GQA architecture},
url = {https://huggingface.co/TevunahAi/Nemotron-3-Nano-30B-A3B-GPTQ}
}
@misc{nvidia_nemotron_nano_v3_2025,
title = {Nemotron 3 Nano: Open, Efficient MoE Hybrid Mamba-Transformer},
author = {NVIDIA},
year = {2025},
url = {https://research.nvidia.com/labs/nemotron/files/NVIDIA-Nemotron-3-Nano-Technical-Report.pdf}
}
This quantization required patching GPTQModel's NemotronH model definition to properly handle:
Special thanks to the GPTQModel team for their excellent quantization framework.
Quantized by TevunahAi
Professional AI Model Quantization Service
Specialized in hybrid architectures (Mamba, MoE, SSM)