Downloads · 30 days
20
3% of all-time downloads
hupakabras/Nemotron-3-Nano-30B-A3B-bnb-4bit
Nemotron-3-Nano-30B-A3B-bnb-4bit is a text generation model from hupakabras. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as other.
Pre-quantized BitsAndBytes NF4 version of NVIDIA Nemotron-3-Nano-30B-A3B for QLoRA fine-tuning on consumer GPUs (16GB+).
Downloads · 30 days
20
3% of all-time downloads
All-time downloads
779
Public
Parameters
31.6B
31 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors30.9 GB · 100%
How the weights are stored.
BF168.4B · 95%
From the Hugging Face model README
Pre-quantized BitsAndBytes NF4 version of NVIDIA Nemotron-3-Nano-30B-A3B for QLoRA fine-tuning on consumer GPUs (16GB+).
Includes full quant_state metadata — loads directly without CPU re-quantization.
The original BF16 model requires 55GB download + 90GB RAM for on-the-fly quantization. This pre-quantized version:
| Original BF16 | This NF4 | |
|---|---|---|
| Download size | 55 GB | 18 GB |
| RAM during load | 90 GB peak | 15 GB |
| GPU VRAM | 14.2 GB | 14.1 GB |
| Load time | ~5 min (CPU quantization) | ~17 sec |
| CPU usage during load | 80% | ~0% |
Nemotron-3-Nano-30B-A3B is a hybrid Mamba-2 + Transformer MoE model:
bnb_4bit_quant_type="nf4")unsloth/Nemotron-3-Nano-30B-A3Bfrom transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
import torch
model = AutoModelForCausalLM.from_pretrained(
"hupakabras/Nemotron-3-Nano-30B-A3B-bnb-4bit",
device_map="auto",
torch_dtype=torch.bfloat16,
trust_remote_code=True,
)
tokenizer = AutoTokenizer.from_pretrained(
"hupakabras/Nemotron-3-Nano-30B-A3B-bnb-4bit",
trust_remote_code=True,
)
⚠️ Required monkey-patches for training on Windows / consumer GPUs.
transformers==4.56.2 # NOT 5.x (meta tensor regression)
bitsandbytes==0.47.0
peft>=0.13.0
trl>=0.22.0
_initialize_weights skip (conflict with quantized Byte weights)Params4bit._is_hf_initialized argument mismatchLinear4bit._save_to_state_dict meta tensor safeQuantState.to meta tensor safetrust_remote_code auto-trustindex_add_ dtype fix (BF16 vs Float mismatch in modeling_nemotron_h.py)from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import LoraConfig, get_peft_model
from trl import SFTTrainer, SFTConfig
import torch
# Manual device_map: 26+26 layers across 2 GPUs (9.7GB + 10.3GB)
# IMPORTANT: device_map="auto" + max_memory does NOT work with BnB 4-bit!
dm = {"backbone.embeddings": 0, "backbone.norm_f": 1, "lm_head": 1}
for i in range(26): dm[f"backbone.layers.{i}"] = 0
for i in range(26, 52): dm[f"backbone.layers.{i}"] = 1
# Load pre-quantized (17s, no CPU re-quantization)
model = AutoModelForCausalLM.from_pretrained(
"hupakabras/Nemotron-3-Nano-30B-A3B-bnb-4bit",
device_map=dm, torch_dtype=torch.bfloat16, trust_remote_code=True,
)
tokenizer = AutoTokenizer.from_pretrained(
"hupakabras/Nemotron-3-Nano-30B-A3B-bnb-4bit", trust_remote_code=True,
)
# GPU0: ~9.7GB, GPU1: ~10.3GB → ~6GB free each for training!
# Attach LoRA
model = get_peft_model(model, LoraConfig(
r=8, lora_alpha=16, lora_dropout=0,
target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
"gate_proj", "up_proj", "down_proj"],
task_type="CAUSAL_LM",
))
model.gradient_checkpointing_enable()
model.enable_input_require_grads()
# IMPORTANT: pass processing_class=tokenizer to avoid trust_remote_code error in SFTTrainer
trainer = SFTTrainer(
model=model,
train_dataset=dataset,
processing_class=tokenizer, # REQUIRED for custom models!
args=SFTConfig(
output_dir="./output",
per_device_train_batch_size=1,
gradient_accumulation_steps=4,
num_train_epochs=3,
learning_rate=2e-4,
bf16=True,
save_strategy="no",
report_to="none",
dataset_text_field="text",
max_length=256,
),
)
trainer.train()
Note: SFTTrainer internally calls
AutoProcessor.from_pretrained()withouttrust_remote_code=True. Passingprocessing_class=tokenizerbypasses this. See TRL Issue #4129.
| Stage | GPU0 | GPU1 | System RAM |
|---|---|---|---|
| Model loaded (manual split) | 9.7 GB | 10.3 GB | 15 GB |
| + PEFT + Training | ~12 GB | ~13 GB | 18 GB |
| Stage | GPU VRAM | System RAM |
|---|---|---|
| Model loaded | 14.1 GB | 15 GB |
| + Meta materialized (P7) | 15.8 GB | 15 GB |
| + PEFT + Training | 15.9 GB (OOM risk) | 18 GB |
Nemotron uses Mamba-2 layers requiring mamba_ssm and causal_conv1d packages which don't compile on Windows. The model falls back to naive PyTorch implementation automatically:
The fast path is not available because one of (selective_state_update, ...) is None.
Falling back to the naive implementation.
This is ~2-3x slower but functionally correct.