Downloads · 30 days
44
3% of all-time downloads
greghavens/fabletron-nemotron-3-super-120b-NVFP4
fabletron-nemotron-3-super-120b-NVFP4 is a text generation model from greghavens. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as other.
QLoRA fine-tune of NVIDIA Nemotron-3-Super-120B-A12B (120B-total / 12B-active hybrid Mamba-2 + Latent-MoE, nemotronh) on the piagent split of Glint-Research/Fable-5-traces, targeting reasoning, agentic planning, and t…
Downloads · 30 days
44
3% of all-time downloads
All-time downloads
1.4K
Public
Parameters
63.1B
71.5 GB on disk
Likes
2
Public
Click a slice to open those files.
.safetensors71.5 GB · 100%
How the weights are stored.
U857.6B · 82%
From the Hugging Face model README
QLoRA fine-tune of NVIDIA Nemotron-3-Super-120B-A12B (120B-total / 12B-active hybrid
Mamba-2 + Latent-MoE, nemotron_h) on the pi_agent split of
Glint-Research/Fable-5-traces,
targeting reasoning, agentic planning, and tool-use. This card documents the NVFP4 export
(NVIDIA ModelOpt format) for vLLM / TensorRT-LLM on NVIDIA Blackwell (FP4 tensor cores).
Status: the ≈67 GB NVFP4 weights (7 safetensors shards) are uploaded here and the export loads and serves on vLLM on an RTX PRO 6000 Blackwell Workstation (sm_120) at the full native 262144-token context. Measured server-side throughput at this precision: ≈63.7 tok/s per-request decode, ≈430 tok/s aggregate decode (concurrent sequences), ≈890 tok/s prefill. See the exact, verified serving recipe below. Siblings: the most-universal GGUF Q4_K_M and the pre-merge LoRA adapter.
Why ModelOpt and not compressed-tensors? vLLM only accepts unquantized, ModelOpt FP8, or ModelOpt NVFP4 checkpoints for
nemotron_h's non-gated MoE (is_act_and_mul=False); compressed-tensors NVFP4 is rejected for this architecture (vLLM #31782). This export is therefore produced with NVIDIA ModelOpt and served with-q modelopt_fp4.
LM Studio download note: there is a known LM Studio bug where its HF-proxy downloader hangs at "Finalizing download…" on empty (0-byte) files and never marks the model ready — tracked upstream at lmstudio-ai/lmstudio-bug-tracker#1477. This is a vLLM / TensorRT-LLM export, not an LM Studio format (use the MLX-4bit or GGUF siblings for LM Studio) — but the empty
__init__.pythat ships withnemotron_hrepos triggers that bug for anyone pulling this repo through LM Studio's downloader. As a precaution it now ships a non-empty__init__.pyso such downloads finalize normally.
This is the exact recipe used in production on sm_120 (RTX PRO 6000 Blackwell) — CUDA graphs
on (no --enforce-eager), FlashInfer autotune on, native FP4 MoE:
# The FP8 Mamba in/out_proj GEMM: FlashInfer's cuDNN per-tensor-q GRAPH plan fails to
# build on sm_120 during CUDA-graph capture. Disable ONLY that one FP8 kernel so it falls
# back to CutlassFP8ScaledMMLinearKernel (graph-capturable). The FP4 experts/MoE stay
# 100% native FlashInfer CUTLASS -> no --enforce-eager needed.
export VLLM_DISABLED_KERNELS="FlashInferFP8ScaledMMLinearKernel"
vllm serve greghavens/fabletron-nemotron-3-super-120b-NVFP4 \
-q modelopt_fp4 \
--trust-remote-code \
--mamba-ssm-cache-dtype float32 \
--kv-cache-dtype auto \
--max-model-len 262144 \
--max-num-seqs 16 \
--gpu-memory-utilization 0.90 \
--enable-chunked-prefill \
--reasoning-parser nemotron_v3 \
--enable-auto-tool-choice --tool-call-parser qwen3_coder \
--served-model-name nemotron-3-super-fable5-nvfp4 nemotron
- Pass
-q modelopt_fp4explicitly: the checkpoint'squant_methodmay be empty and would otherwise be missed (ModelOpt #261).- Context: only 8 of 88 layers are attention, so KV is ≈8 KiB/token (≈2 GiB/seq at full 262144). Serving at a short context truncates BFCL multi-turn and MMLU-Pro 5-shot CoT — the native 262144 guarantees nothing truncates.
- Reasoning/tools: the model emits
<think>…</think>;--reasoning-parser nemotron_v3routes that intoreasoning_contentso the gradedcontentis the post-think answer.--tool-call-parser qwen3_coderhandles OpenAI-styletool_calls. Uses the bundledchat_template.jinja.- The base model's MTP (next-n-predict) head is not carried into this export (ModelOpt #750).
MIXED_PRECISION — NVFP4 (4-bit float, per-block 16-wide, FP8
scales, dynamic FP4 activations) for most matmuls, FP8 for the Mamba projections.
Calibration algorithm: max. (Verified from the export's hf_quant_config.json.)q/k/v/o_proj (32 tensors); all 512 routed + shared expert
up/down_proj (41,040 tensors); MoE latent fc1/fc2_latent_proj (80 tensors).in_proj/out_proj (80 tensors) — kept at higher precision.mixer.gate),
Mamba-2 conv1d, lm_head, token embeddings, all RMSNorms, and the MTP head.data/train.jsonl, ChatML), sequence length 2048.Precision note: this export follows NVIDIA's conservative recipe — Mamba
in_proj/out_projare kept at FP8 (an earlier all-NVFP4 attempt was discarded asnvfp4.uniform-broken). For maximum fidelity prefer the GGUF Q4_K_M sibling repo.
q/k/v/o_proj, Mamba-2 in_proj/out_proj, and the
up/down_proj of all 512 routed experts + the shared expert — this MoE is non-gated,
so up/down_proj is the full expert FFN (there is no gate_proj). The MoE router
(mixer.gate, an nn.Parameter) and the MoE-latent fc1/fc2_latent_proj stay frozen.Glint-Research/Fable-5-traces, config pi_agent, ChatML, response-only loss.pi_agent, 81 rows, base → fine-tuned): cross-entropy
1.12 → 0.78, perplexity 3.07 → 2.18.greghavens/fabletron-nemotron-3-super-120b.This is a derivative of two upstream works, and downstream use must comply with both:
Glint-Research/Fable-5-traces — AGPL-3.0, and distilled from Anthropic
Claude outputs (subject to Anthropic's usage terms).Released as a research artifact. Review the upstream licenses before redistribution or
production use. license: other reflects the combination of the above, not a single license.
Research on hybrid Mamba-2/MoE reasoning models, agentic/tool-use experimentation. Not safety-aligned beyond the base model; evaluate before any production deployment.
<!-- FABLETRON_BENCHMARKS_START -->Fabletron is a Fable-5 fine-tune of Nemotron-3-Super-120B-A12B. This repository hosts the NVFP4 (for vLLM on NVIDIA Blackwell) build. The results below were produced through a deployed NeMo Evaluator at full context length (262 144) with a 32 768-token generation budget, comparing base Nemotron-3-Super against the Fabletron fine-tune in its two served formats — GGUF (llama.cpp) and NVFP4 (vLLM / Blackwell).

| Benchmark | Base Nemotron 3 Super | Fabletron (GGUF) | Fabletron (NVFP4) |
|---|---|---|---|
| IFEval | 89.6 | 90.4 | 87.4 |
| GSM8K | 96.6 | 96.5 | 96.5 |
| MMLU-Pro | 81.1 | 81.5 | 81.2 |
| HumanEval+ | 87.8 | 89.6 | 89.6 |
| MBPP+ | 95.5 | 96.6 | 96.8 |
| BFCL (v3 · multi_turn + ast) | 31.7 | 53.6 | 53.9 |
Scores are % (higher is better). The standout is function-calling (BFCL): 31.7 → 53.6 — base Nemotron largely fails multi-turn and parallel tool use, which the fine-tune resolves (single-tool calls 0.8 → 85.7, multi-turn 0 → 45). General reasoning and coding hold or improve slightly, and NVFP4 tracks the GGUF fine-tune within noise — the only give-back to quantization is IFEval strict (≈2 points).

Mean per-request decode rate across the 6 benchmark families (tok/s, client-side, parallelism 4): base 45.7 · Fabletron GGUF 46.3 · Fabletron NVFP4 63.8. The NVFP4 build serves at ≈1.4× the GGUF (llama.cpp) rate on an RTX PRO 6000 (Blackwell).
<sub>BFCL = Berkeley Function-Calling Leaderboard v3 (multi_turn,ast, native tool-calling, via nvidia_bfcl 26.3). Base and GGUF share the llama.cpp engine; NVFP4 runs on vLLM. Output truncation at this budget is ≈0 across families (MMLU-Pro shows ≈3.5% length-capped generations, intrinsic to the benchmark and near-identical for all three models, so it does not bias the comparison).</sub>