fabletron-nemotron-3-super-120b-MLX-4bit

Fabletron — Nemotron-3-Super-120B-A12B · Fable-5 (MLX 4-bit)

QLoRA fine-tune of NVIDIA Nemotron-3-Super-120B-A12B (120B-total / 12B-active hybrid Mamba-2 + Latent-MoE, nemotron_h) on the pi_agent split of Glint-Research/Fable-5-traces, targeting reasoning, agentic planning, and tool-use. This repo holds the MLX 4-bit export (≈4.5 bits-per-weight, ≈64 GB) for Apple Silicon (Metal) via mlx-lm.

Support caveat: this is a structurally valid MLX 4-bit export (converted and lazy-load verified). nemotron_h is a hybrid Mamba-2 + MoE architecture; confirm your mlx-lm version actually implements it before relying on this for inference. For the most portable runnable artifacts, prefer the GGUF Q4_K_M or NVFP4 siblings.

How it was built: converted with mlx-lm on Linux, CPU-only (pip install mlx mlx-cpu mlx-lm — the mlx-cpu backend package lets conversion run without a Mac/Metal GPU). It runs on Apple Silicon.

LM Studio download note: there is a known LM Studio bug where its HF-proxy downloader hangs at "Finalizing download…" on empty (0-byte) files and never marks the model ready — tracked upstream at lmstudio-ai/lmstudio-bug-tracker#1477. The empty __init__.py that ships with nemotron_h repos triggers it. Until LM Studio fixes the bug, this repo works around it by shipping a non-empty __init__.py so fresh downloads finalize normally. If you have a download already stuck on an older revision, quit LM Studio, touch the missing __init__.py in the model folder, and relaunch.

Run (Apple Silicon)

pip install mlx-lm
mlx_lm.generate --model greghavens/fabletron-nemotron-3-super-120b-MLX-4bit \
  --prompt "Explain the Mamba-2 state-space update." --max-tokens 512
# or an OpenAI-compatible server:
mlx_lm.server --model greghavens/fabletron-nemotron-3-super-120b-MLX-4bit

Needs a Mac with enough unified memory (≈64 GB+ for the 4-bit weights plus KV cache). Prompt format is ChatML (<|im_start|> / <|im_end|>); the model emits <think>…</think> reasoning before its answer.

Quantization

• MLX 4-bit group-wise affine quantization, ≈4.5 bits-per-weight (see the export's config.json quantization block). ≈64 GB on disk across 14 shards.
• Sibling formats for maximum fidelity / broader hardware: GGUF Q4_K_M (…-GGUF), NVFP4 (…-NVFP4), full-precision merged BF16 (…-BF16).

Training

• Method: QLoRA (4-bit NF4 base) via Unsloth. LoRA r=8, α=16, dropout=0, bias=none. Targets (by regex) attention q/k/v/o_proj, Mamba-2 in_proj/out_proj, and the up/down_proj of all 512 routed experts + the shared expert — this MoE is non-gated, so up/down_proj is the full expert FFN (there is no gate_proj). The MoE router (mixer.gate, an nn.Parameter) and the MoE-latent fc1/fc2_latent_proj stay frozen.
• Data: Glint-Research/Fable-5-traces, config pi_agent, ChatML, response-only loss.
• Schedule: 1 epoch (248 steps), grad-accum 16, lr 2e-5 cosine + warmup.
• Final training loss: ≈0.74 true per-token CE (final-phase mean; Unsloth logs loss × grad-accum, so the raw logged value is ÷16 here).
• Evaluation (held-out Fable-5 pi_agent, 81 rows, base → fine-tuned): cross-entropy 1.12 → 0.78, perplexity 3.07 → 2.18.
• Date: 2026-06-29.
• LoRA adapter (pre-merge): greghavens/fabletron-nemotron-3-super-120b.

Provenance & license

This is a derivative of two upstream works, and downstream use must comply with both:

• Base model: NVIDIA Nemotron-3-Super-120B-A12B — see NVIDIA's model license.
• Dataset: Glint-Research/Fable-5-traces — AGPL-3.0, and distilled from Anthropic Claude outputs (subject to Anthropic's usage terms).

Released as a research artifact. Review the upstream licenses before redistribution or production use. license: other reflects the combination of the above, not a single license.

Intended use

Research on hybrid Mamba-2/MoE reasoning models, agentic/tool-use experimentation. Not safety-aligned beyond the base model; evaluate before any production deployment.

<!-- FABLETRON_BENCHMARKS_START -->

Benchmarks

Fabletron is a Fable-5 fine-tune of Nemotron-3-Super-120B-A12B. This repository hosts the MLX 4-bit (for Apple Silicon) build. The results below were produced through a deployed NeMo Evaluator at full context length (262 144) with a 32 768-token generation budget, comparing base Nemotron-3-Super against the Fabletron fine-tune in its two served formats — GGUF (llama.cpp) and NVFP4 (vLLM / Blackwell).

Accuracy

Scores are % (higher is better). The standout is function-calling (BFCL): 31.7 → 53.6 — base Nemotron largely fails multi-turn and parallel tool use, which the fine-tune resolves (single-tool calls 0.8 → 85.7, multi-turn 0 → 45). General reasoning and coding hold or improve slightly, and NVFP4 tracks the GGUF fine-tune within noise — the only give-back to quantization is IFEval strict (≈2 points).

Throughput

Mean per-request decode rate across the 6 benchmark families (tok/s, client-side, parallelism 4): base 45.7 · Fabletron GGUF 46.3 · Fabletron NVFP4 63.8. The NVFP4 build serves at ≈1.4× the GGUF (llama.cpp) rate on an RTX PRO 6000 (Blackwell).

_{BFCL = Berkeley Function-Calling Leaderboard v3 (multi_turn,ast, native tool-calling, via nvidia_bfcl 26.3). Base and GGUF share the llama.cpp engine; NVFP4 runs on vLLM. Output truncation at this budget is ≈0 across families (MMLU-Pro shows ≈3.5% length-capped generations, intrinsic to the benchmark and near-identical for all three models, so it does not bias the comparison).}

<!-- FABLETRON_BENCHMARKS_END -->