Downloads · 30 days
160
2% of all-time downloads
manjunathshiva/Nemotron-3-Super-120B-A12B-tq3a-tq2e-g32
Nemotron-3-Super-120B-A12B-tq3a-tq2e-g32 is a text generation model from manjunathshiva. Use it when you need the model to write or continue text. It is set up for mlx. The card lists the license as other.
TurboQuant hybrid quantization of nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16 — 3-bit attention + 2-bit experts at groupsize=32 — using TurboQuant-MLX.
Downloads · 30 days
160
2% of all-time downloads
All-time downloads
7K
Public
Parameters
11.9B
38.8 GB on disk
Likes
1
Public
Click a slice to open those files.
.safetensors38.8 GB · 100%
How the weights are stored.
U327.5B · 63%
From the Hugging Face model README
TurboQuant hybrid quantization of nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16 — 3-bit attention + 2-bit experts at group_size=32 — using TurboQuant-MLX.
This is the 48 GB-RAM variant of the Nemotron-3 Super 120B quantization. The standard 3-bit (~50 GB) needs ~55 GB peak and only fits a 64 GB Mac after raising iogpu.wired_limit_mb. This hybrid keeps attention at 3-bit (where precision matters) and pushes experts to 2-bit (where the bulk of the weights live), dropping peak memory to ~40.8 GB and lifting decode speed to ~27.2 tok/s so the model fits comfortably on a 48 GB or 64 GB Apple Silicon MacBook with headroom for other apps.
Documentation: manjunathshiva.github.io/turboquant-mlx: the Nemotron-3 page, getting started, choosing a model for your Mac's memory, guides for serving an OpenAI-compatible API, local coding agents and memory and macOS settings.
MEMEMEM*EMEMEMEM*… (M = Mamba, E = MoE, * = Attention)moe_latent_size = 1024iogpu.wired_limit_mb=49152 (48 GB) on a 64 GB MacDon't guess, and don't download 38.8 GB to find out — ask first (needs turboquant-mlx-full >= 0.15.1):
turboquant-plan --model manjunathshiva/Nemotron-3-Super-120B-A12B-tq3a-tq2e-g32
It reads only this repo's safetensors headers over the network (a couple of hundred KB, a few seconds — the weights are never fetched), then projects the peak against your machine and prints the flags to use:
Model
weights (exact) 38.8 GB
experts 35.2 GB (streamable)
resident 3.5 GB
KV cache 8.0 KB/token (Mamba hybrid: 8/88 attention layers)
Verdict: <resident | needs a wired-limit raise | streaming>
Recommended: <the exact flags for your Mac>
Planning for a machine you're not sitting at — deciding whether a 16 GB mini or a 64 GB Mac can take this — is what --wired-gb / --ram-gb are for:
turboquant-plan --model manjunathshiva/Nemotron-3-Super-120B-A12B-tq3a-tq2e-g32 \
--wired-gb 10.5 --ram-gb 16 --context 8192
turboquant-doctor runs the same projection plus a readiness check; both take --json. The projection is calibrated against real measurements on a 16 GB Mac mini rather than estimated from theory — on the 9.4 GB ternary 35B it predicts a 10.44 GB peak where that machine measures 10.42.
pip install "turboquant-mlx-full>=0.2.0" "mlx-lm>=0.31.3"
⚠️ Use
turboquant-mlx-full0.2.0 or newer — earlier versions don't have the per-layer--attn-bits/--mlp-bitsplumbing required to load this hybrid model, the long-context kernel fix for prompts that span more than a few thousand tokens, or the v0.2 KV-cache CLI flags (--kv-k-bits/--kv-v-bits/--kv-min-tokens) shown below.
hf download manjunathshiva/Nemotron-3-Super-120B-A12B-tq3a-tq2e-g32 \
--local-dir ~/models/nemotron-3-super-120b-tq3a-tq2e-g32
For prose, code, format, and long-context tasks, use the empirically-validated decode config (see Phase-1 known limitation below for math/numeric prompts):
turboquant-generate \
--model ~/models/nemotron-3-super-120b-tq3a-tq2e-g32 \
--prompt "Why is the sky blue? Explain in detail." \
--max-tokens 4096 --min-tokens 50 \
--temp 0.7 --rep-penalty 1.04 --rep-ctx 256
The --min-tokens 50 flag is required for Nemotron-3 Super — the model emits a <think> reasoning trace before its final answer, and the chat template primes EOS as the top-1 logit at the start of the assistant turn.
The small repetition penalty (--rep-penalty 1.04 --rep-ctx 256) prevents long-form generation from collapsing into degenerate tail loops past ~1500 tokens. Without it, you may see em-dash runs or repeated phrases at the tail of long essays.
For long-context generation, layer the v0.2 KV-cache compression on top. Mixed K8/V3 is required when the weights are TurboQuant-quantized; symmetric K3 would compound the noise and break long-form output. The 128-token fp16 sink protects attention sinks at the prompt start.
turboquant-generate \
--model ~/models/nemotron-3-super-120b-tq3a-tq2e-g32 \
--prompt "Why is the sky blue? Explain in detail." \
--max-tokens 4096 --min-tokens 50 \
--temp 0.7 --rep-penalty 1.04 --rep-ctx 256 \
--kv-k-bits 8 --kv-v-bits 3 --kv-min-tokens 128
Step-by-step arithmetic on this hybrid is degraded under any non-zero --rep-penalty. The 2-bit experts cause small slips in numeric reasoning that the repetition penalty doesn't compensate for. For numeric/math prompts in this Phase-1 release, omit --rep-penalty:
# Math/numeric prompt — omit rep-penalty
turboquant-generate \
--model ~/models/nemotron-3-super-120b-tq3a-tq2e-g32 \
--prompt "A train leaves Boston at 9:00 AM going 60 mph..." \
--max-tokens 2048 --min-tokens 50 \
--temp 0.7
The trade-off: without the penalty you may see long-gen tail loops on prompts with very long outputs, but the arithmetic will land correctly more often. For serious numeric work, prefer the standard tq3 model: manjunathshiva/Nemotron-3-Super-120B-A12B-tq3.
A permanent fix is planned for Phase 2 of TurboQuant-MLX — likely first/last-layer bit protection, a calibration-data codebook, or a fused QJL Metal kernel.
The fused MoE decode kernel transparently chunks expert routings on long prompts (K_CHUNK=4096), so this hybrid handles long-context retrieval over 4000+ tokens of context without the kernel argument-validation crash that affected earlier builds.
A 4000+ token "needle in a haystack" prompt (recall a password buried in 2000 words of filler on each side) recovers the password reliably.
Measured on a 64 GB MacBook M-series with macOS, MLX, and turboquant-mlx-full 0.1.6.
| Configuration | Size | Peak RAM | Fits 48 GB? | Speed |
|---|---|---|---|---|
| BF16 (original) | ~240 GB | — | ❌ | n/a |
| TurboQuant 3-bit (standard) | ~50 GB | ~55 GB | ❌ (needs sysctl) | ~19 tok/s |
| TurboQuant hybrid (this repo) | ~36 GB | ~40.8 GB | ✅ | ~27.2 tok/s |
temp=0.7 rep_penalty=1.04 rep_ctx=256)| Test | Result |
|---|---|
| 1500-word essay (3500-tok budget) | ✅ clean — proper conclusion + references, no degenerate tail |
| Step-by-step math (train-meeting problem) | ⚠️ Phase-1 limitation — final number off |
Python code generation (merge_intervals + 3 unit tests) | ✅ clean |
| Long-context needle (4000-tok password recall) | ✅ password recovered |
| Numbered-list format (5 benefits, ≤15 words each) | ✅ clean — exits <think>, exactly 5 lines |
| Open-ended explanation (4096-tok budget) | ✅ clean — terminates at ~1.5K tokens with proper structure |
TurboQuant applies, in one shot with no calibration data:
config.json.Nemotron-3 Super is a hybrid model — the layer pattern alternates between:
Plus 1 MTP (multi-token-prediction) layer. There are no dense MLP layers — all FFN compute goes through the MoE.
turboquant-mlx-full 0.1.6 quantizes:
in_proj / out_proj linears (2-bit)fc1_latent_proj, fc2_latent_proj) — the shared expert pantry (2-bit)Embeddings, layer norms, and small bias-style tensors stay in BF16 / FP16.
Released under the NVIDIA Nemotron Open Model License (same as the base model). See https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-nemotron-open-model-license/
mlx-lm maintainers — for landing Nemotron-H + latent-MoE + MTP support in 0.31.3@article{zandieh2025turboquant,
title = {TurboQuant: A Unified Framework for Extremely Low-Bit Weight and KV Cache Quantization},
author = {Zandieh, Amir and Han, Minsik and Dalca, Andre and Shin, Jungwoo and Wang, Brian and Zhang, Yichao and Bordegoni, Matteo and Tian, Yuan and others},
year = {2025}
}
manjunathshiva/Nemotron-3-Super-120B-A12B-tq3