Downloads · 30 days
73
9% of all-time downloads
leonsarmiento/Nemotron-3.5-Lightning-30B-A3B-6bit-XL-mlx
Nemotron-3.5-Lightning-30B-A3B-6bit-XL-mlx is a text generation model from leonsarmiento. Use it when you need the model to write or continue text. It is set up for mlx. The card lists the license as other.
ℹ️ MTP note: The source model ships a NextN/MTP prediction layer (numnextnpredictlayers: 1), but this MLX build does not include MTP weights — the layer is dropped during conversion (52 backbone layers, no MTP tensors…
Downloads · 30 days
73
9% of all-time downloads
All-time downloads
828
Public
Parameters
31.6B
27 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors27 GB · 100%
How the weights are stored.
U3230.8B · 97%
From the Hugging Face model README
ℹ️ MTP note: The source model ships a NextN/MTP prediction layer (
num_nextn_predict_layers: 1), but this MLX build does not include MTP weights — the layer is dropped during conversion (52 backbone layers, no MTP tensors). No MTP speculative decoding on any engine; the residual MTP fields in config.json are inert.
MLX 6/8-bit BaseQuant_XL quantization of nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 for Apple Silicon inference.
BaseQuant_XL is a data-agnostic static mixed-precision recipe — no calibration data, no iMatrix, no activation-weighted sampling. It allocates precision by layer importance using a simple heuristic: routing-critical layers stay at full precision, every-token layers get 8-bit (near-lossless), and sparse MoE experts get 6-bit (natural redundancy from 128 experts, only 6 active per token). This contrasts with data-dependent methods (iMatrix, AWQ, GPTQ, oQ, oQ4e) that can skew representation toward well-represented domains.
| Precision | Layers | Rationale |
|---|---|---|
| bf16 | lm_head, shared expert | Routing/output-critical — errors here are qualitatively different |
| 8-bit | embeddings, attention (q/k/v/o_proj), Mamba2 (in_proj, out_proj, conv1d) | Every-token layers — 8-bit is near-lossless |
| 6-bit | routed experts (128 experts × 23 MoE layers + MTP MoE) | Bulk parameters — natural redundancy tolerates lower precision |
The MoE router gate (gate.weight, e_score_correction_bias) is kept in full precision as bare arrays (not quantizable layers).
Nemotron-H — hybrid Mamba2 + Transformer + MoE architecture:
shared_expert_overlap: truemlp_hidden_act: relu2, routed_scaling_factor: 2.5tie_word_embeddings: false<think> reasoning and tool-call supporttemperature: 1.0
top_p: 0.95
top_k: 40
min_p: 0.01
repeat_penalty: 1.05
reasoning_parser: nemotron_v3
tool_call_parser: qwen3_coder
Thinking mode is controlled via the chat template kwarg enable_thinking (default: true).
from mlx_lm import load, generate
model, tokenizer = load("leonsarmiento/Nemotron-3.5-Lightning-30B-A3B-6bit-XL-mlx")
prompt = tokenizer.apply_chat_template(
[{"role": "user", "content": "Explain quantum entanglement."}],
add_generation_prompt=True,
)
response = generate(model, tokenizer, prompt=prompt, max_tokens=512)
print(response)
Also compatible with LM Studio and oMLX — point it at the model directory.