NVIDIA-Nemotron-3.5-Lightning-30B-A3B-oQ4e-mtp

NVIDIA Nemotron 3.5 Lightning 30B-A3B — oQ4e MTP

An oMLX/oQe mixed-precision MLX conversion of NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16, with the embedded Lightning MTP head preserved.

This is an unofficial community conversion. It is intended for Apple Silicon systems using oMLX / MLX-compatible loaders.

Quantization

• oQ4e enhanced quantization with activation/imatrix calibration
• Affine quantization, group size 64
• 64 calibration samples × 256 tokens
• All 5,888 routed experts received calibration coverage
• Routed experts use the oQ4e mixed-precision plan
• Shared experts are protected at 8-bit
• Mamba projections use sensitivity-selected 6/8-bit precision
• lm_head uses 8-bit precision

The following important tensors remain full precision (BF16/F32):

• token embeddings
• all backbone attention projections
• all normalization weights
• router weights and routing correction biases
• Mamba/SSM state parameters and convolution weights
• the complete embedded MTP head

The included oq_imatrix_report.json records the calibration coverage and which tensors received imatrix weighting.

MTP

The model keeps num_nextn_predict_layers: 1 and the mtp.* weights. This means the embedded Lightning MTP speculative-decoding path remains available when MTP is enabled by the serving runtime. The MTP head is deliberately kept at BF16/F32 to protect draft acceptance.

Usage with oMLX

Point oMLX at the directory containing this model and enable the model's MTP setting in the oMLX model configuration. The artifact uses standard MLX safetensors and does not require a custom quantized-weight loader.

License

The base model is distributed under the OpenMDW License Agreement, version 1.1. Please review the license and NVIDIA's original model card before using or redistributing this derivative model.