Downloads · 30 days
104
34% of all-time downloads
roman220220/nemotron-30b-a3b-gptq3bit-g64
nemotron-30b-a3b-gptq3bit-g64 is a machine learning model from roman220220. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for mlx. The card lists the license as other.
<p align="center" <img src="llmtray-banner.png" alt="LLMTray" width="100%" </p
Downloads · 30 days
104
34% of all-time downloads
All-time downloads
305
Public
Parameters
31.6B
13.8 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors13.8 GB · 100%
How the weights are stored.
U3231.6B · 100%
From the Hugging Face model README
▶ Run it locally in LLMTray
A free, native macOS app for local AI on Apple Silicon — chat, images, music, agents and an OpenAI-compatible API. Point it at this model; nothing leaves your Mac.
GPTQ-calibrated (Hessian-based error compensation, Frantar et al. 2022)
uniform 3-bit quantization of NVIDIA's Nemotron-3.5-Lightning-30B-A3B
(NemotronH hybrid Mamba2 + Attention + MoE, 52 layers), packed with stock
mlx_lm.convert (no custom kernel, no custom model-loading code — runs with
plain mlx_lm/LM Studio like any other MLX model).
mx.quantize(mode="affine")), then packed with stock mlx_lm.convert -q --q-bits 3 --q-group-size 64 — so the calibration is preserved rather
than silently discarded by a second, uncoordinated quantization pass.
Uniform bit-width means this release is NOT affected by the
switch_mlp.fc1/fc2 naming bug found in this project's mixed-precision
releases (a uniform -q --q-bits N conversion applies the same bit-width
everywhere regardless of tensor name).| model | PPL |
|---|---|
| bf16 (full precision, reference) | 5.11 |
| JANG_2L-CRACK (3rd party, mixed-precision) | 5.43 |
| this model | 6.24 |
| naive RTN uniform 3-bit (no calibration) | 6.54 |
GPTQ's Hessian correction measurably beats naive round-to-nearest at the same bit-width and size (6.24 vs 6.54).
Local-first AI tools for macOS by IPSupport — nothing leaves your Mac.
Runs in LLMTray as a local chat LLM.
Distributed under NVIDIA's OpenMDW License Agreement v1.1 (same license
as the base model) — see LICENSE in this repo.
pip install mlx-lm
python -m mlx_lm.generate --model roman220220/nemotron-30b-a3b-gptq3bit-g64 --prompt "..."
Produced with this project's GPTQ pipeline
(poc/gptq_stock_convert.py, poc/run_pipeline.sh) — see
docs/session_findings_2026-09-11.md section 7p for the full methodology
and formula-verification writeup.