Muse-Glimmer-30B-AutoRound-INT4

Muse Glimmer 30B — AutoRound INT4 (W4A16)

This is an INT4 W4A16 quantization of meta-models/Muse-Glimmer-30B produced with AutoRound (auto_round 0.14.2). It quantizes the 52-layer text decoder to 4-bit weights (group size 128, symmetric) while keeping the vision tower, embeddings, norm layers, adapter and the LM head in BF16 — so the full multimodal (image/text) capability of the original model is preserved.

The quantization is text-coherent at essentially BF16 quality: on a held-out general-prose set it matches (and even slightly improves on) the BF16 base in perplexity, with a mean KL divergence of only ~0.001 nats/token.

Scope note on metrics: PPL and KL are measured on a small held-out set (4 general-prose paragraphs in history / biology / literature / astronomy — not from the calibration domains, 431 tokens total) via a disk-offload streaming prefill of the text decoder. These are relative precision indicators against the same BF16 base, not broad general benchmarks.

Key results (text decoder)

Per-example KL(BF16 ‖ AutoRound): 0.00127, 0.00032, 0.00068, 0.00217.

The AutoRound INT4 output is three orders of magnitude tighter to the BF16 base than the earlier AWQ INT4 attempts on this model (AWQ-v2 measured ~0.46 nats/token), and it reproduces coherent reasoning, tool calling, and vision understanding when served.

Quantization details

• Tool: AutoRound (auto_round==0.14.2)
• Scheme: INT4 W4A16 — bits=4, group_size=128, sym=True, data_type="int"
• Calibration: nsamples=72 (28 valid after the seqlen≤1024 filter), iters=300, seqlen=1024, batch_size=1, lr=5e-3, gradient_accumulate_steps=2, seed=42
• Quantized blocks: model.language_model.layers (all 52 text-decoder layers, 8 linears each — all g128-divisible)
• Kept BF16: vision tower (model.vision_tower.*, ViT-G, 50 layers), model.vision_adapter, model.vision_projection, embeddings, norms, lm_head
• Exported format: auto_round:auto_gptq packing (transposed qweight/qzeros/scales)
• Size: ~21 GB across 8 safetensors shards; 416/2268 tensors quantized INT4
• block_name_to_quantize in quantization_config.json is set to model.layers — this is the serving-compatible naming required by SGLang/vLLM to actually dequantize the text decoder on load (the internal AutoRound naming model.language_model.layers would be treated as unquantized and OOM on small GPUs).

Full calibration dataset blend

Calibration used a text blend of NVIDIA Hugging Face instruction/reasoning/agentic datasets (the same blend pipeline used for the sibling Qwen3.8-27B AutoRound work), chat-formatted via the model's own chat template and capped to the calibration sequence length. The on-disk blend is composed of the following 25 categories:

Total: 210 samples across 25 categories. For this AutoRound run, 72 samples were drawn from the blend (with 28 surviving the seqlen≤1024 filter used for the Muse text-decoder calibration).

The vision tower is not quantized — it remains the original BF16 weights — so no vision calibration data is needed for the quantized text path.

Usage

The checkpoint is a standard AutoRound export and is intended to be served with SGLang or vLLM using the native auto-round quantization path.

vLLM / SGLang (validated on 2× RTX 3090)

# SGLang
python -m sglang.launch_server \
  --model-path woooflet/Muse-Glimmer-30B-AutoRound-INT4 \
  --served-model-name "Muse Glimmer 30B" \
  --quantization auto-round \
  --tp-size 2 \
  --kv-cache-dtype fp8_e4m3 \
  --enable-multimodal \
  --reasoning-parser muse --tool-call-parser muse

Transformers (AutoRound)

from transformers import AutoModelForImageTextToText, AutoProcessor

model = AutoModelForImageTextToText.from_pretrained(
    "woooflet/Muse-Glimmer-30B-AutoRound-INT4",
    torch_dtype="auto",
    device_map="auto",
    trust_remote_code=True,
)
processor = AutoProcessor.from_pretrained("woooflet/Muse-Glimmer-30B-AutoRound-INT4")

Note: for inference, load with the auto-round / AutoRound quantized path in SGLang or vLLM (both verified). If you load raw weights with Transformers without the AutoRound runtime, tensors are stored in the packed qweight/qzeros/scales layout and require AutoRound's dequantization.

Validated behavior

When served with SGLang (--quantization auto-round, TP2, fp8_e4m3 KV, muse parsers), the quantized model was verified to be coherent:

• Reasoning: correct reasoning↔content separation on math and multi-step problems
• Tool calling: multi-step tool-use chains (search → read → contacts → calc → email) handled correctly; tool-eval-bench 87/100
• Vision: reads charts, dashboards and photos coherently (chart values, UI elements, scene descriptions)
• Performance: prefill ~1846 tok/s, generation ~54 tok/s, TTFT ~1.03s (2× RTX 3090)

License & attribution

• Base model: meta-models/Muse-Glimmer-30B by Meta (License: Apache 2.0). This quantization inherits the base model's Apache 2.0 license.
• Quantization performed with AutoRound (Apache 2.0).
• Calibration data sampled from the NVIDIA Hugging Face datasets listed above; see each dataset for its individual terms.

This model is a community quantization of a Meta model and is not an official Meta release. Use and redistribute in accordance with the base model's license.

Author

Ford Morris