Muse-Glimmer-30B-NVFP4

Muse-Glimmer-30B — NVFP4 (compressed-tensors)

A 4-bit NVFP4 quantization of meta-models/Muse-Glimmer-30B, in compressed-tensors format, for local serving on NVIDIA GB10 / DGX Spark (Blackwell sm_121, native FP4). Language model shrinks 56 GB → 22 GB and runs in vLLM at ~13 tok/s single-stream.

Recipe

Weight-only NVFP4 (E2M1, 16-element blocks, per-tensor + per-block f8e4m3 scales) on the language-model linears only. The perception encoder is retained in bf16, so image input still works.

• 416 linears quantized, mean weight round-trip error ~9.5% (expected for E2M1).
• Format: compressed-tensors · nvfp4-pack-quantized · group_size: 16 · W4A16 (weight-only).

Serving (vLLM on GB10)

muse_glimmer support lands via vllm-project/vllm#51655 (native model + reasoning/tool parsers + DFlash speculative decoding). Until it merges upstream, build vLLM from that branch:

git clone -b tiezhen/new-model-support https://github.com/xianbaoqian/vllm && cd vllm
VLLM_USE_PRECOMPILED=1 pip install -e .          # precompiled kernels; no full CUDA build

export PATH="$VENV/bin:/usr/local/cuda/bin:$PATH"   # venv bin on PATH so vLLM's JIT finds `ninja`
vllm serve <this-repo> --served-model-name muse --trust-remote-code \
  --max-model-len 8192 --gpu-memory-utilization 0.9 \
  --reasoning-parser muse_glimmer --tool-call-parser muse_glimmer

The reasoning parser is required — muse uses channel-scoped output framing (not <think> tags) and forces skip_special_tokens=False; without it the channels collapse and output is empty.

Performance

1× DGX Spark (GB10), vLLM, greedy, batch size 1:

Decode is memory-bound — the speedup tracks the 56→22 GB weight reduction, and CUDA graphs add only ~13% (launch overhead). Pair with the DFlash drafter for ~3× on decode.

Validation

Smoke-verified with greedy generation on the vLLM branch above — coherent output with correct stop tokens. Full accuracy suite not yet run.

Provenance

Quantized with a pure-torch (CPU) compressed-tensors NVFP4 converter directly from the bf16 safetensors — weight-only, no calibration data. Base model © Meta Superintelligence Lab (Apache-2.0).