Muse-Glimmer-30B-MXFP4-Quark-RDNA4

Muse-Glimmer-30B — MXFP4 (Quark) for RDNA4, 1M-context profile, with bundled DFlash2 drafter

MXFP4 (OCP microscaling) quantization of meta-models/Muse-Glimmer-30B, built with AMD Quark 0.12 for RDNA4 (gfx12) vLLM serving. ~60 GB bf16 → 28 GB on disk, shipped as a two-profile serving package: a concurrency profile with the context window extended to 1,048,576 tokens (YaRN ×8, baked into the config), and a single-stream profile with the bundled z-lab DFlash2 drafter (dflash-draft/, Apache-2.0, unmodified).

Why this model got a quant: a dense 30B vision-language model with an Apache-2.0 license, 2 KV heads and a 13-global / 39-sliding-window attention layout that keeps the KV cache small enough for million-token windows on two 32 GB cards. It fits a class of hardware that mixture-of-experts flagships do not. No cross-model claims are made here; every number below is this artifact on this hardware.

What is quantized

• MXFP4 (fp4 e2m1 weights, group size 32, e8m0 shared scales) on the 52 language-model MLP blocks (gate_proj / up_proj / down_proj, 156 modules). Quark's config also declares dynamic MX fp4 input quantization for those modules (global_quant_config.input_tensors); activations elsewhere are bf16.
• Kept in bf16 (564 exclusions): all attention projections and attention gates, the vision tower (fc1 / fc2 / proj), lm_head, norms and embeddings.
• Image inputs were not benchmarked in this release; the vision tower is untouched but unverified here.
• Export: Quark real_quantized weights, pack_method: reorder; config declares quant_method: quark.

Two configs in one repo

The native config is included on purpose: the extension costs measurable instruction-following accuracy (see Quality), and some deployments do not need the window.

Serving (vLLM on RDNA4 — 2×Radeon AI PRO R9700, TP2)

Get the vLLM that serves this on RDNA4 (gfx12): the prebuilt image capicua25x/vllm-rocm-rdna4 (:0.28.0-rdna4 / :latest), source and port notes at Capicua25x/vllm-rocm-rdna4. On CUDA hardware, recent stock vLLM (≥ 0.28, muse_glimmer support) works with the same flags.

Concurrency profile (default, 1M window — ~16 concurrent users on 2×R9700):

vllm serve <this-repo> --tensor-parallel-size 2 --trust-remote-code \
  --gpu-memory-utilization 0.90 --max-model-len 1048576 \
  --attention-backend TRITON_ATTN --moe-backend triton_unfused \
  --enable-prefix-caching --max-num-seqs 32 --max-num-batched-tokens 8000 \
  --max-cudagraph-capture-size 128 --skip-mm-profiling \
  --enable-auto-tool-choice --tool-call-parser muse_glimmer --reasoning-parser muse_glimmer \
  --default-chat-template-kwargs '{"reasoning_strength": "low"}'

Single-stream profile (bundled DFlash2 drafter; measured at the native 131k window):

vllm serve <this-repo> --tensor-parallel-size 2 --trust-remote-code \
  --gpu-memory-utilization 0.90 --max-model-len 131072 \
  --attention-backend TRITON_ATTN --moe-backend triton_unfused \
  --enable-auto-tool-choice --tool-call-parser muse_glimmer --reasoning-parser muse_glimmer \
  --speculative-config '{"method":"dflash","model":"<this-repo>/dflash-draft","num_speculative_tokens":3,"attention_backend":"TRITON_ATTN"}'

Notes that cost us a window each:

• The reasoning channel is controlled by the chat-template kwarg reasoning_strength (low / medium / high / xhigh), per request via extra_body.chat_template_kwargs or server-wide with --default-chat-template-kwargs. reasoning_effort is inert on vLLM.
• With rope_scaling on this multimodal architecture, the override must land on the text config — a top-level override is silently ignored. The shipped config.json already has it in the right place.
• At the 1M window with --gpu-memory-utilization 0.92 the KV sizing starves the drafter; 0.90 (or 0.80 with fp8 KV) is stable on 32 GB cards.
• Do not serve with the default (auto) MoE/attention backends on ROCm gfx12; the flags above are the measured ones.

Measured throughput (2×R9700 TP2, apexia concurrency-bench v3, tokens/s)

* per-user ≥ 20 tok/s. Aggregate tok/s where marked. Acceptance rises with load on this drafter (1.55 → 2.08 accepted per draft from c1 to c8), so the DFlash profile holds up to c8 on 6k-prefill traffic before the shared drafter KV pool becomes the limit.

Quality

Same weights, two configs, same protocol (lm-eval with chat template, temperature 0.6 / top-p 0.95 / top-k 20, reasoning_strength: low, 5 seeds 1234–1238, IFEval subset n=80). Cells are 5-seed median [range].

The extension costs about 2–4 IFEval points, seed-consistent; that is the YaRN tax and the reason the native config ships alongside. GSM8K cells were run and are not reported: with this model's reasoning channel, lm-eval's answer extraction scored both configs at 0.2–0.4 at ~1 s per sample, an instrumentation mismatch, not a model result.

τ²-bench (agentic tool use), 1M config, agent at temperature 1.0 / top-p 0.95 with reasoning_strength: high, tau2-bench user simulator gpt-5.6-luna and NL-assertion judge gpt-5.6-sol, concurrency 6, seed 1234, one trial, no censored simulations:

Tool calls parse cleanly through --tool-call-parser muse_glimmer in every task above.

Long context: what is verified here

The 1M window is a serve-time YaRN extension of the native 131k model, not a retrained checkpoint. The IFEval and τ² numbers above were measured on the extended config, so the extension's cost on short-context quality is what the Quality table shows.

Retrieval past the native window, measured on this quant, on 2×R9700 with the serve command above (single needle in a Spanish filler haystack, recall question at the end, reasoning_strength: low, temperature 0; the prefill time per rung is the wall-clock of that single request):

Result: needle retrieval verified by us on this quant at every rung up to 832k tokens (3 depths each). The community's llama.cpp probes on the base model reach the same rung (832k). The prefill cost is the practical limit on this hardware: an 832k-token request occupies both cards for the time shown, so treat the window as a capability, not a concurrency profile, and keep the native config for workloads that do not need it.

Credits

• Meta for the base model (meta-models/Muse-Glimmer-30B, Apache-2.0). Meta's LICENSE and USAGE_POLICY.md are preserved in this repo unchanged.
• z-lab for the DFlash2 draft model, bundled unmodified in dflash-draft/ from z-lab/Muse-Glimmer-30B-DFlash2 (Apache-2.0; revision recorded in dflash-draft/REVISION); their README is preserved in the folder.
• AMD Quark quantization toolkit (0.12.post1).