Muse-Glimmer-30B-GPTQ-Int4

Muse-Glimmer-30B — GPTQ 4-bit (W4A16)

4-bit GPTQ quantization of meta-models/Muse-Glimmer-30B, produced with GPTQModel.

Quantization details

As of the quantization date (2026-08-15), GPTQModel had no built-in support for the muse_glimmer architecture — a custom model definition was hand-written (MuseGlimmerQModel) covering the Gemma2-style sandwich norm, the 5-projection attention block (an extra gate_proj sigmoid output gate fed by pre-attention hidden states, alongside q/k/v), and the alternating sliding/full attention pattern. Only language_model.* weights are quantized — the vision tower/adapter/projection are left untouched (this checkpoint is intended for text-only serving; use the upstream model directly for multimodal input).

Native muse_glimmer support later landed upstream on GPTQModel's main branch (2026-08-22, PR #3020). If you're re-quantizing this model yourself, check whether your installed gptqmodel version already covers it before writing a custom definition — look for a "muse_glimmer" entry in MODEL_MAP inside gptqmodel/models/auto.py.

Serving with vLLM

As of 2026-08, vLLM's numbered PyPI releases have no muse_glimmer support — it only exists on vllm-project/vllm GitHub main. Before building from source, check whether it's since reached a release: grep -i muse_glimmer $(python -c "import vllm, os; print(os.path.dirname(vllm.__file__))")/model_executor/models/registry.py. If that comes up empty, build from source (or grab a matching nightly wheel):

git clone https://github.com/vllm-project/vllm && cd vllm
VLLM_USE_PRECOMPILED=1 pip install -e .

vllm serve <this-repo> \
  --served-model-name muse-glimmer-30b-gptq \
  --reasoning-parser muse_glimmer \
  --tool-call-parser muse_glimmer \
  --enable-auto-tool-choice \
  --tensor-parallel-size 4 \
  --gpu-memory-utilization 0.95

Notes:

• Native --max-model-len (131072) can OOM during CUDA-graph capture even with headroom to spare, because vLLM's automatic KV-cache profiling can overestimate available memory relative to actual capture-time usage. If you hit this, use the exact --kv-cache-memory-bytes=<N> value vLLM's own OOM message suggests, rather than guessing at --gpu-memory-utilization/--max-model-len cuts.
• --reasoning-parser muse_glimmer --tool-call-parser muse_glimmer --enable-auto-tool-choice are required for clean output — without them, chain-of-thought text leaks raw into the content field.

Speculative decoding (DFlash)

The official drafter, meta-models/Muse-Glimmer-30B-assistant, works well with this checkpoint — see the benchmark table below for acceptance rates and throughput across speculation depths:

vllm serve <this-repo> \
  --served-model-name muse-glimmer-30b-gptq \
  --reasoning-parser muse_glimmer \
  --tool-call-parser muse_glimmer \
  --enable-auto-tool-choice \
  --tensor-parallel-size 4 \
  --gpu-memory-utilization 0.95 \
  --kv-cache-memory-bytes 6000000000 \
  --speculative-config '{"method": "dflash", "model": "meta-models/Muse-Glimmer-30B-assistant", "num_speculative_tokens": 3}'

(num_speculative_tokens=3 is the recommended default per the sweep below — swap in 15, 7, or 1 if your traffic pattern favors a different point on the c1/c8 trade-off.)

The draft model is replicated per TP rank (not sharded), so it eats into KV-cache headroom — cut --kv-cache-memory-bytes accordingly versus the no-drafter config.

Benchmark (vllm bench serve, TP=4, 4x RTX 3090, 100 prompts, seed 0)

"Mean acceptance rate" is accepted-tokens / drafted-tokens over the whole run; "mean accepted length" is the average number of draft tokens accepted per verification round (out of num_speculative_tokens proposed). Per-round acceptance rate rises sharply as depth shrinks (fewer low-probability late positions to reject), but so does the drafting overhead paid per accepted token at deep settings — the two effects trade off non-monotonically:

• num_speculative_tokens=3 is the best overall default. It dominates =7 at both concurrencies (equal-or-better c1, notably better c8) and sits close to =1's c8 peak while roughly matching =7's c1 peak.
• c1 (interactive) peaks around depth 3-7, not at the extremes — =15 over-drafts (spends compute on positions that mostly get rejected) and =1 under-drafts (too little accepted length per round to amortize the draft forward pass).
• c8 (batched) strictly favors shallower speculation — throughput rises monotonically as depth shrinks from 15 down to 1, though even =1 doesn't recover no-drafter's raw batched throughput (the draft model's per-rank-replicated forward pass still competes for compute at large batch sizes).
• If your traffic is consistently high-concurrency/batch, skip DFlash entirely and use no-drafter. If it's mixed or interactive-heavy, num_speculative_tokens=3 is the recommended default.

Using as a Claude Code / Codex CLI backend

Both point their OpenAI/Anthropic-compatible client at your vLLM server.

Claude Code (Anthropic Messages API, vLLM implements this natively):

export ANTHROPIC_BASE_URL="http://<your-vllm-host>:8000"
export ANTHROPIC_API_KEY="dummy"
export ANTHROPIC_AUTH_TOKEN="dummy"
export ANTHROPIC_DEFAULT_OPUS_MODEL="muse-glimmer-30b-gptq"
export ANTHROPIC_DEFAULT_SONNET_MODEL="muse-glimmer-30b-gptq"
export ANTHROPIC_DEFAULT_HAIKU_MODEL="muse-glimmer-30b-gptq"
claude

Codex CLI (OpenAI-compatible):

codex \
  -c preferred_auth_method="apikey" \
  -c model="muse-glimmer-30b-gptq" \
  -c model_provider="local-vllm" \
  -c model_providers.local-vllm.name="local-vllm" \
  -c model_providers.local-vllm.base_url="http://<your-vllm-host>:8000/v1" \
  -c model_providers.local-vllm.wire_api="responses" \
  -c model_providers.local-vllm.env_key="DUMMY_KEY"