Muse-Glimmer-30B-W4A16-AutoRound

Muse Glimmer 30B - W4A16 AutoRound / GPTQ

This repository contains the W4A16 quantized versions of Meta's Muse Glimmer 30B, optimized for efficient local inference. The model was quantized using Intel AutoRound to 4-bit precision while explicitly keeping the Vision Encoder in its original precision (BF16) to preserve multimodal and visual agentic capabilities without degradation.

Two formats are provided across the repositories:

• AutoRound Format: Vishva007/Muse-Glimmer-30B-W4A16-AutoRound
• AutoGPTQ Format: Vishva007/Muse-Glimmer-30B-W4A16-AutoRound-GPTQ

Quantization Details

The quantization was performed with high-accuracy calibration settings tailored for production-grade inference:

• Scheme: W4A16 (4-bit weights, 16-bit activations)
• Group Size: 64
• Symmetric: True
• Vision Module: Kept unquantized (quant_nontext_module=False) to ensure maximum vision-language alignment.
• Calibration: 512 samples, 800 iterations, sequence length of 2048.

By compressing the language model to under 20 GB, this model fits comfortably within 24GB or 32GB VRAM environments alongside its KV cache and perception encoder.

Usage with vLLM

For optimal performance, including agentic function calling and reasoning parsing, use the vllm/vllm-openai:muse-glimmer image or the latest vLLM build supporting Muse Glimmer.

Run the following command to serve the model (adjust the model path to the AutoRound or GPTQ repo as needed):

vllm serve Vishva007/Muse-Glimmer-30B-W4A16-AutoRound \
  --served-model-name muse-glimmer \
  --gpu-memory-utilization 0.90 \
  --max-model-len 32768 \
  --max-num-seqs 128 \
  --generation-config auto \
  --enable-auto-tool-choice \
  --tool-call-parser muse_glimmer \
  --reasoning-parser muse_glimmer

Key vLLM Flags Explained

• --max-model-len 32768: Sets the context window to 32k tokens.
• --enable-auto-tool-choice: Enables agentic tool selection.
• --tool-call-parser / --reasoning-parser: Configures the endpoint to natively handle Muse Glimmer's multi-step thinking and function schema outputs.

---

About the Base Model (Muse Glimmer 30B)

Muse Glimmer is a 30-billion-parameter causal language model with a dedicated perception encoder (~1.8B param ViT-G/14), distilled from Muse Spark by Meta Superintelligence Lab. It is purpose-built for autonomous agentic tasks on consumer hardware.

Key Capabilities:

• End-to-end Agentic Task Completion: High success rates on DeepSearch QA, MCP-Atlas, 𝛕3-Bench, and SWE-Bench.
• Multimodal Reasoning: Interprets interleaved text and images (charts, documents, screenshots).
• Reliable Tool Use & Failure Recovery: Handles complex tool schemas and automatically diagnoses/retries failed calls.
• Controllable Effort: Supports customizable reasoning strengths (low/medium/high/xhigh) via system prompts.
• Multilingual: Trained on data from more than 100 languages.

🚀 Deploy on RunPod

One-click launch environments pre-configured with PyTorch, CUDA, and dependencies for fine-tuning or quantization.

🎁 Need GPU compute? Sign up via RunPod and get $5–$500 in free credits when you add your first $10.

PyTorch 2.14

PyTorch 2.13

PyTorch 2.12

---

License: Apache 2.0