Muse-Glimmer-30B-mxfp8

Muse Glimmer 30B — MXFP8 Quantization

This is an MXFP8 derivative of meta-models/Muse-Glimmer-30B, pinned at revision a4e59da52a7bc87ae7251dd5545c0dd437c44b68.

LLM Compressor converted the BF16 weights of the language model's linear layers to MXFP8. No calibration data or fine-tuning was used. The vision tower, vision adapter, vision projection, and lm_head remain unquantized.

Quantization details

The quantized layers cover the attention and feed-forward projections in each decoder layer. The checkpoint contains eight safetensor shards.

Hardware requirements

MXFP8 acceleration requires OCP MXFP8 hardware and a compatible inference backend.

This checkpoint passed a deterministic tensor-parallel-1 language-only vLLM smoke test on AMD Instinct MI350X (CDNA4, gfx950) with HIP runtime 7.2.53211. vLLM selected RocmDotScaledMxfp8LinearKernel for the quantized linear layers.

Note: Multimodal generation, model-quality parity, and other hardware configurations have not been validated with this checkpoint.

Usage with vLLM

The validated path loads only the language model:

from vllm import LLM, SamplingParams

llm = LLM(
    model="rkayaith/Muse-Glimmer-30B-mxfp8",
    language_model_only=True,
)

outputs = llm.generate(
    ["Tell me about AMD Instinct MI350X."],
    SamplingParams(max_tokens=200, temperature=0.7),
)
print(outputs[0].outputs[0].text)

Use this command to start a language-only server:

vllm serve rkayaith/Muse-Glimmer-30B-mxfp8 \
  --language-model-only

Comparison variants

License

This derivative is distributed under the source model's Apache License 2.0. Use of the model must also comply with the source model's Usage Policy.

Modifications: The source BF16 language-model linear weights were quantized to MXFP8. The vision components and lm_head were retained without quantization. No fine-tuning occurred.