Muse-Glimmer-30B-DynQuant-4bit

Muse-Glimmer-30B - Hermes function calling (DynQuant 4-bit)

DynQuant-allocated 4-bit. Keeps 78% of the fine-tune's gain at 3.99x compression.

Run this model

No Colab notebook, for a reason worth stating plainly: DynQuant's saving is on disk and on download, not in memory. The weights dequantize to bf16 at load, so this arm needs the same ~60 GB of VRAM as the bf16 repo -- it is 3.99x to store and fetch, not to run.

Engine support above was re-checked against vLLM and SGLang main on 2026-08-30, when this card was generated; it is read from their sources rather than written by hand.

What this is

MuseGlimmerForConditionalGeneration (muse_glimmer, 52 text layers, hidden 6656, untied embeddings, with a vision tower) fine-tuned for tool use on NousResearch/hermes-function-calling-v1.

LoRA r=32, alpha=64 (scaling 2.0) over 416 targeted modules, all 416 injected and confirmed by counting lora_A.default submodules rather than by re-implementing peft's fullmatch rule. Merged to bf16 across 16 shards; the merge's worst per-module relative error was 0.00389 against a 0.0078 tolerance.

Quantized by DynQuant at an average of 3.9999 bits (13.89 GiB, 3.99x smaller than bf16), with per-module bit widths allocated from signals harvested during the fine-tune itself -- activation-RMS saliency and gradient-norm plasticity, collected by forward and backward hooks on the training run that produced these weights.

Results

600 held-out items in single-turn native format, 344 of them carrying structured arguments. Greedy decoding pinned at load (do_sample=False), max_new_tokens=2048, batch 8. Every arm is scored on the same items, so vs bf16 is a paired McNemar exact test on stored per-item hits, not a difference of two independent rates.

The fine-tune is worth +27.33 points over the base model. 4-bit retains +21.22 of that -- 78% of the gain -- at 3.99x compression. The 6.10-point gap to bf16 is statistically real rather than a tie: 28 items lost against 7 gained.

names % is the fraction of items where the predicted tool names match; count % additionally requires the number of calls to match; emitted % is the fraction that produced a parseable call at all. Raw exact-match on this dataset understates by a wide margin -- argument spelling and parallel-call batching alone separate a raw 33.4% from a true 88.4% on the base model -- so the columns are reported together.

Generation budget is not the explanation

A quantized arm that truncates more would lose points to the token budget rather than to its weights. Re-running every paired test with truncated items removed moves each delta by at most 0.02 points, and 4-bit generates shorter than bf16, not longer.

trunc AND exact is zero for every arm. A non-zero value there would mean the truncation flag was derived from the padded batch width rather than from the item, which inflates truncation counts while leaving accuracy untouched.

Compression

What these numbers do not show

They show what quantization costs. They do not show that DynQuant's signal chose good bit widths. The shuffle and uniform control maps were priced but never exported to checkpoints, so there is no matched-bytes arm here. Separating "the allocator worked" from "4-bit works on this model" requires evaluating the shuffled control at identical bytes, and until that arm exists this table should not be read as evidence for the allocation method.

Loading

from_pretrained on its own returns a model that generates fluent nonsense on this architecture. Not degraded -- 0.00% exact match on all 600 evaluation items. The stored weights are not the problem; see below.

model.language_model.embed_tokens is a MuseGlimmerTextNormedEmbedding, an nn.Embedding subclass whose forward is embed_norm(super().forward(ids)) -- a weight-less RMSNorm the class itself documents as un-foldable into the embedding matrix. DynQuant installs DynQuantEmbedding, which derives from _PackedModule rather than from the module it replaces, so that norm is dropped at load and every token enters layer 0 unnormalized.

Measured with identical decoded bf16 weights on both sides:

The plain embedding matches exactly through the same code path. That exact zero is what makes 0.9379 a dropped operation rather than decode error. The norm has no parameters, so it is invisible to every weight, name and byte check -- restoring all 723 quantized weights to their exact bf16 originals recovers nothing.

The repair is at load time: rebuild each packed module by deep-copying from a meta-device model built from the checkpoint's own config, so the original class (and its forward) is preserved, then assign the decoded weight and bias.

Two things below are load-bearing and easy to leave out. First, registration: transformers has no entry-point discovery for quantization methods, so an unresolvable quant_method is downgraded to "not quantized" with nothing but a warning, and from_pretrained hands back a randomly initialised model. Assert the return value -- a registration that declined is the same disaster as no registration, and the loop below would then find zero packed modules and silently do nothing. Second, the decoded weight comes from packed.weight_qt: the packed classes expose neither .weight nor .dequantize() (only DynQuantExpertBank has the latter), so reaching for either raises AttributeError.

pip install "dynquant-core==0.5.0"

import copy, torch
from transformers import AutoModelForImageTextToText, AutoConfig
import dynquant
from dynquant.runtime import ops
from dynquant.runtime.linear import DynQuantEmbedding, DynQuantLinear

assert dynquant.register_hf_quantizer(), "declined; from_pretrained would return noise"

model = AutoModelForImageTextToText.from_pretrained(
    REPO, dtype=torch.bfloat16, device_map={"": "cuda:0"})

# The architecture as transformers builds it: no weights, meta device, so it costs nothing.
# This is the only authority on what class each module was supposed to be.
with torch.device("meta"):
    ref = AutoModelForImageTextToText.from_config(AutoConfig.from_pretrained(REPO))

n = 0
for pname, parent in [(n_, m_) for n_, m_ in model.named_modules()]:
    for leaf, child in list(parent.named_children()):
        if not isinstance(child, (DynQuantLinear, DynQuantEmbedding)):
            continue
        name = "%s.%s" % (pname, leaf) if pname else leaf
        new = copy.deepcopy(ref.get_submodule(name))      # right class, still meta
        new.weight = torch.nn.Parameter(
            ops.dequantize(child.weight_qt, dtype=torch.bfloat16), requires_grad=False)
        b = getattr(child, "bias", None)
        if b is not None:
            new.bias = torch.nn.Parameter(b.detach().to(torch.bfloat16), requires_grad=False)
        elif getattr(new, "bias", None) is not None:
            new.bias = None                    # reference has a bias this module does not
        left = ([k for k, t in new.named_parameters() if t.is_meta]
                + [k for k, t in new.named_buffers() if t.is_meta])
        assert not left, "%s: %s left on meta -- not supplied by the rebuild" % (name, left)
        setattr(parent, leaf, new)             # drops the last reference to the packed child
        n += 1
        if n % 100 == 0:
            torch.cuda.empty_cache()
del ref
torch.cuda.empty_cache()

# Nothing may survive holding a qweight. Test the buffer, not the class: a packed type this
# loop does not know -- an expert bank, say -- would otherwise stay behind silently.
still = [k for k, m in model.named_modules() if getattr(m, "qweight", None) is not None]
assert not still, "%d packed modules survived" % len(still)

# Every number on this card is greedy. transformers v5 refills any generation field the caller
# leaves unset from the checkpoint's own generation_config, and this checkpoint ships
# do_sample=True / temperature=1.0 / top_p=0.95 / top_k=64 -- so a generate() that does not say
# otherwise samples, and will not reproduce them. Setting do_sample=False is what makes it
# greedy, per call or here; the other three are then unused (the temperature/top-k/top-p warpers
# are built only inside `if generation_config.do_sample`) and nulling them just silences the
# validation warning.
gc = model.generation_config
gc.do_sample, gc.temperature, gc.top_p, gc.top_k = False, None, None, None

Rebuild in place, child by child, rather than collecting the packed modules into a list first. A list holds a strong reference to every packed module for the whole loop, so none of them can be freed while the dense copies accumulate beside them, and the peak is the packed model plus the entire dense model at once. On this checkpoint that OOMs a 95 GiB card: the setattr below drops the packed child immediately, which keeps the peak near the 56.9 GiB the dense model actually needs.

The per-module meta assertion is load-bearing too: a module that fails to receive one of its real tensors -- an uninitialised norm scale, say -- stays on the meta device and raises only later, deep inside a forward pass, where the cause is unrecoverable. That is the same silent corruption this whole section exists to fix.

For batched generation also set gc.pad_token_id = tok.convert_tokens_to_ids(tok.pad_token); without it transformers pads with the eos id and the attention mask silently disagrees with the sequence, which changes results for every item in a batch but not for a batch of one.

This is not fixed upstream. integration/hf_quantizer.py and runtime/linear.py are byte-identical across 0.5.0, 0.5.1, 0.5.2 and 0.5.3, so a newer DynQuant changes nothing here. 0.5.0 is pinned because it is the version these numbers were measured under; installing dynquant-core rather than the dynquant meta-package keeps the compiled kernels out, which is correct on any architecture without a matching kernel build -- the recipe above decodes to dense bf16 and never calls one.

Evaluation details

• Dataset: NousResearch/hermes-function-calling-v1, single-turn native format.
• 600 items; 344 carry structured arguments and are the paired-test population.
• Scoring compares tool name, argument names and argument values after normalisation; parallel calls are matched as a set, not in emission order.
• Significance: McNemar exact on per-item hits stored for every arm.

License

Inherits the base model's license. See LICENSE and USAGE_POLICY.md in this repo.